DATA PROCESSING & Normalization

Archival Processing & Normalization

Within two weeks of publication on the Media team’s Distribution Portal, project data must be processed and normalized by the MM Archives for long-term storage. Once a completed project has been processed, it will be ingested into the EditShare FLOW production DAMs and access permissions will be changed to “read-only”. HRW MM editors and producers must follow HRW Naming & Arrangement Standards throughout the production process and the MM Archives will correct errors and omissions made to files created at HRW.

This HRW MM Project cleanup process ensures data will be easy to locate, maintain integrity for future production, prepare data for the HRW Legal department as evidence, and be understandable and renderable to users many years from today. The guidelines below provide information that will help Multimedia staff producing and archiving content to clean and preserve completed projects for long-term, read-only storage.

General Best Practices and Tips

  1. Files that are created outside of HRW (camera native originals, licensed media, social media downloads, etc.) must retain original filenames and metadata to retain provenance and, in some cases, data integrity for use as potential legal evidence.

  2. Files that are created by HRW staff (final master exports, distribution files, project files, etc.) must be named according to HRW Archives naming standards (ie: YYYYMMDIV_Country_Subject) as outlined in the HRW Naming Guide.

  3. Please ensure that spaces and the following forbidden characters do not appear in final file names.
    * . " / \ [ ] : ; | ,

  4. Note that staff often “cut corners” and name files in a way that follow our naming standard but do not match the actual project title. It can be extremely difficult to spot these errors and omissions. Try to spot the file-naming errors below for a project named “202006MENA_Jordan_Kids_Education”

    • 202006MENA_Jordan_Kids_FINAL.mov

    • 202004MENA_Jordan_Kids_Education_FINAL_ARABIC.mov

    • 2020MENA_Jordan_Kids_Education_FINAL_CLEAN.mov

    • 202006MENA_Jordan_Education_FINAL.prproj

  5. Please use the NameChanger application to correct mislabeled files, remove forbidden characters, and ensure all HRW-created files are named correctly.

  6. When large sets of files created through automated processes (such as auto-save files) are present, they must be packaged in a ZIP container to avoid DAMs ingest slowdowns. Please compress folders containing auto-save files in a ZIP container by right-clicking on the folder and choosing “Compress”. When the compression process has completed, please delete the original containing folder to avoid unnecessary data redundancy.

 
folder.png

MASTER TEMPLATE
Please use the following guidelines to process and normalize incoming Multimedia Team projects contained within the Master Template.

audio – This folder contains source audio files and is generally clean. Check for duplicate files such as folders that were extracted from ZIP containers with matching names. If you see a folder and ZIP container of the exact same name and data size, please delete the ZIP container. Delete any audio render or preview files you encounter (usually .CFA files from Adobe Premiere).
distribution – Please ensure that compressed, final MP4 exports for distribution (media.hrw.org, YouTube, etc.) are located in this folder and are named according to our project naming standard. Use the NameChanger application to make file name changes.
documents – This folder is usually clean and does not require normalization or processing adjustments.
exports – Please ensure that all MOV exports for this project are present, named according to our standard, and located in the “finals” folder. Double-check to make sure a “CLEAN” final master copy of the feature is present. Use the NameChanger application to make file name changes.
graphics – If auto-save project files are contained in a folder, please compress them into a ZIP container and delete the original folder, which is now redundant.
logs – This folder is usually clean and does not require normalization or processing adjustments.
originals – With the exception of render files created during the production process, data contained in the “originals” folder is not adjusted or normalized by the archive. This folder should contain a series of folders named and arranged by date, videographer name, and following subject information as defined by the HRW Ingest application. That said, staff who fail to use the ingest app may store data in the “originals” folder in a nonstandard form. HRW Archives policy stipulates that original data must not be renamed or rearranged so that provenance is maintained. Only the editors and producers originally assigned to work on this project are allowed to make adjustments. For legal reasons and to respect data as evidence, the HRW Multimedia Archive does not make alterations to original files, original file names, and arrangement, respecting the archival principle of original order. With that in mind, the originals folder sometimes contains render files that do not need to be retained and can wreak havoc on digital asset management systems. Please delete any render files found in the “originals” folder, specifically Adobe CFA and PEK files.
projects – Please ensure that the final project file for this project is present, named according to our standard, and located in the “finals” folder. If auto-save project files are contained in a folder (“Adobe Premiere Pro Auto-Save”), please compress that folder into a ZIP container and delete the original folder, which is now redundant. Delete any render or preview files you encounter, including the “Adobe Premiere Pro Audio Previews” and “Adobe Premiere Pro Video Previews” folders.
third-party-media – This folder may not adhere to HRW naming and arrangement standards, but files it contains are not adjusted or normalized by the archive. Only the editors and producers originally assigned to work on this project are allowed to make adjustments. For legal reasons and to respect data as evidence, the HRW Multimedia Archive does not make alterations to original files, original file names, and arrangement, respecting the archival principle of original order. With that in mind, please delete any render files found in the “originals” folder, specifically Adobe CFA and PEK files.
transcodes – Transcodes should be placed by editors in folders that mirror those found in the “originals” and “third-party-media” folders. Please do not delete or alter transcode files.