DVRescue Workshop #1: dvpackager and dvloupe
This blog post is part of a series of workshops conducted as part of the DVRescue project. These workshops were originally intended to be held in person over the course of several days, but due to COVID-19, they are held online with members of the DVRescue team and participating organizations.
DVRescue is made up of a series of tools that support archivists in capturing, managing, and packaging DV material. For dvpackager, we aspire to be able to guide archivists through the setup and transfer processes, so we can collectively get more tapes transferred off their physical carriers, have those digital transfers be of higher quality, and allow archivists to be able to perform quality control on the content.
This first workshop focused on the dvpackager and dvloupe tools within DVRescue.
Around a decade ago, there were a lot of different options for transferring DV tape but they have all been abandoned. For now, archivists use old technology and hope that it does not stop working when they update their computers (or refrain from performing any software updates in anticipation of this).Ā
The software that previously existed to transfer DV tape to computers falls into two categories: capture everything into one DV stream, or (with tools such as Apple Final Cut Pro or Adobe Premiere), write all of the data into a container such as Quicktime MOV.
DVRescue takes a different approach and captures a DV stream and then packages that stream into a common video container after the transfer is complete.
To compare the pros/cons of these two approaches: When you are capturing frames off the deck, there's a chance the technical characteristics change during the playback, like having the aspect ratio change from 4:3 to 16:9.Ā
As an example taken from one of our workshop participants: With specific recordings, the content would switch from 2-channel to 4-channel in the middle of the tape. If you tried to capture that in Apple Final Cut Pro, it would cut at this point. With the DVRescue approach, when an archivist wants to get all of the media off of the tape, that is possible. Then, the archivist can sort out the media later. This allows the entire content to be captured and then cut if necessary later, rather than an archivist having to constantly monitor the transfer station to ensure each clip does not stop the transfer from happening.
Video containers that have been commonly used to wrap DV, such as MXF, Quicktime (MOV),Ā and Matroska (MKV), all have problems with key technical characteristics being variable.
These are some examples of variable technical characteristics:
dvpackager analyzes the entire stream and packages these components into the wrapper of the archivist's choosing.
To understand this a bit more, it helps to understand how DV works.
Another tool in the DVRescue package is dvloupe. dvloupe is for letting archivists look at a DV stream using a hexadecimal view.
Using dvloupe, we can see that DV streams are made up of 80 byte long DIF blocks. 150 of these DIF blocks make up 1 DIF sequence, and this sequence is 12,000 bytes long. In NTSC, 10 DIF sequences make up one frame. For PAL, 12 sequences make up one frame (because PAL is taller).
dvloupe color-codes what these bits are all for so that it's easier to read and decipher the results.Ā
There's a header row (pink), similar to a clear barrier in 16mm film, as a break. The timecode lives in the cyan-highlighted portion of the block, the subcode DIF blocks.Ā
The yellow section contains VAUX (video auxiliary data), which has different pieces of technical metadata like channel count or aspect ratio. Closed captioning data is stored in this VAUX section as well. DV can include captioning data directly in the stream.
The green parts are showing the raw PCM audio that is interleaved within the data. The majority of the frame is video data, in YUV. The U and V sections are very small compared to the Y because DV has really low color resolution, and the majority of the stored data is brightness.
Okay, back to dvpackager.
When taking a DV stream and putting it into a container, it is feasible to put the DV directly into the container as it is. But generally, containers don't like to handle data for more than one purpose, so they want to handle everything as video even when DV as a stream is managing different roles at once (e.g. video, audio, captions). When DV comes from tape, it ends up being damaged or inconsistent, so you are having your player try to guess what the technical characteristics are from the first frame.
Putting DV into a container can make it more resilient, when these properties can be mapped over successfully. dvpackager will move the video components into the video stream, and move the other parts of the stream into their own streams in the container, like audio, metadata, captions, chapters, or error information.
One of the internal aspects of the DVRescue project is how we generate structured data, which is stored in XML. Running a DV file with technical characteristics that change several times, the output will be represented by <frames> elements including those characteristics, and report the first and last frame of the sequence, and any frames that contain unusual events like errors or timecode jumps.
As we've seen here, dvpackager can package DV streams into an archivist-determined wrapper, but it can also "unpackage" the file back out into a DV stream, making the process reversible. This is not guaranteed, though, and there are some exceptions when this won't work:
When capturing DV and you unplug your computer in the middle, and the last frame is not a full frame, the partial frame will be filled in at the end
Presentation time can be difficult to map back and forth with two different timebases
There are potentially other edge cases as well when the process will not be 100% reversible (and verifiable through checksums), so it is not a guarantee.
DVRescue is available to download and test on all major operating systems through MediaArea. We encourage all testing feedback, errors, and issues to be filed on our GitHub Issue tracker.
germansingergirl
theartofmadeline
izzy806
romaincrisis