Technical Notes On DVD-Videoby Billy Biggs
I got a dxr2+drive in Christmas 1999, but I didn't start seriously hacking with DVDs until about March 2001. I first attempted a recompressor, and when that was going poorly, I began to convert my code into a player. I learned alot from #livid on freenode.net. If you want to talk about DVDs, I'm likely there.
I gave a talk on the linux DVD development scene, as well as a brief introduction to DVD-Video. The slides for the talk are available online.
DVD subtitlesI finally added subtitle compositing to movietime and learned a few things. For more information on DVD subtitles, visit sam's subtitle page.
A brief overview of DVD subtitlesSubpictures are encoded as 4 colour images, each colour having a 4 bit alpha value. The 4 colours are chosen from a 16 colour palette of Y'CbCr values. This does not leave much room for artistic design. Subpictures are primarily used for menu buttons and subtitles.
A typical subpicture allocates colours similar to this example from Ghostbusters (1984) (the one at the bottom):
Gamma-correct DVD subtitle compositingAs far as I can tell, doing a gamma-correct composite over Y'CbCr images is difficult. Y'CbCr is a nonlinear colour space, that is, it's a linear transform from R'G'B'. One way to do a correct composite between two colours in Y'CbCr would be to:
This process is made difficult because of how the chroma is downsampled. Since we're trying to be all correct, we have to upsample the chroma from the MPEG2 video from 4:2:0 to 4:4:4 (realigning the chroma samples with the luma samples!), then do all the hard work, convert back, and then downsample the chroma back to 4:2:0. Ouch!
It's still unclear to me if I can't just apply a new gamma function to the Y'CbCr channels independantly. I don't want to resample the chroma, and I can get around that by acting on each channel independantly. So the question is, how do we map the gamma function directly to Y'CbCr space? Can we do this at all? This issue is still completely unclear to me. I'll have to play around in maple/matlab more.
Deinterlacing DVDsDeinterlacing and 3:2 pulldown inversion are important for playback of DVDs on progressive-scan displays like computer monitors.
A scary issue is mentioned in MPEG document 2820 which indicates that the 'progressive_frame' flag in the MPEG2 header is unreliable for deinterlacing purposes! This moves intelligent deinterlacing almost completely into the image heuristics area, except for reversing 3:2 pulldown when performed using the repeat_first_field flag. Note below that even then we can have problems, see under 'weird 3:2 pulldown encoding'!!
An example of 3:2 pulldown encodingThe following sequence is taken from the NTSC release of Lawrence of Arabia, Title 1, Chapter 15. It is the first 11 frames.
The first 5 frames are marked as interlaced (thanks!) and so the coded framerate is 29.97fps, but the material is clearly from 24fps source with 3:2 pulldown applied. The DVD then switches into progressive mode, and uses the repeat_first_field flag to offload the pulldown work onto the player. This switches the effective coded framerate down to 23.976fps.
In this DVD, small bits of scenes have been encoded at 29.97fps instead of always coding at 23.976fps. Why would they do this? One thought is that maybe certain scenes were touched up at video speed to remove objectionable artifacts in the pulldown conversion, but that doesn't seem to be the case here. I did notice that often we see some interlaced frames near the beginning of chapters. Maybe they fear some DVD players need time to switch into 24fps mode?
A really weird 3:2 pulldown encoding!!This came as a complete shocker to me. Here is some output of the first 150-or-so frames of The Good, The Bad, and the Ugly (1966). Take a careful look at all the non-repeat_first_field frames! They're interlaced! Not only that, but the progressive_frame flag is high the whole time!
The conclusion here is that a correct deinterlacer must look at _every_ non-repeat_first_field frame, even if we're clearly in a pulldown sequence! What a mess!My biggest question here is why? Why would they ever do this? One observation we make is that every second frame is a blend of the two beside it. So, maybe the only print they found of the opening credits was at 12fps? Maybe it was originally recorded at 12fps and this is a conversion technique? Maybe the quality was so bad, they only decided to restore every second frame? If you have thoughts, please email them to me.
Realtime deinterlacing results on the bjork sequenceThis video sequence is taken from the Bjork: Volumen DVD of music videos. It was filmed at video speed of 59.94fps. We compare images from:
I want to try out VirtualDub's adaptive deinterlacing algorithm, which Jarl gcc'ified in drip, but it only acts on 24-bit R'G'B' images so it will take a while to adapt it to my code. I also intend to port some of the dscaler algorithms, but I haven't got around to it.
I was surprised at how reasonable ffmpeg's filter is. I find the results from it quite reasonable. I tried it out with my motion test, but the results weren't so good. I'll try and get some shots of different algorithms up, even bad ones, just for comparison. That said, it's still an interpolation filter, and so the results aren't all that great.
The shots below are for when we are deinterlacing the top field, that is, we want to get the highest quality image corresponding to the point in time of the top field. These frames are all top-field-first, so, we're deinterlacing using pixels from the future. None of the algorithms currently shown use more than 2 fields to deinterlace, but of course, I will add more shortly.
Dealing with framerate conversionsI have posted a sequence of frames and discussion about the 625/50 (PAL) release of the first season of The Simpsons on DVD.
The page shows the raw fields, not frames, from the DVD. Clearly we are seeing the result of some framerate conversion, and I think it's pretty crazy. Every n fields (I haven't determined n yet) we see a field which is just a half/half blend of the two adjacent fields. The result is a nasty blurry frame. Ugh!
Not only is it ugly, but for someone trying to deinterlace, this is going to make any algorithm suck. I'll post results up soon of what artifacts this leads to when you look closely.
Every top field is a blur of the two adjacent fields, and then after a while we see three fields each of which is not a blur, and now our phase has changed, so every bottom field is a blur. Doing out the math, I'm pretty sure they converted 24fps to 50fps by approximating each refresh of 50hz with the closest two 24fps frames, and if two are present during that refresh show the blur.
See this page for more images with discussion.
Using MPEG's motion vectors for deinterlacingThis idea keeps coming up. Sometimes I think it will work wonders, and other times I think the idea will hurt us really bad when motion vectors don't acurately represent the motion of objects in the scene: Consider a cut between two cameras shooting the same scene at a different angle. The other problem is that we only have motion vector information for predictive frames. Ugh.
Hopefully sometime soon I'll hack libmpeg2 to export the motion vector information and I can try to build a deinterlaced image on the fly. But since I could see this as being alot of work, I'll wait until my generic motion-vector based deinterlacer is written.
Buying DVDsI like to browse and buy DVDs at local Future Shop when they are cheap. Over the net, I have found CNL cheap and Canadian, dvdplanet is good for U.S. releases and titles which CNL does not have, and I intend to order some PAL DVDs from BlackStar but I have not made an order yet (waiting to see if the PAL release of Transformers: The Movie is actually a widescreen transfer).
My DVD collection is sometimes updated.
Mon Feb 3 12:41:38 EST 2003 Updated from openprojects to freenode Wed Nov 14 14:06:24 PST 2001 Added some more text on the Simpsons sequence Mon Nov 12 19:14:52 EST 2001 Added the Good/Bad/Ugly sequence results Mon Nov 5 02:51:21 EST 2001 Added the Simpsons sequence page Sun Nov 4 15:07:16 EST 2001 Added the Bjork deinterlacing results