In a perfect world, we would have zero frame delay but unfortunately this is not possible with current computer hardware; the simple fact that a computer is processing video already adds one frame of input delay, that frame is needed for the computer to store the complete image before having it processed by the video card. Then you can add another frame of output delay, caused by the video card processing the image. Before or after that, each device that is added to the pipeline – such as a camera, a mixer, a LED controller or a projector – will add some delay into the processing. In the end, if the delay becomes too big you will notice that the sound and the video are out of sync, for example you will hear the voice of the speaker before his lips start moving on the screen.

If we cannot tell what the latency will be with your setup, we can however provide you with a point of comparison by measuring how many frames are needed to process a live input on an ideal setup: MediaMaster running on one of our A10 Media Servers that are specially built and configured to get the most out of our software.

Here is how we proceeded: we’ve setup a MacBook Pro with DVI/HDMI output connected to the Blackmagic Intensity Pro input card from the A10 Media Server which was set to an input resolution of 720P 60Hz. Then we started a frame counter movie on the MacBook Pro and launched the live input from MediaMaster full screen on the A10 Media Server. Finally, we put the monitors next to each other and took a picture so we can see the difference between the frame counter on both screens.

As you can see in the movie, when we took the picture the MacBook Pro was at frame count 24429 and the output of the A10 Media Server was at 24424: there is a 5 frames delay between the counter from the laptop and the counter from the server. Since our whole setup was running at 60Hz, this means that the processing delay equals 83 milliseconds (1 second divided by 60 frames = 16.6 milliseconds per frame, 5 times 16.6 milliseconds equals 83 milliseconds).

One thing that appears obvious to us after more than 10 years of working with the various solutions that can be used on Mac and Windows to decompress video is that not all solutions are equal.

We believe that our users should be able to play almost any kind of media with the ArKaos software and at the same time we do everything a team of developer can to get the best performance on the platforms supported.

Practically the ArKaos software can play media files through the following decoders: QuickTime, FFmpeg, the Flash plugin, Quartz Composer and Windows Media Player. Currently (December 2009) we don’t allow the user to decide through which solution a media will be decoded, instead we try to make the best decision based on our many years of experience and tests. In the case where several video codecs (such as Mpeg4) can be played through different decoders like QuickTime, FFmpeg and Windows Media Player, we always prefer, no matter the platform, to first try to handle those codecs with FFmpeg as we have always had the best results with FFmpeg during our tests, and here’s why..

The other solutions..

Of course, on the Mac we could stick to QuickTime.. But sadly it appears that QuickTime has become almost obsolete today and is just used to decompress the codecs needed by the iPods. Most codecs are not properly multi threaded in QuickTime and this is a big problem for us. With GrandVJ or MediaMaster taking advantage of multi core machines by using each core separately to decompress the video layers you are mixing, it’s too bad to miss that great improvement by playing videos that absolutely need the QuickTime support.

On the PC we could be using Windows Media Player, it is a wonderful technology but it’s architecture is completely incompatible with the sophisticated engine we use to allow frame blending and frame prediction in the latest version of GrandVJ and MediaMaster.

Flash animations and Quartz Compositions, of course, are unique in the sense that they can only be read by their own dedicated player.

As you now understand, once you open a media file in ArKaos your content can go through various roads and it’s important to consider that. To get the most out of GrandVJ or MediaMaster, benefit from the advantages of multi threading and perform frame blending you should make sure that your content is always played through FFmpeg.

How does ArKaos process the movie file?

On the Windows platform it’s pretty easy because the software first tries to play a file with the FFmpeg player and then falls back to the others (Windows Media Player or QuickTime) if needed.

But things get harder with the Mac platform because some files that could be played by FFmpeg may be rejected and fall back to QuickTime just because they have not been saved the correctly. Many QuickTime files have been created in a way that make them compatible with Mac OS X but also it’s ancestors Mac OS 9 and 8 and 7 and… Those files make use of what is called a Resource Fork.

To make it short, when Apple introduced the Macintosh years ago they decided that a single file would in fact be two files: the data file itself and an optional resource file. Many software have been using that Resource Fork to store additional data and information. This is now a problem for many .mov files that were created in the past and even still today with some applications that were born under the Mac OS 9 era and still generate QuickTime .mov files that have a Resource Fork.

This is quiet an issue for us because when the FFmpeg decoder can’t handle those files and makes ArKaos fall back to QuickTime, you end with QuickTime’s very bad multi core support even for very common codecs such as Photo JPEG. Another bad side effect of this Resource Fork is that most of the time when you will copy a movie file that has a Resource Fork to a Windows or Linux machine it will simply not be readable anymore because that tiny bit of information contained in the Resource Fork is not read on other platforms although it is vital to play the content.

How to get rid of the Resource Fork?

Thankfully it’s very easy to remove that annoying Resource Fork from a QuickTime file. You can simply open the file in today’s QuickTime player and do a “Save As”, QuickTime will reprocess the file correctly. The process is called “flattening” a movie file, it is very fast and will not re-compress your movie, this is very important to avoid re-compressing when possible.

Open your file in QuickTime and go to File, Save As..

.. and enter a new name

Of course, you will only need to do that if your file was created on a Mac, the free version of the QuickTime player is enough to do this and it works on all Mac OS X versions up to Snow Leopard with the latest QuickTime player. Now if you have lots of movie files to convert, you can of course look for some professional software that can queue and “flatten” a bunch of files from a directory.

FYI, here is a QuickTime file that has a Resource Fork (moov), inspected with a programming utility called Resorcerer:

And here is the same file that has been flattened:

How do I know what decoder is used?

In GrandVJ it’s very easy to see what player is used by the ArKaos engine when processing a file; simply hover your mouse over the cell that contains the video and the contextual help appears.

In this one, the video file with a resource fork will be played through QuickTime:

In this one, the same video flattened with the QuickTime player will go through the FFmpeg path:

(The same information is also available in GrandVJ for each movie listed in the visual library and in MediaMaster you can get that information from the file library window.)

Conclusion

During the first days of the release of GrandVJ 1.2 beta 3 with the new audio engine (which relies on FFmpeg) some users have complained that some old .mov files were not playing audio correctly with the beta version, this was most of the time caused by the Resource Fork and flattening the files fixed the problem. As a bonus, it will make your files cross platform.

This should give you some lights on what is happening when you drop a file on GrandVJ or MediaMaster, as you can see a lot of work is being done under the hood :)

Last couple of month, we’ve been working very intensively on the new release of MediaMaster (version 1.1 – which should be out very soon). One of our goals for this release was to improve radically our engine to be able to run a couple of movies at monitor rate with no frame drops.

60 fps that is.

And we did it.

This means that you will be able – of course depending on the computer and the movies you are displaying – to run movies perfectly locked to the monitor sync.

The result is brilliant – even for movies at 30 fps – because your eye directly catches the smoothness of the playback.

But let’s go back to the beginning of our article. How do you show that ? If we want to show you a smooth 60fps stream, every step of the chain needs to be perfect.

The first step is to be able to capture the stream. We have a couple of BlackMagic cards that allow us to record the dvi output from one computer to another, but it’s always a bit of a bummer to setup since we don’t do that on a regular bases.

Enter Fraps. Fraps is the essence of programs I love. They’re simple, possibly ugly, but work like a charm. It’s been originally mentioned on our forum by Blindskunk and I have to say I’m deeply impressed.

Fraps is a Windows-only program. It hooks to directX and records full screen output. It’s fast, and it works. Just what you want.

So here we are, on a fairely beefed up machine, running at the same time MediaMaster who’s pumping movie pixels at 60 fps (1024×768) and fraps which catches it – also at 60 fps – and we don’t loose one single frame. How amazing ?

The next step is of course how do we display this to you guys ? We’ve tried a couple of flash-based solution and none of them worked. We tried YouTube, Vimeo, standelone player and they all failed miserably… so in the end we decide to use good ol’QuickTime.  Click on the image below to view a reduced version of the file.

For people that don’t have an efficient QuickTime Player, or want to see the thing in all of it’s glory, you can download a full size (22 mb) version in flv format. We recommend you to play it with VLC, it’s really smooth with it.

This movie is running two layers. The first one (the speakers) is a 30 FPS 1920×1080 movie loop from our friends at MotionLoops. The second one is a relatively lightwheight curtain swipe running at 60 FPS. The thing is that with that swipe, if you loose one frame, you see it directly.

Note also that if your monitor frequency isn’t at 60 fps, you will see glitches. At that rate, ALL the elements have to be perfectly tuned.

As you can see, the output is as smooth as the pie for which that town was named after…

Thanks Fraps people !

When you are in the show industry and try to find out the best way to compress your content it’s still a little bit of black magic. While some codec are able to compress video very well they are heavy to handle for the machines, even more, the codec that does the best job at compressing while keeping a good quality such as H264 are very bad when you need to scratch your media.

The best codec for users that need to interact a lot with the content should allow to play forward and backward easily and should allow to jump into the content quickly.

Recently, I’ve heard about experiments with a file format designed by graphic card manufacturers and based on texture compression DXT1 DXT3 or DXT5 and I was wondering how hard it would be to add support for such a format to QuickTime through a new codec. Being programing for the QuickTime API since it’s version 1.0 beta I was considering the challenge fun and interesting.

I started by resuscitating an old sample code from 1999 found on the Apple web site and after a few hours of coding and testing I had an ArKaos codec generating video files that could be played back by QuickTime.

ArKaos codec in the QuickTime player

Then I jumped on the DXT texture compression problem and chose to use the squish library to handle the job. A few more hours later and my experimental codec was working with texture compression.

The codec when used by QuickTime at that point was not efficient because the texture decompression was done on the CPU, so there’s no advantage of using the GPU to handle that texture format.

To measure the speed up I used an experimental player I am working on, and patched ffmpeg to handle my new codec. This allowed me to leave the texture data untouched and pass it to the GPU via the OpenGL extensions that handle texture compression.

When this was done I could start enjoying the show and make some performance tests; in the table below the file size is in MB and CPU load is taken from running top in the console:

If you are curious about frame quality here is a crop from an original frame in Apple intermediate :

Original frame - Apple intermediate (click for full size)

Here is the same cropped frame using Dxt1 with texture compression:

HD frame - Dxt1 compressed (click for full size)

And here is the same cropped frame with Dxt5 compression:

Same frame with Dxt5 compression (click for full size)

So the results of these tests are that pure Dxt1 is the best format when it comes to CPU load, Dxt5 is better if you focus on quality but it generates files that are twice bigger. Actually, using Dxt1 I could have played 7 HD loops at the same time with my laptop … I mean, if hard drive’s data rate would’ve allowed it!

The problem with pure Dxt5 or even Dxt1 files is that they are huge, about 4 times bigger than photo jpeg or H264 encoded files. So my workaround was to use data compression on the CPU, since it’s almost unused!

To conclude, it looks like using Dxt1 with texture compression is the best solution to optimize both CPU and disk usage with a file size that is almost the same as Photo jpeg and a little bit bigger than H264. And the codec is still twice faster than any other codec I tested when using the CPU to decompress the texture data from disk.

On my venerable old MacBook pro I can’t smoothly play a loop in Full HD and even at 1280*720 my machine is still struggling. Thanks to Dxt1 with texture compression I can play two HD layers and three 720p layers.

Now I know you guys would like to play with this new toy so here’s a .dmg file with the test player I have used, along with a set of video loops. Beware that this dmg file is for Mac OS X x86 only (no Power PC): and here’s the link

You can be sure that we will bundle that new experimental codec in the next update of GrandVJ and MediaMaster, so everyone can enjoy it!

Roughly, the way our application works is the following: at each pass, we read  a video frame from the disk if needed, decode it and then send it to the GPU (the Graphic card’s CPU) together with all blending and effect operations. Then we ask the graphic card to flush all operations (that is actually processing all pending graphical instructions) and present it to the display.

If you look at the bottom of the user interface window in either NuVJ, GrandVJ or MediaMaster, you will notice that there are two numbers constantly updated.

Something like:

Graph: 400.00 Total: 60.00

These numbers indicate the speed at which the engine does its rendering.

The numbers next to ”Total” represents an average of the actual frame rate of the display. If you start the application’s fullscreen without any visuals, the total should give you the monitor’s frequency, which is the maximum frame rate you can ever achieve.

The numbers next to ”Graph” will give you an indication of what the framerate would be if we only decoded the movie and send the graphic instruction to the graphic card without asking it to actually flush the instructions and update the display.

So unless you use a lot of heavy effects (like the rotozoom for example), the graph value is a rough reading on how heavy it is to decompress your loops.

From these indications you can conclude the following:

• If your framerate drops and but the “Graph” reading stays high, it’s most likely resolving the graphical output that is the bottleneck. Probably you run at a resolution that is too high for your graphical card or your card doesn’t cope well with downloading the textures back to central memory for preview display.
• On the other hand, if both the graph and the total value go down, it’s most likely the movie decompression that is the bottleneck. Either you use a codec that isn’t effective (make sure all you videos are decoded through our FFMPEGplayer – preferably photoJPEG, mpeg2, H.264), or you are decoding to many movies for what your harddrive can cope with.

It features  Interviews and Footage from The Light Surgeons, D-Fuse, Hexstatic, Vj Anyone, Addictive TV, Vector Meldrew and Fatamorgana.

About V.I.A.

With over six years experience in the live events industry, the V.I.A – (Visually Impaired Artists) cultivate a range of digital disciplines for the mutual love of live performance.

With professional backgrounds in the Broadcasting, Film and Live events industries their live performances bring together a mix of Animation, Motion Graphics, 8mm, Film & Video, Video feedback, live camera work and Lighting Design.

Here is the result:

Ok there is a black frame when switching queues but hey, it’s a prototype :)

It’s fully multithreaded so development is moving forward carefully. There are threads for the network synchronization, video decompression, OpenGL rendering and status update to the server