One thing that appears obvious to us after more than 10 years of working with the various solutions that can be used on Mac and Windows to decompress video is that not all solutions are equal.

We believe that our users should be able to play almost any kind of media with the ArKaos software and at the same time we do everything a team of developer can to get the best performance on the platforms supported.

Practically the ArKaos software can play media files through the following decoders: QuickTime, FFmpeg, the Flash plugin, Quartz Composer and Windows Media Player. Currently (December 2009) we don’t allow the user to decide through which solution a media will be decoded, instead we try to make the best decision based on our many years of experience and tests. In the case where several video codecs (such as Mpeg4) can be played through different decoders like QuickTime, FFmpeg and Windows Media Player, we always prefer, no matter the platform, to first try to handle those codecs with FFmpeg as we have always had the best results with FFmpeg during our tests, and here’s why..

The other solutions..

Of course, on the Mac we could stick to QuickTime.. But sadly it appears that QuickTime has become almost obsolete today and is just used to decompress the codecs needed by the iPods. Most codecs are not properly multi threaded in QuickTime and this is a big problem for us. With GrandVJ or MediaMaster taking advantage of multi core machines by using each core separately to decompress the video layers you are mixing, it’s too bad to miss that great improvement by playing videos that absolutely need the QuickTime support.

On the PC we could be using Windows Media Player, it is a wonderful technology but it’s architecture is completely incompatible with the sophisticated engine we use to allow frame blending and frame prediction in the latest version of GrandVJ and MediaMaster.

Flash animations and Quartz Compositions, of course, are unique in the sense that they can only be read by their own dedicated player.

As you now understand, once you open a media file in ArKaos your content can go through various roads and it’s important to consider that. To get the most out of GrandVJ or MediaMaster, benefit from the advantages of multi threading and perform frame blending you should make sure that your content is always played through FFmpeg.

How does ArKaos process the movie file?

On the Windows platform it’s pretty easy because the software first tries to play a file with the FFmpeg player and then falls back to the others (Windows Media Player or QuickTime) if needed.

But things get harder with the Mac platform because some files that could be played by FFmpeg may be rejected and fall back to QuickTime just because they have not been saved the correctly. Many QuickTime files have been created in a way that make them compatible with Mac OS X but also it’s ancestors Mac OS 9 and 8 and 7 and… Those files make use of what is called a Resource Fork.

To make it short, when Apple introduced the Macintosh years ago they decided that a single file would in fact be two files: the data file itself and an optional resource file. Many software have been using that Resource Fork to store additional data and information. This is now a problem for many .mov files that were created in the past and even still today with some applications that were born under the Mac OS 9 era and still generate QuickTime .mov files that have a Resource Fork.

This is quiet an issue for us because when the FFmpeg decoder can’t handle those files and makes ArKaos fall back to QuickTime, you end with QuickTime’s very bad multi core support even for very common codecs such as Photo JPEG. Another bad side effect of this Resource Fork is that most of the time when you will copy a movie file that has a Resource Fork to a Windows or Linux machine it will simply not be readable anymore because that tiny bit of information contained in the Resource Fork is not read on other platforms although it is vital to play the content.

How to get rid of the Resource Fork?

Thankfully it’s very easy to remove that annoying Resource Fork from a QuickTime file. You can simply open the file in today’s QuickTime player and do a “Save As”, QuickTime will reprocess the file correctly. The process is called “flattening” a movie file, it is very fast and will not re-compress your movie, this is very important to avoid re-compressing when possible.

Open your file in QuickTime and go to File, Save As..

.. and enter a new name

Of course, you will only need to do that if your file was created on a Mac, the free version of the QuickTime player is enough to do this and it works on all Mac OS X versions up to Snow Leopard with the latest QuickTime player. Now if you have lots of movie files to convert, you can of course look for some professional software that can queue and “flatten” a bunch of files from a directory.

FYI, here is a QuickTime file that has a Resource Fork (moov), inspected with a programming utility called Resorcerer:

And here is the same file that has been flattened:

How do I know what decoder is used?

In GrandVJ it’s very easy to see what player is used by the ArKaos engine when processing a file; simply hover your mouse over the cell that contains the video and the contextual help appears.

In this one, the video file with a resource fork will be played through QuickTime:

In this one, the same video flattened with the QuickTime player will go through the FFmpeg path:

(The same information is also available in GrandVJ for each movie listed in the visual library and in MediaMaster you can get that information from the file library window.)

Conclusion

During the first days of the release of GrandVJ 1.2 beta 3 with the new audio engine (which relies on FFmpeg) some users have complained that some old .mov files were not playing audio correctly with the beta version, this was most of the time caused by the Resource Fork and flattening the files fixed the problem. As a bonus, it will make your files cross platform.

This should give you some lights on what is happening when you drop a file on GrandVJ or MediaMaster, as you can see a lot of work is being done under the hood :)

We have optimized the Quartz Composer integration in GrandVJ 1.2 so that the compostions are rendered natively through the fastest available interface.

Finally, when using a composition as an effect the application will scan the published inputs and map them as effect parameters, if the format is compatible with the engine. GrandVJ can automatically map parameters that are float values, indexes, colors and booleans and there’s one visual input of the composition that receive the cell visual.

Now let’s have a quick look at how you can use Quartz compositions with GrandVJ..

Using compositions as sources

The quick way to use a composition as source is to import it just like you do with any movie, by drag and drop on a cell or by reaching your .qtz through the file browser. When doing so, GrandVJ will save the path to the .qtz file in your project as it does for any other visual.

For your convenience, GrandVJ installs a set of source compositions in “/Users/<username>/Library/Application Support/GrandVJ/Quartz Composer/Visuals”, these are listed in GrandVJ under the “Quartz-Composer (User)” section of the sources browser so that they are always available when you create a new project.

The idea is that you can also add your favorite source compositions there so that you always have them ready when you work on your projects.

Playing compositions as source does not allow modifying the variables from within GrandVJ.

Path for user compostions as Sources	User compositions as Sources

Using compositions as effects

System compositions as Effects

For effects, GrandVJ will add all the compositions from your system that are supported. It does it by automatically scanning the system folder “/System/Library/Compositions”. These compositions will appear in the application under the “Quartz-Composer (System)” section of the effects browser. You can use them as starting point to develop new effects. You can also add your own compositions as effects, just drop them in the local directory “/Users/<username>/Library/Application Support/GrandVJ/Quartz Composer/Effects”. They will appear in the “Quartz-Composer (User)” section of the effect browser.

Path for user compostions as Effects	User compositions as Effects

About the parameters

As a quick example, let’s have a look at the effect parameters from the system composition Image Hose. With the Quartz Composer application (found in Developer > Applications – you need to have installed Apple’s developer tools of course), open the file “/System/Library/Compositions/Image Hose.qtz” (ore better, open a copy) here’s what you should see:

Image Hose composition

Open the inspector (Editor > Show Inspector) and from the menu, look at the Published Inputs & Outputs, here’s what you have:

Image Hose composition parameters

In GrandVJ, the first 4 parameters that can be mapped automatically will be displayed. In this case it will be PrimaryColor, SecondaryColor, Pace and Size:

Quartz composition parameters in GrandVJ

As you can see, GrandVJ will only display the meaningful part of the parameter name and remove all _ProtocolInput strings from parameter names – this allows to easily display the full parameter name in the panel – it will also make things a little nicer by capitalizing the first letter of the parameter if necessary.

There are many resources on the web where you can find information and help about creating your own Quartz compositions, you can start by looking at Apple’s Developer Connection.

Note: you can download GrandVJ 1.2 beta 2 from our forums

The genlocking we refer to is the action of locking the frequency of a media to a reference signal or clock (check out the related wikipedia article for a complete explanation).

Let’s examine quickly the process of displaying frames. Roughly, 3 steps must be acheived:

1. get video frames from the disk
2. upload them to the graphic card
3. do the composition / blending of the layers and  present them to the display

Now let’s describe how these steps are achieved without genlock and with genlock:

Without genlock:

At each new monitor frame, the software finds if a new movie frame needs to be processed. If so, it starts working sequentially through the 3 phases. The problem is that the time necessary for a codec to read the data from the disk and convert it into a decompressed frame is not constant and might, in some cases, even be quite lengthy. Mutliply this by 8 (once for each layer), add composition and effecting to it and you end up with a process that is highly variable, and might sometimes take more time than a monitor frame (typically 1/60th second), which creates fluidity problems.

Let’s illustrate this process with a graphic to make it easier to understand; in a classical real-time video processing application this will work like this:

The above graph shows an application trying to play a video loop encoded at 30 fps on a monitor running at 60 fps. In a perfect world the application should present each frames of the movie exactly twice.

There are 2 problems with this “traditional” way of doing the video processing:

• The time base of the computer is different from the time base of the graphic card. This means that even an extremely powerful computer will display video with small hiccups once in a while because the time you use to compute movie frame is not exactly synchronized with the time it’s going to be shown, introducing a jitter effect.
• When the software start working to display a new frame it has only in this example 1/60th of second to read a video frame, upload it to the graphic card and display it. The available time depends on the monitor fps and not on the video source fps. Which means that, the higher the fps of the monitor, the more stress you give to your system.

With genlock:

The processing done by MediaMaster 1.1 is much more efficient. Three different engine accuracy modes are now available to the user in the preferences dialog: “Minimal” was the default mode in MediaMaster 1.0, “Frame Blending” another new option in version 1.1 which we will discuss later and “Buffered”, which is the subject of this article.

In Buffered mode, the software keeps a predictive buffer that stacks frames that will be needed in the future. As soon as a frame has been decoded, it evaluates what are the next frames that are going to be needed later and queues their decoding directly. This process is running in the background and is separated from the monitor lock needed to present frames without tearing. Whenever the software needs a frame to display, it takes it from the buffer that was prepared upfront.

Also, in buffered mode the content syncs with the monitor frequency rather than with the computer clock. As far as I know, very few other media players are capable of doing something as subtle as this.

So, if we keep the same timing as in the first example but adapt our graphic according to the way the buffered mode is working, here is the result:

Because the movie clock is genlocked to the monitor and because the software always have minimum one frame in advance, the movie plays with a perfect regularity and timing : 1 1 2 2 3 3 4 4 … each frame is played exactly twice.

Why is it so cool?

If you are still not too sure why it’s nice to be genlocked at 60 fps here is a real time recording of the output from MediaMaster running 2 layers (captured with Fraps). The lower layer with the speakers runs at 30 fps while the top layer is a pattern that runs at 60 fps, this one is part of our test loops, it’s a ramp that allows to see very easily if the system is perfectly genlocked.

Most importantly, when applying effects to your content these effects will be rendered at 60 fps so with a perfect synchronization, your eyes will have the feeling that everything is crisp and fluid.

In order to see the videos in this article you need to have QuickTime installed. Of course, ensure your monitor frequency is set to 60 fps and the video below should play perfectly smooth. It’s an original screen capture of the MediaMaster output in 60 fps, it has just been resized to fit in this article.

Now in order for you to be able to see the difference here is the same loop at 30 fps:

Let’s continue to lower down the fps, here’s the 20 fps version:

And finally the 15 fps version:

Try for yourself!

If you are curious, you can test this for yourself, don’t hesitate to stress MediaMaster 1.1 when it’s out but also why not compare and test with any other software video mixer.. For your convenience, you can download our 60 fps test loops as QuickTime movie files further below.

For best results you should let them loop for a while. In a VJ application you can simply add them as the latest layer of your composition with an addition blending and you will see if your system is powerful enough and your software correctly coded.

You can download MediaMaster from our web site.

When you are in the show industry and try to find out the best way to compress your content it’s still a little bit of black magic. While some codec are able to compress video very well they are heavy to handle for the machines, even more, the codec that does the best job at compressing while keeping a good quality such as H264 are very bad when you need to scratch your media.

The best codec for users that need to interact a lot with the content should allow to play forward and backward easily and should allow to jump into the content quickly.

Recently, I’ve heard about experiments with a file format designed by graphic card manufacturers and based on texture compression DXT1 DXT3 or DXT5 and I was wondering how hard it would be to add support for such a format to QuickTime through a new codec. Being programing for the QuickTime API since it’s version 1.0 beta I was considering the challenge fun and interesting.

I started by resuscitating an old sample code from 1999 found on the Apple web site and after a few hours of coding and testing I had an ArKaos codec generating video files that could be played back by QuickTime.

ArKaos codec in the QuickTime player

Then I jumped on the DXT texture compression problem and chose to use the squish library to handle the job. A few more hours later and my experimental codec was working with texture compression.

The codec when used by QuickTime at that point was not efficient because the texture decompression was done on the CPU, so there’s no advantage of using the GPU to handle that texture format.

To measure the speed up I used an experimental player I am working on, and patched ffmpeg to handle my new codec. This allowed me to leave the texture data untouched and pass it to the GPU via the OpenGL extensions that handle texture compression.

When this was done I could start enjoying the show and make some performance tests; in the table below the file size is in MB and CPU load is taken from running top in the console:

If you are curious about frame quality here is a crop from an original frame in Apple intermediate :

Original frame - Apple intermediate (click for full size)

Here is the same cropped frame using Dxt1 with texture compression:

HD frame - Dxt1 compressed (click for full size)

And here is the same cropped frame with Dxt5 compression:

Same frame with Dxt5 compression (click for full size)

So the results of these tests are that pure Dxt1 is the best format when it comes to CPU load, Dxt5 is better if you focus on quality but it generates files that are twice bigger. Actually, using Dxt1 I could have played 7 HD loops at the same time with my laptop … I mean, if hard drive’s data rate would’ve allowed it!

The problem with pure Dxt5 or even Dxt1 files is that they are huge, about 4 times bigger than photo jpeg or H264 encoded files. So my workaround was to use data compression on the CPU, since it’s almost unused!

To conclude, it looks like using Dxt1 with texture compression is the best solution to optimize both CPU and disk usage with a file size that is almost the same as Photo jpeg and a little bit bigger than H264. And the codec is still twice faster than any other codec I tested when using the CPU to decompress the texture data from disk.

On my venerable old MacBook pro I can’t smoothly play a loop in Full HD and even at 1280*720 my machine is still struggling. Thanks to Dxt1 with texture compression I can play two HD layers and three 720p layers.

Now I know you guys would like to play with this new toy so here’s a .dmg file with the test player I have used, along with a set of video loops. Beware that this dmg file is for Mac OS X x86 only (no Power PC): and here’s the link

You can be sure that we will bundle that new experimental codec in the next update of GrandVJ and MediaMaster, so everyone can enjoy it!

This is due to the fact that on a PC, the internal QuickTime decoding routines are not capable of multi-threading, which makes it impossible to decode frames in parallel. So even if one layer can run properly using QuickTime libraries, the more layer you pile up, the more the decoding bottleneck will show up.

On Mac OSX, QuickTime is now mostly multi-threaded decoding so it’s not such an issue but we recommend to try to use codec supported by our FFMPEG player in any case.

Important: make sure you understand we refer to the internal QuickTime player and NOT to QuickTime files in general. Some QuickTime files (for example, files based on mpeg/mpeg2/mpeg4/PhotoJPEG/Motion JPEG/H.264 codec) are played through the FFMPEG player.

To know what internal player is used for what file, use the information shown in the visual library:

Visuals library

In this case, both movies are QuickTime files. The first one is read through QuickTime because the Cinepak Codec is not handled by FFMPEG.It will play but it is not optimal.

The second is read through FFMPEG because it’s using H.264 codec, it is therefore better for performances.