As soon as I saw an early version of the music video shot in Ottowa for the single off his new “In a Cabin With” album, I thought; this should be interactive. So now three months later; here it is.

Read on for some behind-the-scenes technical details.

I started by making a prototype to see if it was possible to combine high-res h264 video video with 3D rendering. And thanks to Flash Player 10 it was! The most efficient way to render 360 panorama’s is to use cubic projection, as this method requires way less geometry (4 triangles at most at any one time) then cylindrical or spherical projection. In this case saving a lot of performance needed for the high-res h264 decoding. For the 3D rendering I used an early version of Papervision3D 3.0 for Flash Player 10.

Once I had the panorama video working, I thought that re-creating the YouTube player seemed like the best interface for this piece. Once the interface was done came the time to create the final video edit. I re-edited the two cylindrical panorama’s created by Steye, one of ouside and one of inside, into one cylindrical panorama which transitions between ouside and inside. The frames of that final cylindrical render were converted to cubic faces using the batch processor of CubicConverter. The resulting cubic faces were then comped back into one video render with the audio track added.

Encoding the final video files was the final hurdle to overcome as there were two aspects that seemed to be hard to unite into an acceptable end result, both quality-wise and performance-wise. The first was to stay within the bitrates currently concidered standard. And what better standard to take for online video delivery then YouTube. Normal YouTube video’s are streamed at around 700kbps, and their HD video at around 2000kbps. So those were my targets for the h264 encodings. However YouTube delivers acceptable quality within these bitrates with material of much lower resolution, then this video required (1024×728 for the low quality version and 1440×1024 for the HQ version). That’s because this video contains the video for all four faces (front, back, left and right).

So it was clear that I had to use the best h264 encoder out there to encode these files. So, off I went testing different h264 encoders from Apple, Sorenson, MainConcept, mencoder and x264, and different applications, each allowing access to different encoder settings. I tried Adobe Media Encoder, Quicktime, Sorenson Squeeze, Episode, Handbrake and ffmpeg. And I’m very glad, and also not surprised, to say that I achieved the best results by far with the open-source alternatives; ffmpeg using the Doom9′s x264 encoder. Ffmpeg allows you to use b-frames and CABAC which are the two features of h264 that really give a significant quality improvement. My findings were that the x264 encoder also preserved the color and contrast information best from all encoders (especially MainConcept in Squeeze resluted in a very high contrast and unacceptably dark video in my case), while surpisingly also requiring the least CPU upon decoding in the Flash Player. So, like I said; x264 rules them all at the moment. And if you need more tweaking of the encoder, x264′s command-line interface gives you access to all the parameters that really make h264 shine.

I hope you enjoy the end result. Happy Holidays!

Below a diagram describing the footage from the two photo shoots.

You have a number of options displaying those images in a website; Loading JPEGs individually, forget about it! You could embed the images on the timeline in an SWF. This really isn’t a very good option as your memory usage quickly grows beyond acceptable norms and really doesn’t perform very well due to the JPEG decompression it needs to do.

No, video is your best option. FLV embedded in an SWF with every frame encoded as a keyframe, that is. It handles memory usage beautifully and allows you to smoothly jump to any frame backwards or forwards.

Ok, so back to our diagram. Add to it the portfolio cases that need to be suspended around us and be masked when falling behind us and the obvious setup would be to have one video with the background and one video with an alpha channel of us and use the latter to mask the cases behind us.

But the problem with that is that video, especially at this resolution, is quite cpu intensive. So in the case of viewing it in anaglyph it would mean you end up with a total of four videos on stage and two of those with alpha channels, which makes them even heavier. The solution we opted for is to combine us and the background into a single video, leaving only two videos on stage when viewing in anaglyph and no alpha channels.

But then how do you achieve the masking of the cases? Well, we also exported PCT image sequences of the alpha channel from us keyed out. Those 144 PCT files per side were imported onto the timeline in the Flash IDE and converted into vector shapes using the Trace Bitmap function. To avoid RSI we automated this process by creating an JSFL script that loops through the timeline and traces every Bitmap it comes across. The result is two SWFs, one for left and one for right, with vector shapes defining the masking area for each single frame in the video. Click on the image below to see the left mask SWF used in the website. This method is way more performant and smaller in file size than encoding the images into JPEGs.

Each of the cases have a separate RenderLayer in Papervision3D. The mask SWF is also placed on its own RenderLayer with BlendMode set to Erase. Set the correct z of your mask layer and voila, you’ve got your masking.

In an attempt to reduce file size for the video we came up with the following idea; FLV is able to contain 4 channels; Red, Green, Blue and Alpha. When you view the website in normal mode, so not in anaglyph, the Red, Green and Blue channels of the right video are used. So that’s 3 channels, right? When viewing the website in anaglyph you use the same Green and Blue channels of the right video and only the Red channel of the left video. So while the left video contains all 3 channels, we only need one; the Red channel. In other words we only need a total of 4 channels; RGB of right and R of left. Meaning we could store those channels in a single FLV by encoding the RGB channels of right into their appropriate channels and encode the R channel of left into the unused Alpha channel.

This can easily be done in After Effects. Just make sure you encode the video using Straight Alpha and not Pre-multiplied Alpha. In After Effects this can only be set when using the Render Queue instead of simply using the Export Movie option.

Ok, so now we have just one FLV with all the data we need to render both normal view and anaglyph view. Great, right? Well, not exactly. The On2 codec really isn’t made with this use in mind. It uses most of the available bytes to encode the areas on the color channels that are going to be visible according to the alpha channel. Which really is a good thing when using the alpha channel for what it is supposed to be used for. Why spend precious bytes to nicely encode parts of the video that aren’t going to be visible anyway? But that kinda ruined our little plan.

Besides the poor encoding quality, this method would require to copy the necessary channels from the video onto a canvas. Verdict; much lower quality and less performant. So we went back to the two FLV setup. Shame for the redundant data being loaded but at least it performs well and looks good.

To keep video file size low, minimizing loading time, we started with a low bit rate video to be used when animating the panorama. As soon as you would stop scrolling a high quality JPEG was loaded and placed on top of the video. However, this caused a short freeze in the physics when you stopped scrolling. This was caused by Flash using all the cpu resources to decompress the JPEG into a bitmap once loaded. Also the transition between the low quality video and high quality JPEG was annoyingly obvious.

Since we couldn’t find a work around for the freeze and couldn’t live with it we decided to get rid of the JPEG stills. But that meant that we had to use way higher bit rates to encode the video to get acceptable quality at that resolution (1376×576). Once we found the appropriate bit rate and encoding settings we ended up with a video of roughly 16Mb per side! So we had to find a way to reduce the preloading time required to enter the panorama view.

The solution we came up with is illustrated in the following diagram:

We split the 144 frames circle into three separate SWFs. Requiring only the first set of 36 frames to be loaded before being able to grant access to the panorama view. While you can start exploring the panorama the second and third set continue loading in the background improving the smoothness of the turn as they load.

This solution reduced the required preloading time back to acceptable norms. To further reduce the annoyance of the wait we decided to add a little game to the preloader. Maybe more on that in a later post.