As soon as I saw an early version of the music video shot in Ottowa for the single off his new “In a Cabin With” album, I thought; this should be interactive. So now three months later; here it is.

Read on for some behind-the-scenes technical details.

I started by making a prototype to see if it was possible to combine high-res h264 video video with 3D rendering. And thanks to Flash Player 10 it was! The most efficient way to render 360 panorama’s is to use cubic projection, as this method requires way less geometry (4 triangles at most at any one time) then cylindrical or spherical projection. In this case saving a lot of performance needed for the high-res h264 decoding. For the 3D rendering I used an early version of Papervision3D 3.0 for Flash Player 10.

Once I had the panorama video working, I thought that re-creating the YouTube player seemed like the best interface for this piece. Once the interface was done came the time to create the final video edit. I re-edited the two cylindrical panorama’s created by Steye, one of ouside and one of inside, into one cylindrical panorama which transitions between ouside and inside. The frames of that final cylindrical render were converted to cubic faces using the batch processor of CubicConverter. The resulting cubic faces were then comped back into one video render with the audio track added.

Encoding the final video files was the final hurdle to overcome as there were two aspects that seemed to be hard to unite into an acceptable end result, both quality-wise and performance-wise. The first was to stay within the bitrates currently concidered standard. And what better standard to take for online video delivery then YouTube. Normal YouTube video’s are streamed at around 700kbps, and their HD video at around 2000kbps. So those were my targets for the h264 encodings. However YouTube delivers acceptable quality within these bitrates with material of much lower resolution, then this video required (1024×728 for the low quality version and 1440×1024 for the HQ version). That’s because this video contains the video for all four faces (front, back, left and right).

So it was clear that I had to use the best h264 encoder out there to encode these files. So, off I went testing different h264 encoders from Apple, Sorenson, MainConcept, mencoder and x264, and different applications, each allowing access to different encoder settings. I tried Adobe Media Encoder, Quicktime, Sorenson Squeeze, Episode, Handbrake and ffmpeg. And I’m very glad, and also not surprised, to say that I achieved the best results by far with the open-source alternatives; ffmpeg using the Doom9′s x264 encoder. Ffmpeg allows you to use b-frames and CABAC which are the two features of h264 that really give a significant quality improvement. My findings were that the x264 encoder also preserved the color and contrast information best from all encoders (especially MainConcept in Squeeze resluted in a very high contrast and unacceptably dark video in my case), while surpisingly also requiring the least CPU upon decoding in the Flash Player. So, like I said; x264 rules them all at the moment. And if you need more tweaking of the encoder, x264′s command-line interface gives you access to all the parameters that really make h264 shine.

I hope you enjoy the end result. Happy Holidays!

I’m proud to announce that our first game in collaboration with Hanazuki has just been launched. The game plays an important role in an installation that Hanazuki did for the Cinekid festival.

Working with the internet legend Niko Stumpo has been a delight, so we definitely are not going to stop at one game, we are already cooking up the next one

Click here to play the Fattybumbum game.

This demo was made shortly after the initial release of Jiglib. It was a study of how far you can push realism in the sence of rendering and animation. Here CompositeMaterial is used with a BitmapMaterial and a GouraudMaterial. This is much faster then using a ShadedMaterial. All you need to do is modify the GouraudMaterial to use a transparent light map. Andy Zupko’s ShadowCaster is used for casting the die shadows. Jiglib is used for the 3D physics, also slightly modified to facilitate fixed timestep. Unfortunatelly the end result is still far from the desired 60fps. Hit spacebar too see the stats, check it out.

UPDATE: Something troubled me when checking the performance in the browser with Flash Player 10 compared to Flash Player 9. On my machine FP9 ran at approx 40fps but FP10 hardly reached 25fps. Changing the wmode from “direct” to “normal” fixed the performance issue in FP10. Especailly in Firefox the performance difference is significant. Still on my machine FP9 seems to perform slightly better then FP10. Mental note: next time take more care in setting the proper wmode.

We are proud to announce the launch of the portfolio website we created for the London-based design and art direction studio Wallzo. No flashy 3D stuff this time, but a clean, sober browsing experience. Have a look.

While playing with FlickrAPI, we came up with a cool way of exploring the Flickr interestingness. It quickly became an addiction. You can view it here, we hope you will enjoy it as much as we do.

It seems that these days everyone is working on extrusion of some sort so I thought I post my experiments before it’s going to become yesterdays news

So what’s the big deal with extrusion? Well in order to do it well we first need to triangulate our vector shapes, not an easy feat in the first place but it gets really tricky when your shapes contain holes, I first used the “ear-cutting” method and then try cut out the holes form the triangulated shapes. That unfortunately created huge excess of vertices and triangles and before I could figure out how to fix it, Tim Knip has posted this fantastic method in alchemy form that triangulates polys with holes without creating any extra vertices! All I had to do I to find a way how to tell what is a hole and what isn’t, again it happend to be harder than it seems but my buggy code succeeds 90% of the time

Check out these examples:
example1, example2
you can download the code the but be warned it’s not in a very good condition.

Others have been exploring this subject as well, you can find their experiments here and here and here.

What if Joseph and Mary had access to faster modes of transportation when seeking a place to give birth? And what would have happened if they had taken a wrong turn leading them away from Bethlehem? Those are questions that have been occupying the minds of theologists for centuries. Finally, with the help of today’s latest 3D technology, Y & R, Flashfabriek and Barcinski & Jeanjean have tried to answer those questions. Have a look.

Merry Christmas and a happy new year!

Mark & Adrien

There are some big changes happening to vectorvision due to the fact that we are adding triangulation for the fonts, I will write more on it in other post. Other stuff recently added to vectorvision is:
word3d is now text3d and as you would expect form text it now supports multiline, align, letter-spacing, and line-spacing and all can be changed on the fly. see example.

I also have been experimenting with scalable outlines and the results are already very interesting, see this example.

The source code can be found in the vectorvision repository.

Below a diagram describing the footage from the two photo shoots.

You have a number of options displaying those images in a website; Loading JPEGs individually, forget about it! You could embed the images on the timeline in an SWF. This really isn’t a very good option as your memory usage quickly grows beyond acceptable norms and really doesn’t perform very well due to the JPEG decompression it needs to do.

No, video is your best option. FLV embedded in an SWF with every frame encoded as a keyframe, that is. It handles memory usage beautifully and allows you to smoothly jump to any frame backwards or forwards.

Ok, so back to our diagram. Add to it the portfolio cases that need to be suspended around us and be masked when falling behind us and the obvious setup would be to have one video with the background and one video with an alpha channel of us and use the latter to mask the cases behind us.

But the problem with that is that video, especially at this resolution, is quite cpu intensive. So in the case of viewing it in anaglyph it would mean you end up with a total of four videos on stage and two of those with alpha channels, which makes them even heavier. The solution we opted for is to combine us and the background into a single video, leaving only two videos on stage when viewing in anaglyph and no alpha channels.

But then how do you achieve the masking of the cases? Well, we also exported PCT image sequences of the alpha channel from us keyed out. Those 144 PCT files per side were imported onto the timeline in the Flash IDE and converted into vector shapes using the Trace Bitmap function. To avoid RSI we automated this process by creating an JSFL script that loops through the timeline and traces every Bitmap it comes across. The result is two SWFs, one for left and one for right, with vector shapes defining the masking area for each single frame in the video. Click on the image below to see the left mask SWF used in the website. This method is way more performant and smaller in file size than encoding the images into JPEGs.

Each of the cases have a separate RenderLayer in Papervision3D. The mask SWF is also placed on its own RenderLayer with BlendMode set to Erase. Set the correct z of your mask layer and voila, you’ve got your masking.

In an attempt to reduce file size for the video we came up with the following idea; FLV is able to contain 4 channels; Red, Green, Blue and Alpha. When you view the website in normal mode, so not in anaglyph, the Red, Green and Blue channels of the right video are used. So that’s 3 channels, right? When viewing the website in anaglyph you use the same Green and Blue channels of the right video and only the Red channel of the left video. So while the left video contains all 3 channels, we only need one; the Red channel. In other words we only need a total of 4 channels; RGB of right and R of left. Meaning we could store those channels in a single FLV by encoding the RGB channels of right into their appropriate channels and encode the R channel of left into the unused Alpha channel.

This can easily be done in After Effects. Just make sure you encode the video using Straight Alpha and not Pre-multiplied Alpha. In After Effects this can only be set when using the Render Queue instead of simply using the Export Movie option.

Ok, so now we have just one FLV with all the data we need to render both normal view and anaglyph view. Great, right? Well, not exactly. The On2 codec really isn’t made with this use in mind. It uses most of the available bytes to encode the areas on the color channels that are going to be visible according to the alpha channel. Which really is a good thing when using the alpha channel for what it is supposed to be used for. Why spend precious bytes to nicely encode parts of the video that aren’t going to be visible anyway? But that kinda ruined our little plan.

Besides the poor encoding quality, this method would require to copy the necessary channels from the video onto a canvas. Verdict; much lower quality and less performant. So we went back to the two FLV setup. Shame for the redundant data being loaded but at least it performs well and looks good.

To keep video file size low, minimizing loading time, we started with a low bit rate video to be used when animating the panorama. As soon as you would stop scrolling a high quality JPEG was loaded and placed on top of the video. However, this caused a short freeze in the physics when you stopped scrolling. This was caused by Flash using all the cpu resources to decompress the JPEG into a bitmap once loaded. Also the transition between the low quality video and high quality JPEG was annoyingly obvious.

Since we couldn’t find a work around for the freeze and couldn’t live with it we decided to get rid of the JPEG stills. But that meant that we had to use way higher bit rates to encode the video to get acceptable quality at that resolution (1376×576). Once we found the appropriate bit rate and encoding settings we ended up with a video of roughly 16Mb per side! So we had to find a way to reduce the preloading time required to enter the panorama view.

The solution we came up with is illustrated in the following diagram:

We split the 144 frames circle into three separate SWFs. Requiring only the first set of 36 frames to be loaded before being able to grant access to the panorama view. While you can start exploring the panorama the second and third set continue loading in the background improving the smoothness of the turn as they load.

This solution reduced the required preloading time back to acceptable norms. To further reduce the annoyance of the wait we decided to add a little game to the preloader. Maybe more on that in a later post.

In the previous article we talked about how we shot our footage in stereo. Now we are going to talk about how to render those stereo images for 3D glasses. And how to render Papervision3D for 3D glasses.

There are many techniques for displaying stereoscopic 3D images. We opted for Anaglyph, the one you need the red/cyan glasses to view, since those are the glasses people are most likely to have laying around somewhere. Also these are the least expensive glasses of the lot and that mattered since we were also going to do a mailing of Barcinski & Jeanjean 3D glasses.

In short, an anaglyph image is obtained by taking the red color channel from the left camera and combining it with the blue & green color channels from the right camera. The drawback of this technique however, is the loss of color information, mostly affecting the red color component. So that was an issue we had to take into account as to what colors we would use in the website. Black doesn’t suffer in any way by anaglyph, so that kind of pushed us towards a basic graphic black & white design for the intro and menu cube.

On to rendering Papervision3D in anaglyph! For our website we have created an AnaglyphView class which offers anaglyph renedering in Papervision3D with the ease of use of the BasicView class. Click on the image below to view the demo. Toggle anaglyph rendering on and off by clicking on the Stage.

The AnaglyphView has one camera and two viewports. For each frame it moves the camera to the left, renders the camera projection to the left viewport, then moves the camera to the right, renders the camera projection to the right viewport and finally moves the camera back to it’s initial position. This way you still have complete freedom of camera movement in between the renders. Best is to use a Target Camera and move the target object to the focus point, creating the same mechanics as our eyes when focusing.

The left and right viewports are combined using ColorTransforms and BlendModes. While this offers a fast way of combining the two channels to an anaglyph image it is important to note that the viewports will not render correctly when not completely opaque. That’s why the AnaglyphView uses extended Viewport3D classes that make sure the viewports have a background filled with the supplied background color.

To combine 2D stereo images with 3D stereo rendering you need to attach the 2D stereo images to their respective viewports. That’s what we did in our website. The left panorama video is added to the DisplayList of the left viewport and the right video to the DisplayList of the right viewport.

Needless to say stereo rendering in Papervision3D can be very heavy. Your aim would be to keep the scene as simple is possible and limit the redraw regions as much as possible. We’ll be going into some optimization tricks in our article covering the Papervision3D side of our website.

Feel free to download the AnaglyphView class and start playing with it. The example above is also included in the package.

Earlier this month we created some visuals for a social RFID installation. More details, photos and videos here.

Well, here it is. After 3 months of hard work fun our “Coming Soon” page has finally been replaced by a brand new website. The coming weeks we will be posting a series of “Making of” articles about our use of Papervision3D, Vectorvision, 3D Physics, Photography and more. Today a few lines on the general concept of the website. 

The vision for the panorama view came to us in november last year while we were brainstorming on another project (the one for which we prototyped a spherical video panorama). We would stand in one of Amsterdam’s many beautiful scenes with everybody around us walking by going about their business. Around us our portfolio work would be hanging in the air in the form of paper flyers. Envisioning viewing that with 3D glasses being able to move around us, filled us with excitement. We are already familiar with viewing photo’s and movies with 3D glasses, but I can’t really say I had had an interactive experience using 3D glasses before. 

By creating a 360º panorama from a stop motion video rather than from multiple stills stitched together, allowed us to capture the movement of the subjects in the scene. The price to pay however was much bigger file size. To make up for the wait while the video content preloads before being granted access to the panorama view we thought it would be cool to revamp the traditional Pong waiting game into a solitary 3D version that could also benefit from viewing with 3D glasses.

But the added file size was worth it. When we first reached the point in development that we had a placeholder 360º video panorama overlaid with a badly keyed-out test shoot of us taken in from of a purple screen (yes, we hadn’t ordered the green screen yet), that we realized just how interesting it actually is to gently turn the 360º and observe the many people, boats and occasional “motor agent”, as they call them here in Holland, pass by.

The location was almost an automatic choice. Besides the fact that it’s beautiful, it’s right around the corner from our studio. But this is mearly the first panorama to feature on the site. As it’s simply a question of modifying some XML to change the panorama in the website, and, well, shooting the new footage of course, we plan on adding new panoramas to the site in the future, probably starting with a new shoot this fall.

This little experiment is a proof of concept of fog using papervision’s render layers and the color transform object. This would make a lot of sense for the frustum camera where the culled triangles wouldn’t just pop out of void but would nicely fade in. Also to my surprise there is almost no performance penalty for the fog.

Take a look here or download the source.
Other interesting effect is using alphas on the render layers.