Mind blown

A little over a year ago I stood in a lab at the Oslo School of Architecture and Design (AHO) and put my head into a pre-release HTC Vive HMD. I stepped into a concept design for an innovative ship bridge that was part of a research project at AHO. I must admit that the real-time 3D model of the bridge looked less than impressive. But then my jaw dropped as a person stepped into view inside the virtual space and started talking to me like the most natural thing in the world. That person was Dr. Kjetil Nordby, an industrial designer and researcher at AHO who headed up the project. He explained to me that they needed a solution to have the designers be able to communicate with the stake holders that were inside the VR experience.

Eerie but not eerie

To solve his problem Kjetil hired a developer, Stian Børresen whom he had previously worked with to create a VR capture and play back system for human avatars. It had to be real-time and it had to be cost effective. While the virtual representation of Kjetil was not pixel perfect, there wasn't really any uncanny valley and the playback was near instantaneous. As most will know the uncanny valley is the hypothesis that human replicas which appear almost, but not exactly, like real human beings elicit feelings of eeriness.

I have a fairly good knowledge of the efforts needed to represent a real person in CG on film and now had a hard time understanding how it could be that a VR PC with a Kinect V2 could drive such a realistic human character in real-time. It took me only a couple of seconds to decide that I wanted to be involved in a project using this technology if at all possible. It was a magical, life-changing experience that helped me focus on what I wanted to do next in VR - create virtual architectural spaces that supported a pure VR experience.

I Know Kung Fu

Almost exactly a year later I find myself at VRLA 2017 in Los Angeles in front of the booth of HoloCap, the newly established company that is behind the magic I saw a glimpse of that day in March 2016. HoloCap is led by developer Stian Børresen who nearly 20 years ago did his first attempts of creating virtual humans in CG using Unreal Engine version 1.0 (!), a camera and a green-screen. The demo at VRLA is the evolution of a project that started with an email to Stian where I offered him to design a virtual room, an architectural space for his tech demo.  Stians references were solid right off the bat - the Holodeck from Star Trek anno 1974 and the dojo from the 1999 Matrix movie where Neo fights Morpheus. After some creative sparring we decided on a smaller dojo space with a modern look, taking inspiration from both films and crafting an original design in which the technology could be demonstrated with the capture of the award winning Norwegian Wushu artist, Kim Gibson.

Demo or die

We were extremely optimistic and naive in that we believed we could pull off the production of the demo in a matter of a few weeks. In reality as we came to realise, things take more time. The benefit of more time enabled me to revise the design of space, using IrisVR and Prospect to evaluate the space in VR and make changes to improve the experience of the space. I was lucky to get help from my friend Yuan Wu, a Chinese graphic designer and illustrator who was able to identify artwork and calligraphy that referenced wushu. In the Matrix movie the dojo looks very much Japanese which is ironic since Neo awakens to having learned Kung Fu, which is a Chinese martial art. My solution was creating a fusion of Japanese and western design and infuse it with Chinese art. When I found the Wushu Manual by US artist Albert Gea I was sold, a perfect fit for the sliding doors in the back of the room.

Sound advice

We did not finish the demo that summer as planned, but we now had a new deadline for the Digital Storytelling VR event which I was organising with Angela Amoroso and Eric Hanson and putting up in collaboration with the Norwegian Film Institute. As the dojo started looking better and better our ambitions for the demo grew. Stian wanted to improve the quality of the play back of the captures and I wanted VR sound design. Enter sound designer Gullik Gulliksen and Dag von Bunkholt whom helped creating an environmental sound design for the space and spatialized Gibson Kim's performance to fit the dojo space.

In unity we stand

The original code for the HoloCap software was written to work in Unity, therefore it made sense to create the IKKF-VR demo in the same engine rather than Unreal which might have enabled a better looking space. With the diffuse lighting concept for the dojo I opted for a baked texture solution that resulted in lighting and materials becoming one. I also enlisted the help of 3D artist and former student Solveig Stake whom took my 3ds Max scene, baked the maps and created the IKKF-VR Unity scene. While this approach would not result in the highest form of realism it would be a good match for the HoloCap characters as well as allow the scene to be converted for lower performing platforms if we ever were to create a Gear VR or mobile VR version of the demo.

The place to be

The response to the demo at VRLA 2017 was by all accounts fenomenal, resulting in hundreds of connections for HoloCap. While I am in no way unbiased, the demo was one of a handfull that were breaking new grounds. While the interest in attending expos such as the traditional annual SIGGRAPH has steadily diminished since the hey days of the 90s much thanks to the web, Virtual Reality require you to be present in person to experience the technology. Los Angeles seems also to have become *the* place for VR development, some speculate because of the recent demise of the many VFX studios with an abundance of talented VFX artists and developers looking for new job opportunities. At the moment there is no place I would rather be than in LA for doing VR, but there is a good chance we manage to create spectacular VR experiences here in Norway too. We do have cutting edge tech as seen with HoloCap and there is a growing number of talented 3D artists and developers that know how to create great looking virtual spaces, and we will just have to learn by doing how to create great VR experience. To boldly go where no man — where no one — has gone… before.

For more on the I Know Kung Fu VR project see the project page.