Matthew Hope

Games + VR Consulting

[themlhope]

Games from scratch

I starting getting into original Gameboy hardware customization and homebrew development earlier this year. While completely unrelated to my everyday work, finding a busted up DMG, repairing it, refurbishing it and then building a ROM that ran on the original hardware was one of the most fulfilling development-related experiences I’ve ever had. It was humbling, like putting together a piece of IKEA furniture or baking a cake for the first time.

Bought a ¥1500 DMG off of Yahoo! Auctions Japan. The poster mentioned that it needed the LCD repaired as it had the ever so common “dead pixel columns” issues. Fixed this with by carefully re-soldering the ribbon cable contacts. 

I also installed an LED backlight while I was in there. Had to learn some soldering skills along the way. Here’s the beast in action, pre-refurbishment. The game I built myself in C using Zal0′s amazing update of GBDK, ZGB, written to a blank ROM cart.

I turned out that the power regulator on my DMG was an older model that couldn’t handle the backlight. Later models of the Gameboy had improved power regulators to handle larger, more power-hungry cartridges like the Pokemon games. I found one of these and sourced it for my Frankenboy.

Cleaning the cartridge contacts made this into a daily driver pocket gaming machine.

Last but not least I ordered up a third-party glass replacement screen since the one that came with my Yahoo! Auction unit was all scuffed up and hazy. 

[themlhope]

VR storyboarding: a naive approach

Recently I've had a chance to talk to other people in the XR space about the work I did on Tales of the Wedding Rings VR (Oculus/Steam) and it's had me reflecting on the planning processes we used. When I joined the team the initial prototype had already been finished and we were starting up production with a fairly clean slate. Despite this, however, there wasn't an established method of documenting direction, so I decided to design one and authored the first drafts of the production. 

Here’s a sample:

I called this a "storygraph" instead of a storyboard, as it was less granular in terms of its representation of time and was really designed to plot out the action occurring in the various spaces of our experience rather than specifically inform the animation. Time is represented on the X-axis; it’s read from left to right. The rows or "tracks" which are arranged vertically represent the different spaces, and were assembled from clippings from the source material, hand renderings and screenshots from the Unreal editor. The tracks contained the following:

track 1 : Images from the source material to serve as a rough timeline track 2: The user space as seen from above to orient spatial relationships track 3: The ideal user field of vision per our design intent. tracks 4~6 : The action within each manga panel, in/out transitions, etc.

These was printed out into strips which I would arrange in my cubicle and sometimes the enclosure of black dividers which served as our dedicated VR testing space. The reasons for which I consider this method naive are probably clear to anyone who has worked in VR: 1.) It isn't VR native, therefore it fails to express some of the medium's particulars like depth and 360 degree immersion. 2.) It requires special knowledge, meaning that it isn't intuitive for new members in the team to "just pick up." You have to learn how to "read" the document. 3.) It lacks fine timing details which would otherwise be helpful to animators and other staff. 4.) It paints a deceptive picture of the user experience in that it can lead team members to believe that we have control over the user's field of vision at any given time. We typically don’t. The purpose of the document was to record what the user could see and when and in what order, however it was really just a way for myself to keep track of the director's desired shots. I proposed using Oculus's Quill to produce VR storyboards (which, as it turns out, is the direction that Disney's Cycles team took) but it was thought that we didn't have the resources required to take on what we thought was such a specialized workflow. That said, animators ended up using the storygraphs for the initial layouts after all because: 1.) It were easy to iterate on when changes from the director came in. 2.) It gave a holistic view of an entire scene, whereas a 360 degree VR storyboard might require linear viewing. These kinds of productions definitely require more VR-native workflows, but if adopting or developing such workflows isn't an option for your team you can still get by in the analogue world.

Just for fun, here is one of my very first attempts -- my apologies to anyone who actually reads Japanese:

------------------------ *結婚指輪物語 clippings belong to Square-Enix with characters and stories created by Maybe. They are represented in this article for reference only.

[themlhope]

Notes on sequential art in VR

Here's some of the thoughts I shared with my team as Feature Developments Lead on my last project. It was a VR narrative piece that was released on the Oculus Store in June 2018.

The format below wasn't strictly followed but drove a lot of the other design documents I authored throughout the project, including VR storyboards, feature specifications and animatics review notes.

Note: it was copy and pasted from Google Docs, so please excuse the broken formatting.

-----------------------------

Overview

Essential Experience

[VR comics] is a virtual reality experience where users are immersed in comics narratives. The story is conveyed through scenes acted out by sequential art and animated characters accompanied by fully voiced dialogue, sound effects and dramatic music scores.

As an interpretation of the comics medium, the main mode of storytelling is sequential art, or a series of visual elements arranged in a deliberate sequence. One example of such a visual element are 3D comics panels, or “Live Windows”. These Live Windows can be accompanied by speech bubble and onomatopoeia visual effects which materialize in the VR space to deliver the comics experience's essential literary element.

Users are invited to peer into the diorama-like Live Windows and at times even literally step into the characters' world to experience the scene from a first-person point of view.

As the user views each scene, they will be unlocked and made available in a a scene select hub. The user will thereafter be able to experience any unlocked scene at their convenience.

What it isn’t:

While [VR comics] borrows conventions from cinema and animation, it is not a VR short film like Henry or Dear Angelica where the user must sit and watch the entire production from front to back. Linear animated sequences will be leveraged to express things that only motion can express, but they are only one element of the experience.

What it is:

Rather than relying entirely on temporal pacing, pace can also be established by the speed by which a user navigates their gaze across compositions purposefully arranged in the VR space. This is similar to how a comics artist controls pacing by arranging their panels on a page. Consider this example:

An image like the one above may be interpreted as a single moment in time.

However, we can guide the audience’s sense of time by introducing visual elements which are consumed sequentially across space. The panel above accomplishes this using dialogue bubbles.

Another way we might illustrate the concept of space as time would be to imagine a tourist making their way through a Hindu temple or an ornate cathedral whose bas reliefs and stained glass windows come to life, depicting a sequence of events:

II. Look and Feel

Characters and environments can be represented as 3D models illustratively rendered in comics's hallmark monochrome aesthetic. If a character were to stand perfectly still the image would be indistinguishable from a hand-rendered ink illustration. The narrative should feel fluid, with visual and aural cues guiding the user's attention through living images artistically arranged in the VR space.

III. Spaces

At times the user should have the experience of reading an animated comics whose panels fill the VR space; at times the user should also feel as though they have literally stepped into the comics world. Regardless, the experience of traversing between these two spaces must feel clear and deliberate yet seamless and unobtrusive.

User Space

In this space the user interacts with the comics medium itself rather than the world represented in the narrative. These interactions are abstract rather than emulative, such as advancing the narrative’s progress by turning a virtual page, choosing between two branches in the narrative, or bookmarking the user’s progress. The space must be distinct from others by a characteristic aesthetic -- a monochrome post-process, for instance.

Comics Space

The comics space is the world which the comics characters occupy. The user can enter this space by being enveloped by a Live Window for example. This space is distinguished by the presence of characters and environments from the narrative. The comics space should also be somehow visually distinct, if not only subtly, from the user space: a subtle duotone post-process, for instance.

IV. Features

Pillar Features

Immersion and presence

The experience of stepping into the comics

Vivid three-hundred sixty degree 3D environments

Intuitively connected comics and cinematic narrative elements.

comics / sequential art storytelling

Narrative delivered with sequential art compositions

comics-like reading experience with speech bubbles & onomatopoeia

Narrative rhythm is directed by purposeful, sequential arrangement of compositions in VR space.

Supporting Features

Content navigation supported by utilitarian interaction

Intuitive motion controller input

Scene navigation

Scene playback features (pause, skip)

Replay value added by allowing users to experience the content in different ways

Discoverable content spread across the VR space during playback.

Additional modes of playback, experience

User gaze dependent playback elements

Branching content, optional scenes

Cinematic storytelling

Narrative supported by the animation of 3D subjects

Deliberate, authored pacing

Controlled viewing experience

Narrative rhythm is established by carefully timed animation sequences

Exotic Features

Immersive elements enhanced by novel interaction

Character acknowledgment of user presence by reacting to user gaze

Grasping and moving panels and speech bubbles with motion controllers

Grasping and moving small props in the comics space

Non-features (“Things We Won’t Do”)

Exposition not reflected in the source material

Original material should support the essential experience in a meaningful way (storytelling grammars) or be warranted by necessity of the platform (VR acclimation grammars).

Animation for animation’s sake

Animation should be used when direction wishes to express something that only animation can express.

------------------------

*Panels from Scott McCloud’s Understanding Comics are represented here for reference only. It’s a great book, and anyone with interest in sequential art should go buy it along with McCloud’s other texts.