Here is the collection of what I’ve worked on in AR, VR, ranging from mobile, HMD, to projections. Recent years my work at Unity focused more in the 3D tooling space and I decided to use this page as a self-record, not one that is polished for a public design portfolio page. Feel free to dig around, there is still a lot of interesting ideas that I wished I could have more time and proper resource to work on.

Annotate All!

A Perspective Preserved Annotation System for Asynchronous Collaboration based on Head Mounted Display Augmented Reality

Augmented Reality

1. It superimposes virtual content over the real-world environment.

2. It is interactive, cameras are commonly used to gather information about the surroundings of the user and the data become digitally manipulatable by computer vision.

3. It is registered in three dimensions.

AR Headset

Core Questions

Before jump in to the system, four questions need to be answer to form a clearer picture of why you should care.

1. Why annotation?
   Annotation is a fundamental technique for sharing information. We leave notes, illustrations, signs to guide others through an intricate sequence of information such as operating a machine or familiarizing a new environment. It is the essential way to meaningful content creation which is the fundamental of the success of a new media.

2. Why asynchronous?
   Asynchronous interaction allows us to share information across timespan and it makes meaningful interaction scalable. It is so common to see asynchronous annotation in mixed reality games and professional training softwares.

3. Why preserve perspective?
   In annotations, angle of vision information help foster clearer communication. For example, a technician might be looking at the indicator light above the valve that he is operating. If we were on the side of the technician, we would naturally follow the angle of view of this technician and learn that the indicator light is related to the valve. On the other hand, if this kind of interaction is recorded without preserving the technician’s perspective, it is difficult for viewers to link those details together.

4. Why now?
   Developer-ready headsets armed with powerful sensors are becoming more accessible than ever, i. Magic Leap One and Microsoft Hololens. In the past, we annotate physically in real world, digitally in virtual world. We learn to interact with digital information by haptic input such as keyboard, mouse clicks, and touches on devices. Then there was the rise of voice interface smart home products that requires less learning, we simply need to speak to perform tasks. Although none of the above is perfectly intuitive, we see a trend of us human want to interact “naturally.”
   Now, we are grant with these headsets to explore new ways to interact with the world. By combining our gestures, voices, views, and other inputs sources,  we are able to see virtual content overlaying on to the real world immersively, and we could use either barehands or controller to interact with digital content intuitively. It opens up new ways of interaction design possibilities.

System overview

Here’s an example, a laboratory need to train a new research assistant everyday for an entire year? You attach some sticky notes on the arm and record a instruction video? Nope, you use are system! Instructor creates annotations and record voice with the AR headset right on the robot arm, and then the research assistants wear the headset to view the instructions asynchronously. Problem solved!

Conclusion

My work demonstrates a virtual annotation method for asynchronously sharing information about the real world.the annotation system represents a step forward in allowing non-technical users to create visual and audio annotations without prior training. Viewers can easily review the recorded drawings, head poses, and fixation points by operating the simple playback control of each annotation mark. Furthermore, viewers can move freely in the real-world space to view the recorded annotations in different angle. We see our work as a stepping stone toward more complicated content creating tools in the mixed reality space. Some use cases are:

Further study may include user testing on how effective this system is compare over existing non-mixed reality, mobile AR, or VR annotation system. In addition, detail interface design could be improved to create a more intuitive annotation experience.

VR Keyboard Design

This is an experiment of testing out and figuring out what I considered the most comfortable keyboard input at the time. Researching a range of commont techniques of keyboard in VR space and a set of layout and input ways such as the drumsticks, the finger taps, the physical keyboard. Prototyped in Unity, multi-person available (Just the not left by the keyboard is visible by others).

Mobile AR Space Invader

A good old Space Invaders game in the mobile AR space, multi-player allowed, shoot not only aliens but other players. Mobile AR, multi-person. Prototyped on iPad Pro and Unity.

VR Maker Space

Imagining a VR Makerspace, where users interact with all sort of interaction techniques in VR, and with live-data pulling from the realworld. Prototyped in HTC Vive, Unity, single person VR HMD setup.

360 Car Simulator

The driving simulator was setup for autonomous driving simulations, we are able to drive remotely from the external steering or to set up preprogrammed route to mimic the self-driving scenario for user testing. I help set up and work with the car simulator mainly for a research on Feedforward Audio icons in Self Driving Cars. Which means we set up a set of scenarios with designed sonification for each scene, and see how the passengers in the car perceives those sounds to create trust and less intrusive awareness between vehicle and passengers. Unfortunately, most recordings during the actual research were not available to share at this moment. 3D experience prototyped in Unity and Land Rover open sourced driving simulator (which they mapped the San Francisco roads in 3D).

Artsy - Wifi Signal Visualizer

The Artsy wifi signal visualizer is a project funded by Charter Spectrum and NYC Media Lab. At the beginning, we were told to build an AR app that help post installation customer service, which I created the storyboard below.

We were too naive, after user research with a first prototype, we found two problem,