VIew

House of Hype

House of Hype

AR Experience
Photo Booth
Touchdesigner
Webcam

House of Hype

AR Experience
Photo Booth
Touchdesigner
Webcam

 Role - 4  People  Team  Project

 Role - 2  People  Team  Project

 Role - Individual Project

 Role - Individual Project

 Role

 Role

 Role

 Role

 Role

 Role

 Role - Individual Project

 Role - Individual Project

 Role - Individual Project

 Role - Individual Project

Responsible for React-o-tron photo booth (Experience Build, Technical Development & Integration, Visual Design), Concept Ideation & Tech Support for other HoH experiences

 Duration

Nov 2023 open

 Tool

TouchDesigner, MediaPipe, Webcam, Blender, Illustrator

 Overview

Hypebeast launched "Crunch Verse," an immersive AR & VR campaign in collaboration with Cap'n Crunch® and the fashion brand Advisry. This interdimensional scavenger hunt, styled in retro PSX graphics, blends nostalgic breakfast memories with the adventurous spirit of Cap'n Crunch. ⚓

Flipping boxes of three iconic cereal flavors unlock portals to whimsical and imaginative worlds, where the Cap'n and his friends embark on exciting adventures. Once completing the tasks, you'll have the chance to win the entire collaborative capsule featuring eye-catching streetwear designs from Keith Herron of Advisry. Dive into the Crunch Verse and explore new dimensions!

House of Hype is a 60,000-square-foot immersive theme park in Riyadh, Saudi Arabia.  HOH is an exciting hybrid of content creation arena and physical video game. It combines 46 creator experiences, 29 immersive attractions, and 32 gaming experiences into one overall visitor experience.

Our innovation team of 10 was responsible for creating 23 unique experiences using a range of technologies and one visual identity to tie them all together.

Two Reels AR filters were featured in Meta Beach 2023 at Cannes in partnership with KidSuper.

Inside the Reels SuperStudio, collaborated and created with KidSuper, festivalgoers created their best Reels content using AR effects, sound, and KidSuper-inspired backdrop.  Guests received a Reels-branded T-shirt with an idea-centric graphic, which is the tracker to trigger the AR effect, called Whose Idea. Another incredible AR effect, called Doodles Scape, transforms you in your environment into a big sketchpad. This dynamic effect underscores the KidSuper brand’s aesthetic and playful ethos.

Three Reels AR filters belong to Meta Beach 2024, inspired by the Reels Cinema featured Es Devlin's immersive 9:16 vertical film "There Is No Other," showcasing the innovative capabilities of Instagram Reels. Guests could recreate beautiful moments from the film using three creative filters in the physical Reel boxes. Action!🎬

Meta Beach at Cannes Lions 2024 embraced immersive experiences and connection to innovation with their Reels Cinema, Ray-Ban x Meta glass maze room, Meta Quest 3 VR headset, and several Meta AI activations such as Seamless Expressive and AI kiosks.

This is an healing interactive experience using Kinect motion sensor that evokes people's childhood innocence and memory with simple yet fun intuitive action. Players can fold their paper airplanes and make the best airplane. And then launch the magic paper plane; the trace's color and music's speed are associated with the flight of the paper airplane.

Deepen the player's curiosity about how flight affects the digital content by repeating the simple and enjoyable nostalgic game. I hope this work can give players a carefree-like experience.

This is an Augmented Reality relaxing experience with lovely low-poly graphics. It brings the experiencer the joy of combing miniature interior and room decorating—simple, healing, and fun!

The cube room was divided into four parts (bed, closet, desk, personal items) plus the wall part, and each part has four styles so far. Experiences can choose and switch the style they want by touching furniture/items in the AR environment on a mobile device.

After choosing the favorite combination, print and scan the final room card (top view of room), and then put the card on the landscape board to fit the blank square. Next, the virtual cube room will show above the card.

Ctrl T is a 4-minute long stop motion animation. We transfer parents' longing for control toward their children into literal genetic modification. Would children still be happy, confined in the frame of their parents' expectations? Can we escape from such value like this? The unique texture and the warmth that stop motion possesses bring out the wishes deep inside the boy. We deliver warmth to the boy's life, yet at the same time, control him as well.

This project is designing a narrative amusement park attraction and using the game engine (Unreal) and Virtual Reality to create and simulate the real-time immersive environment of the ride.

Promisedland is an attraction with shooters and a suspended ride system integrating Mixed Reality glasses.

Neverisland is a healing/entertaining interactive experience that raises an adult's happiness based on the Mixed Reality immersive environment by integrating the natural environment and nostalgia components.

This world is set on floating islands with a low poly style. Experiencers can choose to land on any island to experience different interactive stages. Users can experience and interact with Neverisland in their familiar physical space instead of a completely virtual environment. By comprehensively blending both physical and digital realities, MR is able to build new levels of engagement and immersiveness.

This is an immersive interactive installation that integrates 200 LEDs (neopixel), a pulse sensor, and a temperature sensor. This work visualizes the data inside our body and shows everyone's vital signs/characteristics of life on the light strand. The experiencer walks inside these led strands and touches sensors, then the pulse sensor will change the beat of the light strand, and the temperature sensor will change the color of the light strand. Stay within this installation as if everyone is standing inside themselves. These data are silently functioning in our bodies. Besides, I use the 3D print pen to create a DNA shape to present and decorate these LED strands. DNA is unique for everyone to show biological data and secrets inside our bodies.

These are a few selections from all the social filters I've created. More filters are coming soon!

All effects and interactions are developed using Spark AR and are available on Instagram.

This is a Social VR experience. A safe haven where you can roam in a virtual amusement park with partners during the Covid pandemic. Each entertainment themed land is built on a floating island that users can travel around to navigate, enjoy various lands, and interact on facilities with partners.

An AR Try-on mirror experience belongs to Uber’s Reserve Lounge.

To increase awareness of Uber’s Reserve feature,  an exclusive Airport Dad Lounge was established at Chicago O’Hare Airport. A planner paradise, it had everything an Airport Dad could need or want.

All you had to do to access the lounge was reserve your ride. Inside, guests could experience an AR try-on mirror, engage with a planning wall, get AI-generated postcards, and take on a luggage packing challenge.

Yacht Viewer is an AR binocular that will allow users to look inside to identify Yachts across from the SRMG Beach.

Saudi Research and Media Group (SRMG) has unveiled the SRMG Beach experience at the Cannes Lions Festival 2023. SRMG Beach represented the MENA region’s creative hub and also featured immersive and interactive AI and AR activations from SRMG’s leading brands, including an AI reporting video booth, a Real or Fake arcade game, an AI Manga Character Cover photobooth, an AR Try On, and an AR binocular.
Our innovation team was responsible for creating these activations using various technologies.

 Trailer Video

 Full Experience Video

 Full  Video

 React-o-tron Photobooth  Video

 Official Whole Park Video

 Background

Concept

In our growth phase, we often find out that there is something similar between our parents and us. Parents shadow our life more or less whenever we make our own choices or behave the exact way we wish to do.

Most parents expect us to be the perfect children they want us to be. They have high expectations, wishing their children to reach the goal they set for them. However, the expectation parents hold is not suitable or even good for every child. The expectation might be shaped by the majority of society; meanwhile, the universal value and standard created by the majority may not be essentially the best. The biggest question is: If we cannot keep our own thoughts, what is the point of creating a world in praise of diversity?

Worldview

In this world, we have a fantasy machine that can control our figures. Parents can use it to control and deform their children. For example, if children do not have the intelligence for learning, parents can use the machine to reform them. After the reformation, parents can find their children to be more intelligent, but the appearance will become freaky; they might have a significantly huge head.

We use this exaggerated way as a metaphor for parents’ control, according to our own experience as children. We find out parent’s control is something rather imperceptible that sometimes we cannot even notice. Hopefully, our animation can form an intense impression in the minds of parents or those who want to forcibly affect or control others. Every person in the world is a different individual, and it is wrong and unnecessary trying to shape another individual.

 Process

Character  Design

Protagonist - Teagan

This animation's title, Ctrl T, is derived from this little boy, which means "control Teagan."

Facial Expression Sketch
Transformation Process Sketch
Transformation Process Color Manuscript
Parents

At the beginning of the story, the camera only shows the lower body of the parents; their faces will reveal until the end of the story. The parents' faces are exactly the same as the little boy's scary face at the end.

Storyboard

Animatics

Technique

We use stop-motion to present our story. Stop motion is a more meticulous and real animation technique in comparison to others. We know that, nowadays, CGI is trying to emulate the qualities of stop motion. But the emotion that stop-motion conveys is impossible to be replaced. We persist in the maker spirit, trying to create everything that appears in our animation on our own, for example, the completely self-made puppet armature.

After the accurate measurement and sufficient research, we went to the maker space and built the metal armature with our own hands. The unique texture and the warmth that stop motion possesses bring out the wishes deep inside the boy. We deliver warmth to the boy's life, yet at the same time, control him as well.

Doll  Making

Face

1. Use Super Sculpey, this material is harder, and it will be solid and not deform after baking, is appropriate for making highly delicate. inner mold.

2. After sculpturing, scoop out the clay of head to be hollow.   (to reduce the weight)

3. Bake to dry.

4. Use silicone to cast outer mould.  (silicone is soft and elastic, easy to remove and take out mold)

5. Cast with poly. (hard, cheap, and does not tend to shrink)

6. After casting replication, use the engraving machine to do engraving, and use mountain meal and clay to do sculpture.

7. After sculpture is done, use sandpaper to rub till smooth, use pen to paint with putty to fill up holes if any.

8. Once the surface gets smooth, spray paint for colors layer by layer, then use the pen to do a detailed drawing.

Eyes

1. Use Styrofoam balls as a base for the eyes.
2. Coat with hardening clay; then polish and color.
3. Paint with crystal glue. (to create a crystal clear sense)

Hands

1. Use iron wire and epoxy putty to make the skeleton.
2. Make the hand model with Super Sculpey.
3. Make a silicone mold and fill it with silicone.

Body

1. Combine a variety of metal mediums to create a unique skeleton.
2. Use the high-density sponge to cover the skeleton as the body.
3. Sewing the exclusive clothes for characters by myself.

Final Doll

Furniture  Making

1. Measure the size.

2. Made of foamboard, styrofoam board, styrofoam ball, thick cardboard, wood sheet, plastic sheet, hardening clay, light clay, etc.

3. Apply putty to the styrofoam surface and the adhesive gap.
4. After the putty is dry, smooth it with sandpaper.
5. Use acrylic paint to color and create an aged appearance.

Shooting  Process

Because the whole setting of the filming process needed to be fixed in the same place for a long time so as to have coherence of photos, we rented a studio as a filming site in order to present the best work.

Dubbing  &  Post  Production

We went to the recording studio to personally dub the soundtrack and add sound effects in order to produce a unique sound that matches our animation.

It was also a time-consuming process during post-production after shooting. In order to present a smoother stop-motion animation, we took up to 7,000 photos and had to remove the dolls' metal support one by one using Photoshop.

Finally, I used After Effects to assemble all the photos, add visual effects, and do the editing.

 Outcome

Under the restriction of limited time and money, nevertheless, my teammate and I accomplished this animation together and won Red Dot Award and other animation prizes; we did it with the perfect outcome. This was a great recognition and encouragement to our team!

We have organized several exhibitions to showcase our work and have been invited to show our work in many animation exhibitions and galleries.

 Process

Max/Msp  Development

I use Max/Msp as a bridge between Kinect and P5js sketch (launch JavaScript to socket.io server).

In this part, I apply cv.jit.centroids node to find the object's centroid in front of Kinect's depth camera, and I adjust parameters to capture the most appropriate proportion.

In  next part, I utilize node.js on Max. It allows me to write JavaScript code on my computer, and node.js can find this file.

Then node.js can run and launch code to the Socket.io web server. This max patch lets anyone on the local network can enjoy this interaction. (Open a browser and type "http://localhost:5003")

P5js  Content  Iteration

To create the paper airplane’s trace/trail when players fly it, I have tried many different codes to achieve a more refined outcome.

I used the mouse's position instead of a paper airplane first when I was developing on p5js.

Finally, I modified it to this version, using transparency, size change, and delay time to achieve this effect.

And then, apply the color and add some glowing effect.

Final Version

Add a dynamic background to enrich the visual effect and add a music. (demonstrate by the paper plane)

Connect to the Socket.io server, and the trace's color and music's speed will change according to the paper airplane's position.

Final Code

 Prototyping

 Outcome

This seems to be a relatively straightforward interactive work, but it was completed within a week, and multiple tools/systems must be involved and integrated to successfully connect the Kinect sensor and p5js.

Besides, I spent a lot of time modifying the time the position stays on the screen because there is a difference between the time the Kinect camera captures the position of the paper plane and the time it is presented on the screen.

This is just the first version of the prototype, but I'm enjoying it. It's amazing and fun for me, whether when I'm folding a paper airplane, when the airplane is interacting with digital content, or when it's bringing back memories of my childhood.

 Process

3D  Modeling

I used MAYA to build the cube room 3D model.

Then, I used Unity to integrate and transfer models to USDZ. format in order to import to Reality Composer.

Final Render

 Experience  Build

Final  Interaction

‍Choose and switch the style they want by taping furniture/items in the AR environment on a mobile device.

Print the favorite combination of cube room and landscape board.

Scan the room card (top view of room), and then put the card on the landscape board to fit the blank square. Next, the virtual cube room will appear above the card. Enjoy it!

 Outcome

This project was my first project developed in Reality Composer(bundled with XCode); I think Reality Composer is an impressive software for creating interactive augmented reality experiences. I’m satisfied with the current prototype, which was completed only in three weeks from modeling, texturing, file conversion, and producing the interactive AR environment.

In the next step, I want to add more style to the room and enrich the interactivity for AR to generate more adorable outcomes.

 Process

 Prototyping

Making  Process

1. Use 3D pen to produce DNA shape.

2. Paint black acrylic on the wood stick.

3. Drill holes in the wood stick (to put the DNA led strip).

4. Weld Neopixel Led port (total 60 ports).

5. Create pin for 60 ports (enable them to connect to the Arduino board.).

6. Put 20 Led strips into wood sticks.

7. Test each LED strip can be lit up before installation.

8. Drill holes in the wooden box (to put sensors).

9. Engrave symbols on the wooden box with the wood-burning pen.

10. Hide components, Arduino board, and wire inside the box.

11. Organize wires.

 Outcome

Although there is still a lot of room for improvement in this prototype, the final result is still a great achievement. The outcome of the combination of neopixel LED and sensor was presented well, and I received many positive comments throughout the exhibition.

It was an extremely challenging experience to connect 200 LEDs in series, execute successfully, and polish the installation in two weeks by myself. I am very satisfied and proud to have achieved this result!

 Ideation

Background

A virtual Social Amusement park during the pandemic; An entertainment environment, a space full of joy, memories, and imagination. It can also be a story that users have experienced or created with their partners before, and users can rebuild that scenario again inside this virtual world.

Concept

Since we are all influenced by the Covid-19 pandemic, going to Disneyland or an amusement park could be an unrealistic dream or a luxury thing to do now( this project was created during the serious Covid pandemic period). What if people would like to experience a traditional theme park, but the parks are almost closed all over the world or are too afraid of getting infected by the virus, so they can’t fully enjoy it? This project provides people with a virtual theme park full of imagination, which you cannot experience in the physical world.

For example, unlike traditional parks, the themed land I built are floating in space. My project is based on Disney theme parks because Disneyland carries a happy memory for almost everyone, but it has been shut down worldwide. In my virtual world, users can hang out together with family, friends, or even strangers and interact with facilities. Each facility will have different ways of interaction, which may be an indoor exploration or simple interactive games so that users can achieve happy memories and fun social life with partners in this virtual Disneyland. I think that “people” is a crucial factor/influencer for the experience in amusement parks, which will bring a different atmosphere, different events, and different sparks.

Reference

Research

Literature Review

To strengthen my concept, I have read a lot of papers, articles, and books about Disneyland and mental health.  In particular, a book called "The Birth of the Magic Kingdom." This book specifies the story behind Disney's Magic Kingdom. From Walt Disney's childhood to the birth of Disneyland, every major event and the turning point in the development of the Disney Kingdom has been recorded and introduced in detail.

An excerpt from this book, "The most interesting is not Disneyland as a game and leisure space, but Disneyland in which many people place a special emotion similar to faith. It wasn't meant to be a playground for children. What Walt Disney aimed at in his mind is the "childlike innocence " hidden in everyone. The "happy place" in Disney's mind is a fantasy world completely detached from daily life."

Book by Masako Notoji
Questionnaire

I also created a Google questionnaire before starting the prototyping and integrated these responses to modify and enrich my initial concept.

 Process

Themed  Land  Topic

Currently, there are only three themed lands, and more creative islands are still being developed. The choice of the current three topics is based on the spirit or moral of each animation that corresponds to my project's concept.

Topic One - Cinderella

"No matter how your heart is grieving, if you keep on believing, the dream that you wish will come true."  -Cinderella

Topic Two - Frozen

In this world, here can “froze” the time to be a timeless space.
Froze happy/wonderful memory here.

Topic Three - Alice in Wonderland

“No matter what happens in the world around you, you have the power to create your own reality.”  -Alice

3D  Modeling

Because Cinderella castle is Disneyland park’s most central/signature part, I put a lot of effort into building it by myself.

First, I began modeling based on the 2D castle picture by Blender. I referred to its general architectural style and elements because it doesn’t show the 3D dimensional details, so I built this castle from my perspective and my imagination to model.

Then, I build the whole castle step by step in detail.

Next, refine and apply the texture of my castle on Blender.

And then, add floating landscape.

Last, extend to the entrance and garden part.

Final Screenshot
Blender Recording

 Prototyping

VR  Environment  Develop  in  Unreal

After building the models in Blender, I imported them into Unreal. Then I integrated all the themed lands, developed a VR entertainment environment, and added interactions.

Process

Setting for VR functionality.

Create portals that enable the experiencer to travel around different islands. Also, make the basic instructions for each island. The info will appear when the player gazes at the sphere near the portal.

Portal's blueprint and material. This part took me a lot of time to successfully get the character to teleport.

Instruction's blueprint.

Create triggers for interaction with the controller.

Add VFX for interaction.

Each  Themed  Land  in  Unreal

Island One - Cinderella & Fireworks
Island Two - Frozen
Island Three - Alice in Wonderland

Upload  Apk.  File

In order to let more people experience it and do user testing, I launched and uploaded apk. file to the Oculus headset via SideQuest. This allows my prototype to be installed directly on Oculus without connecting with unreal.

Documentation

Stereoscopic VR Recording on Oculus
VR Environment Render

 Outcome

This project was my first VR interactive experience developed in Unreal, so it was a challenging process from modeling, texturing, building scenes, adding the interaction, integrating effects, and executing it successfully on VR glasses in just six weeks. I am a big fan of Disney parks worldwide; when I saw the virtual park presented in front of me, I was very moved, so I like this work very much. I am delighted and proud to have achieved the current outcome!

Through the exhibition and presentation, I received many positive comments and opinions, and I will continue to modify this work based on the feedback and add more theme lands.

In addition, due to the time limit, the function of multiplayer has yet to be incorporated. In the next stage, I will add this feature to further strengthen the social VR experience concept.

 Elevator Pitch

This project is developing in Unreal with Magic Leap 1 glasses using Mixed Reality technology.

This is an attraction with  Shooters and Suspended ride system integrating Mixed Reality glasses.

The explorer from the moon met with the first challenge when they arrived on Earth. The spaceship kept warning of malfunction and asked them forcibly land. Magical energies pervade this new earth after 400 years. The five elements, metal, wood, water, fire, and soil, are the basis of the magic power. In this world, some species can absorb these five elements and store them in their bodies, thus generating magical power.

The explorer will experience the whole story through the ride with the fairy Fay and her friend, a butterfly vehicle named Vanny. They must collect five elements to restore the power of Fay and repair the spaceship. In the end, the explorer will go back to the moon.

The biggest feature of the whole attraction is the combination of reality and virtual to increase the entertainment and interactivity of the whole experience.  Explorer will have an adventure with virtual fairy, dragon, and monsters, and the interaction of shooting and collecting the five elements in MR headset. The landscape and environment are built in physical world, and the audience will experience water spraying, wind, and other physical effects during the ride as well.

Attraction/Ride Logo Design

 Process

Overall  Storyline

Attraction  Layout

We decided to use the spaghetti layout and one-person ride, so the player could have a better immersive experience and avoid the influence by other players. These curves are designed to leave space for scenes.

There is a strong start; the player will start shooting the monster as long as the ride starts. Then there is a short instruction between scenes one and two to teach and inform the play that they need to collect the five elements during the journey.

Script

 Ride Prototyping

Diorama

We built a diorama according to the layout, including every scene and simulation of the ride path.

Process
Scene 1 (Monster)
Scene 2 (Fire)
Scene 3 (Metal)
Scene 4 (Wood)
Scene 5 (Soil)
Scene 6 (Water)
Scene 7 (Spaceship)
Final Model
POV  Recording

Unreal  Scene  Build

Process

Since this is a simulation experience of the amusement park ride, I chose to use the RealityScan app (by Epic Games) to do the 3D scan of our diorama and import the real model of each scene into Unreal instead of rebuilding the virtual scene in order to preserve the authenticity of the scenery.

However, the scale of our entire diorama is too large to thoroughly scan the scene in detail.

Therefore, I scanned the scene one by one and then imported them into Blender to remove the excess meshes and finally imported them into Unreal to integrate the whole scene.

In this way, the original appearance and details of diorama be presented in a more complete way and perfectly in Unreal.

Scene 1 ( import into Blender)    
Integrate  in Unreal        
Scene 1.5 ( import into Blender)                
Integrate  in Unreal                   
                                              Scene 2 ( import into Blender)           
                               Integrate  in Unreal  
              Scene 3 ( import into Blender)                 
 Integrate  in Unreal   
 Scene 4 ( import into Blender)                                 
 Integrate  in Unreal                                    
 Scene 5 ( import into Blender)                                        
 Integrate  in Unreal                                        
 Scene 6 ( import into Blender)                                              
 Integrate  in Unreal                                                    
 Scene 7 ( import into Blender)                                       
 Integrate  in Unreal                                    
Final Scene Integration in Unreal
POV Recording in Unreal

Immersive  Experience  Build  in  Unreal

Character Design
Fairy Fay - Visual Guider
Generate Animation by Mixamo
Ride Vehicle Design
Butterfly Vanny - Ride Vehicle
Adding Ride Path
Adding VFX and Level Build
Adding Interaction for VR Environment
Final Screenshot for Each Scene

 Further Development

Stewart  Platform

In order to enhance the user's immersion and enrich the authenticity of the whole attraction experience, we apply the Stewart platform to our project, simulating the vibration and movement that the player will feel physically in the actual ride.

We use Max/Msp as a pipeline to connect and transfer data between Unreal and the Stewart platform and use the OSC plug-in in Unreal to send transformation data to Max/Msp so that when the camera is shaking or moving, it will implement on the Stewart platform.

Real-time Recording

LED  Wall

To level up the immersive experience, we also tried different mediums to present this project. We used the Led wall to create an immersive ride and cinema camera to document the interactive process. By projecting the VR headset's content onto the Led wall, other audiences can also immerse in the scene and enjoy the attraction simultaneously.

Real-time Recording

 Ideation

In recent years, the impact of chronic stress on mental disorders has become the subject of many studies. Emotional pressure has become a high-frequency word used in modern society. It is an unavoidable psychological state that everyone is facing. In particular, the global pandemic has exacerbated people's psychological pressure in the past two years. This global public health concern and personal experience led me to choose the development direction of this thesis project.

Therefore, I desire to make an entertaining interactive experience to help more people like me who are under stress to relieve themselves. After doing plenty of background research and reading (stress-relieving Importance, happiness, nature in well-being, nostalgia in well-being, virtual environment therapy), I have narrowed down and integrated two central aspects into my project: nostalgia and nature.

Many studies are revealing the value of nature experience for happiness; in my research, nostalgia has a great connection and influence on human mental health. From what has been mentioned above, I was inspired to create an immersive entertainment experience to help adults relieve stress by combining nature environment and nostalgia components.

 Setting

World

This project is set in a stylized world of floating islands. Floating cities or islands are a common concept in speculative fiction. Many of these magical locations can be found in ancient mythology.
Floating islands remain places of wonder and mystery which coincides with the magical ability of Mixed Reality; MR can present fantasy and nonexistent things to people's eyes. Therefore, I choose the floating islands as my magic world's setting.

Aesthetic

I chose the low poly and cartoon stylized design for this project's visual aesthetic.
The sharp edges and abstract shapes of triangular polygons are easily reminiscent of a memory of past emotions, such as computer games in the 1990s. Owing to the unique meaning given by the times, the low poly has a new aesthetic tendency of nostalgia and retro.

Low poly uses advanced rendering, light contrast, and color matching to restore simple objects in an old-fashioned way, giving a more realistic and futuristic feeling. Low poly is a kind of retro-futuristic style design that goes back to the past and goes to the future, finding an aesthetic balance between the swing of the past and the future in digital art. Low poly's retro character just corresponds to my nostalgia factor, and its sense of future is also in line with the scientific and technological nature of the new technology- Mixed Reality.

Moodboard
Mood Board

Island  Stage

After seeing floating islands in the first scene, players can choose to land on any island to experience different interactive stages. For each island's experience content, I did the research and tried to find out what factors can evoke nostalgic memories in most people, not just in the Asia area.

Currently, I have developed three islands - Whack A Mole, Fishing in your Space, and  Build the Snowman. Then I include these nostalgia components into the different islands with the natural environment and let players interact with them.

Besides, when players enter an individual island, the island's area will become square, like a cube room, rather than a complete landscape appearance. I made this modification because I consider that this is an MR experience; the virtual scene in the glasses needs to interact and make a connection with the real-world space. If the presentation of the island is too complex, it will be difficult to fit in each user's physical space, such as interfering with furniture or personal belonging. In contrast, the more simplified cube can be integrated into any corner of the actual space.

 Process

Development  Environment

This project is developing in Unreal with Magic Leap 1 glasses using Mixed Reality technology.

I was amazed and inspired by the object's occlusion and interactivity with familiar physical space on Magic Leap. This is one of the main reasons I changed to using MR technology instead of VR for this project ultimately.

Demo from Magic Leap

When seeing or playing games and things with usual elements, people will involuntarily recall the good memories and times in the past because of the mere-exposure effect and unconsciously have a natural affinity and favor for these games with nostalgic elements.

Users can feel safe, experience, and interact with Neverisland in their familiar physical space instead of a completely virtual environment. Everyone can access the imaginative Neverisland by Magic Leap and have privacy in their own personal space.

By comprehensively combining both physical space and digital renders, Mixed Reality is capable of building new levels of engagement and immersiveness. This helps to create links between human, computer, and environmental interactions.

Connection Pipeline

Technical  Difficulties

I encountered great difficulties when I first started to develop my project. Magic Leap is a relatively new area and device, so there are not many related resources and templates for reference and learning.  Even the tutorials issued by the official channel have only a few videos for development in Unreal. What's worse is that the Magic Leap’s API document and example for Unreal stopped updating in version 4.24 two years ago (the newest version of Unreal Engine is 4.27) when I developed this project. 

Because of the outdated version, the only toolkit template that can be referenced is incompatible and has a lot of errors when I open it with Unreal. The toolkit template cannot execute at all due to several errors in Unreal’s blueprint.

Also, the input part for Unreal to perform the motion controller is empty, which means nothing will happen when users click or move the Magic Leap’s controller.

However, it is unrealistic to change the development game engine in this phase, and I still want to keep the Mixed Reality's advantage of object's occlusion and interactivity with the real-world rather than change to using Virtual Reality devices. There are very few resources to solve problems.

Thus, I decided to fix these errors by myself and make my project  executable using Magic Leap in Unreal.  This is an extremely time-consuming and tough process, but I am glad I did it in the end.

Functionality  Testing

Before I start developing and building the experience content, I want to test what functions and interactions I can achieve based on time limitations. To avoid increasing the challenges of later production because I have already spent too much time solving technical difficulties.

Hand Gesture Detection
Hand Gesture Control Object
Touch Virtual Button by Hand
Surface Detection
Eye Gaze
Meshing Real World and Interaction
Replace Surface Texture (Ceiling)
Replace Surface Texture (Floor, Wall)

 Prototyping

Island  1 - Whack  A  Mole

Scene building

According to my previous research and topic, for the island one's scene, my overall environment setting is natural in the low poly style, and the season is set in autumn. Thus, the color and material saturation of the whole cube island is low and a little dim to create the atmosphere of autumn.

On the part of the landscape, I deliberately set its height lower because I don't want them to block too many visual areas and physical objects when the virtual cube island is presented in real space inside the Magic Leap glasses.

Assets building

In order to correspond to my research, I included the elements such as the mole and the hammer to relive the nostalgia and childhood memory from the whack-a-mole game that most people are very familiar with.  The model of the mole and the hammer also apply the low poly style to match the whole project’s aesthetic setting. Whack-a-mole is not only entertaining; it can also relieve the user’s stress and help restore emotions.

Interaction/ Experience mechanics

Another important reason I choose the Whack-A-Mole is that it is a casual game; players can play in a relaxed mood. The casual game has a straightforward design and goals, repetitive gameplay, and no complex mechanics or previous knowledge required to play- enabling can simply get started playing. Because of these characteristics, causal games can instantly divert the user’s attention and relieve stress .

Level building and integration in Unreal

In the final phrase, I integrated the natural cube island in autumn and whack-a-mole game to island one’s level in Unreal with the soft lighting. Besides, I replaced the ocean part with the dynamic material made by Unreal to enrich the atmosphere of the natural environment.

Lastly, I implement the Place On the floor and Surface Detection from previous function testing to let the cube island spawn on the floor and enable the whack-a-mole module to generate on the actual table surface in the user’s real physical space.

Further, I edited and compiled the whack-a-mole interactive mechanics by Unreal’s blueprint system, which lets the user can hit the mole with the hammer inside the glasses.

Execution in Magic Leap

Explore and enjoy on Magic Leap headset. During the whole experience, the participant can press the home button on the controller anytime to return to the main menu,  then select and play different islands.

Island  2 - Fishing  in  your  Space

Scene building

In line with my previous research and topic, for the island two's scene, the overall environment setting is still natural in the low poly style, and the season is set in summer. Thus, the color and material saturation of the whole cube island is relatively high and bright to create the atmosphere of summer.

On the landscape aspect, I also deliberately set its height lower because I don't want them to block too many visual areas and physical objects when the virtual cube island is presented in real space inside the Magic Leap glasses. In addition, I contain the sandcastle in the scene to enhance the nostalgic memory.

Assets building

In agreement with my research and topic, as I mentioned, I want to include some nostalgia factors that can evoke nostalgic memories in most people globally, not just in the Asia area. After doing research and interviews, I applied the fish model to recall the nostalgia and childhood memory from the magnetic fishing game toy that many people have had contact with.

Then, I recreated the fish toy model into 3Ddigital form by Blender and generated different color textures to assign to thefish model The fish models also use the low poly style tocoordinate the whole project’s aesthetic setting.

Interaction/ Experience mechanics

Another important reason for choosing a fishing toy as the reference is that it can be perfectly combined with the interactive mechanism of “fishing.” Fishing is unexpectedly associated with happiness and wellbeing.

In an article, psychologists Mark Wheeler and Nick Cooper promoted fishing as an effective method to reduce symptoms of post-traumatic stress disorder. They found that a fishing trip reduced the symptoms of PTSD in war veterans by 30%. For most fishing amateurs, fishing is a way to relieve pressure, it is just an activity to relax, breathe fresh air, embrace nature, and relieve work stress, but it also has an entertaining effect.

Level building and integration in Unreal

In the final step, I integrated the natural cube island in summer and the fishing game to island two’s level in Unreal with the soft lighting. Besides, I also replaced the ocean part with the dynamic material made by Unreal to enrich the atmosphere of the natural environment.

Last, I utilized the Place On the floor and Meshing Real World from previous function testing to let the cube island spawn on the floor in the user’s real physical space. Also, I edited and compiled the fishing interactive mechanics by Unreal’s blueprint system to enable the user can fishing inside the glasses.

Execution in Magic Leap

Explore and enjoy on Magic Leap headset. During the whole experience, the participant can press the home button on the controller anytime to return to the main menu,  then select and play different islands.

Island  3 - Build  the  Snowman

Scene building

In accordance with my previous research and topic, the overall environment setting for the island three's scene is also natural in the low poly style, and the season is set in winter. Thus, the color and material saturation of the whole cube island is relatively cold and raw to create the atmosphere of winter.

I intentionally set its height lower on the landscape aspect and deleted unnecessary meshes; because I don't want them to block too many visual areas and physical objects when the virtual cube island is presented in real space inside the Magic Leap glasses.

Assets building

Corresponding to my research and topic, as stated above, I did the research and interviews; I aimed to find out what factors can evoke nostalgic memories in most people globally so that the user can embrace the nostalgia emotion in this world.

Thus, I applied the snowman to relive those carefree, happy moments of childhood memories with family or friends when building a snowman and also to fit the winter environment setting. The snowman model also applied the low poly style to follow my whole project’s aesthetic setting.

Interaction/ Experience mechanics

In order to simulate the behavior of building a snowman, I choose the “assembling” to be the primary interaction mechanics on this island. Hence, I edited and compiled the assemble function by Unreal’s blueprint system, enabling the user can assemble and build the snowman with snowballs inside the mixed reality glasses.

Level building and integration in Unreal

For the last stage, I integrated the natural cube island in winter and the snowman interactive module to island three’s level in Unreal with the soft lighting. Besides, I still replaced the ocean part with the dynamic material made by Unreal to enrich the atmosphere of the natural environment.

Last, I implemented the Place On the floor and Meshing Real World from previous function testing to let the cube island spawn on the floor in the user’s real physical space. Also, I edited and compiled the build the snowman interactive mechanics by Unreal’s blueprint system to accomplish the interaction of assembling snowballs inside the glasses.

Execution in Magic Leap

Explore and enjoy on Magic Leap headset. During the whole experience, the participant can press the home button on the controller anytime to return to the main menu,  then select and play different islands.

 Outcome

After doing the user testing, most of the feedback is positive, such as some people feel that the combination of virtual objects and actual physical space is very impressive, or some people were really healed and decompressed to interact on each island.

For the suggestion aspect, most I received was about the size and scale of the virtual objects presented in the mixed reality glasses. Some models on each island block too much view of the actual space. Therefore, according to the suggestion, I changed the scale of some interactive components like snowballs and hammers, and I also adjusted the height and area for each island's landscape of my project's prototype.

In addition, other suggestions are about the selection of the input method. My current prototype mainly uses controllers to interact; some people suggest that I can add more gesture control parts because I have done many gesture function tests before, which may enrich and deepen the experience of mixed reality. Overall, I will continue to do more user testing and constantly change my project based on these evaluations.

I hope this entertaining interactive experience can help more people like me who are under stress to relieve themselves by combining nature and nostalgia aspects.

Users can feel safe, experience, and interact with Neverisland in their familiar physical space instead of a completely virtual environment. Everyone can access the imaginative Neverisland by Magic Leap and have privacy in their own personal space. I desire to implement my strength and the extraordinary power of new technology to make some contribution to people's mental health in modern society.

 Next  Step

For the future of Neverland, I plan on continuing to refine and iterate the prototype based on much additional user testing and research.

In addition, I will develop more floating islands and interactions for Neverisland, such as the starring night bonus island that I wanted to include initially and spring island. Users can have a variety of experiences inside Neverisland by Magic Leap.

Moreover, I will create the tutorial and upload the Unreal demo project with the blueprint system to the development platform like GitHub for those who also want to use Magic Leap one in Unreal. I hope this will allow those developers to take the less difficult process and save time in troubleshooting.

 User Journey

Reels-branded T-shirt AR Filter - Whose Idea?

Point toward t-shirt’s idea-centric graphic to trigger AR effect.

Reels AR Filter - Doodle Scape

Explore Reels SuperStudio and create your creative Reels content with door frame scenes using AR technology.

 IG Reels Output

Reel creation by combining AR filters

Credit : @therobryan
Credit : @N/A
Credit : @worldli
Credit : @danzy_world
Credit : @N/A
Credit : @_stopatnothing
Credit : @mashkilleyson
Credit : @shelrang

Try filter !

Whose Idea?

Doodle Scape

 User Journey

 IG Reels Output

Reel creation by combining 3 AR filters

Credit : @cashrobinson
Credit : @danneary1
Credit : @pomeranskaolya
Credit : @ritzpiano
Credit : @franciswongny
Credit : @woshiguigui
Credit : @missxbrit
Credit : @miranda_acts

Try filter !

Hold Box

Sky Box

Flower Box

 Editorials & Social

Related Articles

https://hypebeast.com/2024/8/capn-crunch-advisry-ar-scavenger-hunt-announcement

https://hypebeast.com/2024/8/capn-crunch-advisry-collection-preview

https://hypebeast.com/2024/8/capn-crunch-advisry-ar-scavenger-hunt-announcement

https://www.adsoftheworld.com/campaigns/crunch-verse

 Share  Moment

Scan QR code and save your AR try-on photo

 Final Output

 Celebrity Tailored Filter

Concept

This is a customized filter made for a celebrity. I modified and created the effects and interaction according to his request. This filter may seem relatively uncomplicated, but I've added a lot of detail and tweaked it many times.

It applies words on the face and allows users to switch to the freckles mode without words by touching the screen; it also employs some makeup, retouching, and slim effects on the face and noise effects for the background.

Process

Download the official face model and then sculpt the face slightly by SculptGL
Download the official face image and then create words in Photoshop
Download the official face image and then draw freckles in Photoshop
Create the noise effect for the background in Photoshop.
Add interaction and other effects like makeup and retouching, and adjust the final effect in Spark AR

Try it !

 Funny Birthday

Concept

This is a virtual birthday celebration filter with playful interaction.

Spark AR has the function of detecting the kissing expression. I use this function to achieve the interaction of blowing candles. Also, I made some modifications to the face, like adding little freckles, smoothing the skin, making a face smaller, etc. Besides, I added an interesting element when the user finished blowing out the candle.

Process

Download the official face model and then sculpt the face by SculptGL
Download the official face image and then draw wrinkles in Photoshop
Draw the birthday cake by iPad
Add interaction and other effects like makeup and retouching in Spark AR

Try it !

 Cosplay

Concept

This filter allows people to “wear” the costume digitally on the Instagram/Facebook application. I choose some famous costumes to present. There are currently three types of 3D costumes available that users can select by touching the little icon at the bottom of the screen.

It also includes basic makeup and retouching effects; in addition, I applied the hand detection function that allows users to switch the mask part of the costume.

Try it !

  Explore More Work

 Outcome

House of Hype

AR Experience
AR Experience, TouchDesigner, Face Tracking, PhotoBooth
© 2022 LILY HSIAO
© 2024 LILY HSIAO
Mobile Website WIP 🛠️
Please use a desktop/tablet for a better experience✨
Work In Progress 🛠️