Acafellows

VR Development @ Cornell Tech XRC
Project Info
ROLE
TEAM
INFO
TIME
TOOLS

VR Developer, UX Designer
Molly H, Pallavi B
Topics in Mixed Reality Final Project
Spring 2023
Unity, XRI Toolkit, Oculus Quest Pro

Project Description
Acafellows is a hand-tracking 3D creativity support tool and game that aims to provide users with an immersive experience of conducting a choir to create a music piece of their own.
Players can assign each individual chorus with built-in soundtracks, and even record their own voice into the song piece.

Wanna try it on your Oculus?

Click here to download APK

Start Scene

Player starts the game in front of a podium, where they are encouraged to select either Tutorial or Aca Now (Main Game) to continue.

Tutorial Scene

- Tutorial is consisted of 5 steps to familiarize user with the game.
- User must finish the previous step to proceed.
- The scene is reduced to minimal to avoid distraction of the learning.

Main Scene

- 6 Poohs and 1 Finn are present at the main scene.
- 20 music notes are available for user to choose from.

Main Interactions

- Pinch at the pooh to turn it on and off. 
- Spotlight is cast on the pooh that is toggled on.
- Pooh would join the choir singing and dancing at the end of every 8 second (a cycle of the tracks).

(De)Activate Tracks:
Pinch at Pooh

- Pinch on the note, drag and drop the note at the pooh to assign the track.
- A pin is displayed above the pooh’s head during selection.
- A particle feedback is played when the track is assigned.
- The tracks currently on the pooh are turned gray on the book.

Switch Tracks:
Pinch to grab and assign music notes

- Pinch to grab the microphone to record.
- The choir stop singing and the scene is filtered without pooh.
- Recording will auto-stop if the user releases the microphone in the middle, recorded input will not be saved.
- Recorded track is added to the character Finn in the middle, and joins the choir automatically.

Record & Play:
Pinch to grab mic

Post Project Evaluation

Summary

We started with learning hand interaction techniques. We really enjoyed scene building, scene management to toggle/change pooh interactions. Implementation of recording music feature was challenging. The logic to find the closest pooh was a bit difficult at first but after understanding the radius logic, we were able to implement it.

Challenges Faced

- Limited Hand Interaction documentation found online. 
- Recording the whole scene music with a button in the game.
- Create good lighting for the scene using light probes.
- Scene Management and pooh interaction in different scenarios. 
- Git Collaboration

What we learned

- XRI Hand tracking
- Baked lighting maps
- Complex Scene Management
- Recording your own voice in VR

If we have more time, we would implement...

- Volume Control mechanism
- Microphone recording feature: A more realistic scene when recording your own music.
- Save and download the created music piece functionality.

References

Scene Building

https://www.turbosquid.com/3d-models/3d-character-finn-model-1296287 https://www.turbosquid.com/3d-models/free-c4d-mode-winnie-pooh/588967 https://www.turbosquid.com/3d-models/free-basic-microphone-3d-model/725806 https://www.turbosquid.com/3d-models/free-podium-stage-presentation-3d-model/595005

Scene Recording

https://forum.unity.com/threads/writing-audiolistener-getoutputdata-to-wav-problem.119295/ https://gist.github.com/darktable/2317063 https://gist.github.com/R-WebsterNoble/70614880b0d3940d3b2b741fbbb311a2

Wants to see more?

Lets go!