[ reposted from blog.sokay.net ]
Raybeem is available on Steam for Oculus Rift & HTC Vive.
Raybeem is the first VR project from Sokay. It's not 2D. It's not a game. Yet, it's as close to my heart as any of the previously released Sokay products. I spent 2017 working full-time on Raybeem. I can describe it as a whirlwind, which was hard to make sense of while I was in the middle of it. I'm writing this mess to dump what's been going on in my head into a huge blog post.
Raybeem is a virtual reality project that started out as a desire to listen to my favorite Drum & Bass station, Bassdrive.com, in Virtual Reality. As an adolescent, I spent dozens of hours of my life gazing at Winamp visualizers. Switching through visualizations to find my favorite, tweaking settings, and downloading new ones. This was the prime way of listening to music for Bryson in the year 2001.
This was my dream for 2017, listening to Drum & Bass music in VR. In a world of my creation.
Raybeem's been a large project for me so I've had a hard time figuring out what to say about it. At it's core, I consider Raybeem to be a VR music visualizer. That was the starting point for the concept, anyway. I just wanted to create an application for listening to music in VR. The different visualizations in Raybeem are thought of as "Themes." The way I imagined, a Raybeem theme is any environment that reacts to music in some way. So a theme could be anything - realistic or abstract, interactive or non-interactive.
When I started working on Raybeem, I had a strong idea of what I was trying to build. As it turned out, even with all the notes, sketches and prototypes, I fooled myself into thinking it would be less work than it was. Here's a brain-dump of a lot of the things I figured out on my Road to Raybeem's release.
Audio Spectrum
This was a demo to figure out how to access the audio spectrum data in Unity.
The heart of Raybeem is a component that analyzes the audio spectrum. This is accomplished in Unity by using the AudioSource.GetSpectrumData() method. This gives me the audio data for the current frame of audio being played through Raybeem. This doesn't care what the source of the audio is, as it can come from local MP3's, web streams, microphone input, etc.. I took this data and divided and averaged it to come up with LOW, MID, and HIGH range values between 0.0 and 1.0. This essentially gives me a percentage for each of those ranges. I still consider it a naive approach but it worked GE (Good Enough).
I used that audio value to determine a variety of different things. For the most part, these components control things like object scale and color. Another example is the particles that boost their speed above a certain audio threshold in the "Blue" theme. These reactions are defined simply through Unity components that I can drop on objects in themes.
An example of a Raybeem audio react component within Unity.
Since then, I've learned a few more things about working with audio. At some point I'd like to revamp the audio analysis aspect of Raybeem to allow for more nuance in theme reaction. I'll get to that sooner or later!
Themes
The only thing that Raybeem's themes have in common is that they react to music in some way, or provide a great experience for listening to music. In most cases, I've created components that listen to an Audio Spectrum range and react based on its percentage of activity. For example, in the "Blue" theme, I have particles that float around and react to the MID range of the audio spectrum. When the MID range is at 0%, the particle is yellow and when it's at 100%, it'll be completely red. If the MID range is at 50%, it'll be exactly halfway between yellow and red, so some orange-y color. I've created different components in the themes which apply the same logic to properties like 'scale' or more complex color shifting for the windows in 100's of instances of buildings for the "Highway" theme.
That was a mundane explanation, but essentially I made a bunch of these components and playing around with them I stumbled upon these themes.
"Blue" Theme
The initial theme for Raybeem is this particle system called "Blue." The algorithm for the particle system is actually based on a tutorial I read by Robert Hodgin. I believe Apple bought the iTunes visualizer from him as it was originally a plug-in he developed for iTunes. It's quite a shame that they took out all the bells and whistles from it, but that's another story...
Anyway, the particles were initially going to be something like faeries or butterflies in this "Zen" theme I was developing when I first started working on Raybeem. When I started testing them as I was blocking out an environment for VR, I felt that it was mesmerizing to just sit and watch what the little particles floating around were doing. This was before I even had music in the demo.
From there, I was unclear with the exact visual direction for this "Zen" theme so I dropped the particles in a new theme test with a black backdrop. The black backdrop was too harsh on my eyes, with the solid black with white particles on them. So I tinted the backdrop Sokay blue. Then I noticed that having a solid color as a backdrop didn't feel right, so I added a gradient so that you can get a sense of which direction is up at least. I then added in a isohedron wireframe dome to give some sense of a horizon. Then I dropped in faded building planes in the distance to give a better sense of depth. Additional tweaks to the particle shaders to give a fog-like effect to show more depth.
This was the testbed for figuring out if there was a visual connection to the music. Initially it was on Gear VR so I could easily show it around to people to gauge interest in it. I felt that most people had a very positive reaction, even if some couldn't see past the initial prototype. I found out that a lot of people couldn't get past the abstract nature of this theme, so I wanted to show a variety of themes so people wouldn't get things twisted.
"Highway" Theme
Once I hit the ground rolling, I had this concept for a "Highway" theme for Raybeem. I had gone to Tokyo, Japan for 3 years in a row and the vision of a huge never-ending city has been forever scarred on my mind. I'm from LA, which is a big city, but it's all spread out. Not so dense. I wanted to try to make an endless procedural theme where you could "drive" on a traffic-less highway forever. The LA dream, a city with no traffic (as my homie Chris pointed out to me).
Don't ask me why, but I decided on modeling this thing myself. I've got an art background so I'm not shy to throwdown when it becomes necessary. But it's also necessary to flex once in a while to maintain said skillz. I was aiming for a lo-fi Star Fox kinda look since I'm still nostalgic of that era of 3D graphics (shout out to Virtua Fighter and Virtua Racing!) I was experimenting with that look with my WebGL Rush-D demo a few years ago, so I started right where I left off.
A city block laid out within Unity.
The city didn't end up being procedural, but it's a bit randomized. I setup about 10 different arrangements of city-blocks. Each block, I meticulously placed buildings by hand and adjusted them based on how they looked as you passed by while in VR. The order in which the blocks come by is random. I used the Unity asset Curved World to randomly bend the environment to give a sensation like you're turning around corners while on this endless road.
Initially the plan was to take you through different areas of the city - tunnels with trains and underground structures, coastlines, and endless bridges. I scaled back my vision because I had to actually make all of that stuff! It became time consuming because I ended up needing to add more details to the theme like benches and trees.
I sketched out concepts for city block types.
I cut corners wherever I can. Since I was going for a flat shaded look, I decided to use the COLR Unity asset. I figured the only way I was going to finish Raybeem was by setting limits and accepting what I could get within whatever restraints. COLR saved my ass because it shades polygons based on which direction it faces (essentially North, South, East, and West). I had to combine this shader with the Curved World shader and it was a fairly easy integration.
Where things got tricky was trying to update so many instances of materials. So for this theme, I isolated the window materials for buildings and adjust those dynamically. Since I've got dozens of instances of each particular material, I decided to just modify the sharedMaterial for each material. You're generally not supposed to do that because it has the unintended consequences of modifying the original asset. Since these material would always be only updated by code I just decided to go that route. But at first that didn't work and I tried some other things. Right now it's all cloudy and in the end I ended up building a convoluted method of finding a reference to the material but none of that matters anymore. Let's put the past behind us shall we? It's working after all! ;)
Oh yeah, well the reason why the material wasn't updating after I changed the colors dynamically was because of the way the COLR asset was setup. It used Unity Editor scripts to update the material when using the color picker in the Unity Editor. I think in the end I had to make something to force those updates to happen from within Raybeem. It was a nightmare to figure out but in the end it was an easy fix.
"Lightshow" Theme
I'm still a raver somewhere deep down in my cold heart. Since I started Raybeem I wanted to have a "purely raver" theme but didn't know which direction to go. With the perfect timing, just as I was trying to figure out which direction to go for a new theme, I found this awesome post on Reddit. Yes, if you can't tell, I shamelessly ripped off the feel of this theme. It barely took any time to figure out how to make a particle system to create this warm glowiness in the distance that just felt great in VR. I combined it with a Kaleidoscope asset that I've been looking for an excuse to use.
I wanted people to be able to give VR lightshows with this theme. I'm still working on the hand functionality, but right now you can leave light trails and shoot butterfly graphics. Hopefully later I'll add other graphics like unicorns and other raver kinda iconography.
"FireDancer" Theme
So I've never been to Burning Man, but I've heard stories from homies. This theme is directly based on my experiences at a rave in the middle of a desert near Lancaster about 10 years ago. I cracked my windshield driving off-road in a Toyota Camry chasing glowstick waypoints in the darkness. It was worth it to experience firedancing first-hand, in the bitter cold of the California desert. Yolo.
For this theme, I actually got the fire from one of Unity's tutorial projects. It's on the official Unity YouTube somewhere if you're interested.
"StarStream" Theme
This is the first theme that I released in an update after Raybeem's release. With this theme, I set out to give more control to the user and give them a world that they can have more interaction with. Much of this interaction comes in the form of an optional mini-game.
When starting the theme, you appear in a space-like environment with star bits of nebula-ish clouds flying by you giving you a sensation of movement. You can use the control sticks to move in the direction your facing, backwards or side-to-side. You have a mini-map in the HUD locked to your view, it appears in the bottom left corner of your view. Using that, you can find a "Sokay coin," which appears as a beacon. Upon collecting the coin, a mini-game is triggered and you must defeat all of the spawning robots before the time is up.
I kitbashed some Unity assets for the robots and the guns you use. I might eventually create new artwork for this theme, and revamp the gameplay, but I'm planning to focus on new themes at this point. Here's a video of it in action if you want to take a look.
VR Input
Getting this thing working for both of these controllers wasn't that fun.
Initially Raybeem was a prototype for GearVR and the Oculus DK2, which at the time relied solely on gaze for UI selection since the VR motion controls weren't readily available at the time. As I began serious development of Raybeem, I had to rework the UI input system to be more flexible to allow it to work with any amount of selection inputs. In addition to the new UI functionality, I had to figure out a system for dealing with VR input with both the Oculus Touch and the HTC Vive controllers.
For the gaze-based UI, I ran an update loop that raycasted from the camera and checked for collisions with any object containing an interactive UI component. If that UI component wasn't already selected, it became the selected object. From there, any selection button press would trigger that interactive UI component. To make it work with controllers, I instead made a list of different sets of raycasts that interact with interactive UI components. So now I had:
Camera Raycast
