The Forgotten Importance of Game Audio

We recently wrote an article about ray tracing and how it can impact game design in more ways than most people realize. But ray tracing isn’t the only technology that is underappreciated; sound is another.

Sound effects design is often an afterthought in games today, with some modern AAA games even doing a particularly poor job in this area, as we explained in our Prey 2017 review which had some serious flaws in the audio department.

I haven’t seen any game studio try to seriously innovate with sound design since the first two Thief games in the late 1990s (let us know of others). It’s a shame, because so much is lost. In this article we’ll look at the potential impact modern sound technology can have in game design.

The first two Thief games incorporate 3D HRTF into its game design more than any other game I’ve played. Being able to accurately pick up sounds on all axes (yes, even vertically to some degree), and being able to hear sounds through objects if you get close enough via environmental occlusion, these things are more advanced than most other games and certainly more important. And these games are over 20 year old.

Sound technology has improved a lot since then. Even the technology it uses, EAX, continued to evolve, but its implementation largely did not. AMD paved the way forwards again with TrueAudio, which eventually lead to the open source Steam Audio which integrates TrueAudio Next. Some of its main features include:

• Steam Audio delivers a full-featured audio solution that integrates environment and listener simulation. HRTF significantly improves immersion in VR; physics-based sound propagation completes aural immersion by consistently recreating how sound interacts with the virtual environment.
• Combine accurate occlusion, reflection, reverb and HRTF effects for natural sounding immersion.
• Easy to implement and deploy for all major audio engines, game engines and platforms. Tweak high level parameters to optimize immersion and performance quickly and easily.
• Exercise full control over spatial effects – from fully automatic physical sound simulation, to unique handcrafted tuning and everything in-between.

Physics based sound propagation is a big one that has yet to be properly explored, to our knowledge. This can even be combined with hardware ray tracing (rather than the software based Radeon Rays) for more accurate physical sound propagation; think of how sound waves travel and bounce in reality depending on the materials and environment.

What does this mean for game design and its evolution? Well, think back to the Thief example. That is a stealth game where it is absolutely vital to use your ears, often times when you can’t even use your eyes due to dark environments. You’re sneaking around in the dark, you must hear enemies before they hear or see you, you mustn’t be detected. This can easily apply to horror games as well. I love having to listen to sound cues in the environment.

Simple enough. This is of course not exclusive to stealth games; being able to pick up sound cues in the environment is important for most kinds of games. Competitive FPS players need it. How about trying to identify dynamic distant artillery fire – what kind of artillery and how far? How about hearing dynamic, real echoes in a dungeon, giving you some insight as to what you might face next? Or perhaps the sound isn’t clear enough to do anything but frighten you, great for horror.

Even in a third person action adventure game with platforming and a physics enabled environment, you might hear a rockslide before seeing it. There are endless similar examples for most kinds of games.

With modern sound technology, the game’s sound can become a real gameplay element. Ideally, it could also replace a lot of static, predictable, fake ambient sounds in games – sounds we know are just fake and ambient and not a part of the environment. Now imagine if they ARE part of the environment? This can be used not only to build tension in games, since you can no longer just pinpoint a sound as being fake, it is also fantastic for immersion. Think of a blooming rainforest, the sounds of nature and more of these sounds being real, dynamic, and 3D.

Although, to fully appreciate these advancements in sound technology, surround sound is needed. 2 channels is just inherently too limited, even with binaural. Ultimately nothing beats physically surrounding the player with sound. But those with two channel systems can still enjoy all of these benefits to some degree, particularly when using binaural HRTF.

VR in particular would benefit heavily from technology such as Steam Audio; VR games are designed with binaural sound in mind, but to maximize sound design in virtual reality, we need a 3D HRTF meant for surround sound systems. If a VR player is using surround sound in a non-seated game, they will be turning around so that their front speakers aren’t always their front speakers. With a proper 3D HRTF designed around this, if a player is facing their surround channels, they can serve as the front channels, and vice versa and everything in between. This way, VR won’t be limited to just headphones/ear speakers.

But even Steam Audio and TrueAudio Next are not perfect. Steam Audio doesn’t have built-in support for Dolby Atmos or DTS:X, so height speakers cannot be used. There’s no support for a surround sound VR HRTF, for VR gamers who would rather use a surround sound system than headphones, so the ultimate sound solution does not exist for VR. The existing ray tracing solutions for its physics based sound propagation are very limited; only Intel’s Embree ray tracer (which works on any modern CPU) supports dynamic geometry occlusion (e.g., if a door is blocking sound but the door opens, sound will no longer occlude on the door), but this does not extend to destructible/deformable geometry; for example, if sound is propagating down a corridor but the ceiling collapses mid-way through the corridor, the collapsed rubble won’t block sound. Radeon Rays has no support whatsoever for dynamic geometry. This issue would be solved by using Vulkan ray tracing or DirectX ray tracing instead, but Radeon Rays should also improve since dynamic geometry is possible with software ray tracing.

There have been some advancements in sound engineering in modern games, especially from a recording and mastering perspective since these aren’t unique to games. We also have a few games like SOMA where just about every possible variation of surface collision makes a unique, appropriate sound.

But there is so much more we’re missing out on, with the likes of Steam Audio which is open source and has both Unreal Engine and Unity plugins. It is quite mind boggling how rare this particular technology is in games. There’s also Dolby Atmos and DTS:X which, when used with proper height speakers actually physically mounted far above normal speakers, can deliver proper vertical sound. A few games support Dolby Atmos, but not nearly enough.

How else do you think modern sound technology can evolve gaming? Let us know in the comments below.