Apr 11, 2019
(via https://www.youtube.com/watch?v=cbN4nLdLWzI)
seen from Russia
seen from China
seen from Malaysia
seen from Japan
seen from United States
seen from China
seen from China
seen from Malaysia
seen from Singapore
seen from Türkiye
seen from Singapore
seen from Malaysia
seen from Russia
seen from Malaysia
seen from China
seen from Canada
seen from United Kingdom
seen from China
seen from United Kingdom
seen from United States
(via https://www.youtube.com/watch?v=cbN4nLdLWzI)
#WHPsightsandsounds #Vehiclesounds #Beforedawn (at Fort Wayne Mancinos)
Influenced by the results to my 'what does fast sound like?' study, I have revisited the Wipeout HD video I had previously created a linear sound design for, and created an interactive vehicle audio system for the sci-fi racers with FMOD. The video shows the audio output of the system.
Taking direction from the 'what does fast sound like?' results, I designed the vehicle sound to have a high pitched, harsh sound, and tried to also capture the sound of the rushing wind. In FMOD Designer, the vehicle event has four layers, as shown in the picture. The first is the 'main' engine sound, an exotic machine sound that represents the power source of the vehicle. The second layer is a turbine-like whine that gives the impression of the speedily moving parts with in the engine. The third is a noise based loop to give sound to the vehicle's thruster and the wind rushing past. The final layer is a low rumble that builds as the vehicle accelerates, giving a sense and power at high speeds. The FMOD project also includes events for damage, air brakes and impacts.
The four vehicle layers in FMOD Designer
The four vehicle layer loops
What Does Speed Sound Like?
I'm planning on doing a sound system for a science fiction-esque vehicle, building on all the car racing game work I've been doing. There is, of course, a very different balance between 'realism' and 'imagination' between a car racer and a futuristic racer, so I wanted to get an indication of what properties of a vehicle's sound give players a sense of speed. I prepared a short video with four pairs of short A/B comparison's of a game car accelerating with different sounds and asked people to note which one of the pair they thought sounded faster. Here is the video:
The Results
Clip A has a lower pitched sound with a high rate of change of pitch. Clip B has a higher pitched sound with a much lower rate of change that levels out mid way. In this case the results went against my expectations. I felt that the higher rate of change in Clip A would create a greater sense of speed, but the great majority of viewer choose Clip B.
Interestingly however, three was an exact 50-50 split in responses to Clips C & D, which had the same rate of change, with Clip D in a higher pitch than Clip C (Clip C is actually exactly the same as Clip A). Here I am assuming that the identical pitch delta in both clips rendered them similar enough to each other to prevent a clear distinction in the results
Clip E compared the 'harsh' sound of a close mic'ed engine with the 'smoother' sound of a car's exhaust recorded at a distance of about half a meter. The clear majority felt that Clip E sounded faster. Here perhaps people associate the smooth exhaust sound with regular, everyday production cars, while the close mic'ed harsh sound is more reminiscent of the sound of racing and rally cars from TV, film and games.
The final pair of clips presented the same car sound with and without the additional sounds of tyres over tarmac and rushing wind. The 100% result for the clip with the additional sounds clearly shows that these help create a sense of speed.
This video shows a side by side comparison of the Core Audio car sound system and the FMOD car audio system. To be as close a comparison as possible, I modified my game in Unity to output 2 identical streams of physics data, one feeding the Core Audio system and the other feeding the FMOD system. The audio output of both systems were recorded separately. In the video, the first clip has the Core Audio audio, the second has the FMOD audio and the third has both, with the Core Audio on the left channel and FMOD on the right, so you can stick on earphones and do a real side by side comparison.
I've made a slight change to both systems since I last posted about them, based on the comparison I did with Need For Speed: Shift. The exhaust sound now fades down over the course of a gear change, fading back in very rapidly when the change is complete. This gives the car a more aggressive feel.
Unsurprisingly there is little difference between the two systems' rendering of impacts, skids and road noise. The interesting differences are with the engine & exhaust sounds.
Generally I think the Core Audio system engine/exhaust sounds better. The changes in dynamics and pitch are much smoother, giving a more natural/believable sound. In the FMOD system the pitch shifting is noticeable as a DSP effect, rather than a natural sounding change, particularly in the lower RPM range. However, it is definitely worth noting that this may be due to the 'second hand' nature of the samples used (see this post for details). The dynamic changes in the FMOD system have a noticeable 'stepping', rather than the smoother fades in the Core audio system. As both systems are receiving identical input data, I wonder if the FMOD engine introduces a latency between receiving and processing parameter updates, or perhaps the it drops updates if its processing capacity becomes saturated.
Speaking of processing capacity, on my 2.93Ghz Intel Core 2 Duo machine the Core Audio system uses between 0.23% and 0.25% of available CPU power when the car is driving at speed. The FMOD system uses between 1.2% and 1.33%, five times higher. I suspect this is due to the entire FMOD engine running, despite only a fraction of its capabilities being used for the car audio. Additionally, FMOD on the Macintosh runs on top of Core Audio, which may also introduce a greater processing demand.
Car Audio with FMOD, Part 2: FMOD API
In this post I’ll detail how I got the car audio system I built FMOD Designer (see this post), running with the racing game I created with Unity (see this post), by using the FMOD Ex API. There are links at the end to download the source code and completed project.
FMOD Ex API Fundamentals
I found it is vital to have an understanding of the FMOD Event System hierarchy of components and content to know which API calls to make and when to make them. When using API you use a single Event System object (FMOD_EVENTSYSTEM), which can contain multipleProjects (FMOD_EVENTPROJECT), where a project is the contents of a single FEV file. A project can contain multiple Event Groups (FMOD_EVENTGROUP), each of which represents a group created in the Designer application, and Groups can contain multiple sub-groups andEvents (FMOD_EVENT), which represent the individual events created with Designer.
Organizing events into groups is vital for memory management, as groups are the mechanism by which the programmer can load and unload audio data for events. Therefore, it is important to keep RAM in mind when organizing events and groups in Designer. Audio data for groups can be loaded into RAM using the function FMOD_EventGroup_LoadEventData()(EventGroup::loadEventData() in C++). If the event data has not been loaded when an event is fetched from a group, then the data for that event is loaded at that point, which will cause an roundtrip to disk and may cause a stall in the audio.
Building FMOD Designer Projects
When an FMOD Designer project is built two key files are produced, a .fev file containing the event definitions with parameter and effect info, etc, and a .fsb file containing all the audio data for the event. These are the files that the FMOD Ex engine requires to run the design. However, Designer can also optionally create a .txt report file and a .h header file at build time. The former gives a human readable breakdown of the events and their properties and the later creates a load of macro definitions for event and parameter names, which can be used in code in place of plain numerical indices, making to code much easier to read. Having these files around makes life a lot easier when coding, so even if the designer and programmer are the same person, I’d recommend generating these files each time a Designer project is (re)built. I also found it useful to set the output directory for my Designer build to a folder within my XCode project for the playback system, which saved on some file management and also ensured that when I built the XCode project I was using the most up-to-date build of the sound design.
Building a Playback System with FMOD Ex API
Once I got the FMOD dynamic libraries properly linked to my code (see my previous post), actually implementing interactive playback of the car audio system created with FMOD Designer was reasonably straight forward. I stuck with the same UDP messaging system that I’d used previously to hook the Core Audio system up to the racing game, so the structure of the playback application remained essentially the same as in that previous project:
As such I was able to reuse the UPD socket, datagram parsing and input conditioning code from the previous project, and simply replace the audio system with a new object to setup and manage the FMOD event system. The FMOD APIs come in C and C++ flavors. I stuck with C as I'm more comfortable with that than C++ and it meant that I could wrap up the C code in an Objective-C object so that it would integrate very easily with the code I reused from the previous project.
My AudioManager object receives the conditioned physics data from the game engine on each update, and uses the data to set the parameters on the various events that make up the car system. As these events are therefore being updated continuously and used throughout the gameplay, the AudioManager object keeps and instance variable for each event, and loads all the events and their audio data at startup time. The AudioManager's -init: method therefore consists of the following steps:
Get the Event System
Load the FEV into the Event System and get a pointer to the Project Object
Get the Event Group from the Project
Get Each Event from the Group.
The code looks like this:
You can see that I have not used the FMOD_EventGroup_LoadEventData() function to load the audio data for the car event group. I am calling FMOD_EventGroup_GetEvent() for each event, which causes the data to be loaded, and as all this happens at initialization time, there is no impact on audio playback.
You'll also notice that all the calls to the FMOD API are wrapped up in an error checking macro. This is just a little function I put together to log error details to the console. Here is the code:
Starting and stopping playback is simply a case of calling FMOD_Event_Start() and FMOD_Event_Stop() for each event. As the engine, exhaust, tyres and road sound events play continuously throughout gameplay each of these events are started when the Audio Manager gets the -startAudio message, like this:
When a physics update is received, the process for updating the affected events breaks down like this:
Get the relevant parameter object from the relevant event
Get the required range of the parameter value from the parameter object and constrain the incoming physics value to be within this range
Set the constrained physics value as the parameter objects value
Each of the AudioManager's update functions work in this way, for example the -setCarSpeed: method updates the speed parameter of the road noise event like this:
The crash events are the only events that are triggered by gameplay, as opposed to running throughout, however the implementation is very similar. The parameter value (in this case car speed) is updated in exactly the same way as with the looping events, and then FMOD_Event_Start(crash); is called to trigger playback.
Cleaning up after gameplay requires the audio data used for the events to be unloaded from RAM. A group's data can be removed from RAM with the FMOD_EventGroup_FreeEventData() (EventGroup::freeEventData() in C++). However, in this situation, gameplay only ends when the application is quit, so instead I simply unload the Event System in the AudioManager's -dealloc method, which unloads all projects and their audio data:
The Complete Project
Those are all the significant points from the project. Getting up and running with the Event System API is reasonably straightforward - the AudioManager was implemented in <300 lines of code. Use the links below to download the compiled audio system and the racing game to try it out, and download the audio system source code to see the full implementation. The complete project also includes the Event Network System which allows FMOD Designer to hook up to the game in real time to make adjustments to events. In my next post I'll do a direct comparison between this system and the Core Audio car implementation to see how they compare aesthetically.
Download the compiled game and audio system
Download the audio system source code
Car Audio with FMOD, Part 1: Designer
I have been rebuilding my car audio system, porting all the audio processing into the FMOD audio middleware platform. I've created the audio system in an FMOD Designer project and used the FMOD Ex API for playback of the designer events and hooking the audio up to the racing game I made with Unity (you can download the Designer app and the API libraries from the FMOD site). I'll get into the FMOD Ex API and the integration in a later post, for now here is how I developed the system in Designer.
Implementing the tyre sounds was very straightforward. I have Sound Definitions containing a set of skid samples and a set of wheel spin samples, randomizing the choice of sample on each usage. I created tyre events with layers for skid and spin and mapped volume and pitch curves against the corresponding physics values, as in my previous system.
Creating a crash was also straightforward. Sticking with the approach of having different sets of sounds for different crash velocities, I created Sound Definitions for Very Low, Low, Medium and High velocity impacts and for Medium and High velocity debris. Each of these definitions contains a few different samples to picked from at random when a crash event is triggered. The crash event is very simple, with a primary parameter of 'Speed', a layer for impact sounds and a layer for debris sounds, and a little volume enveloping. It looks like this:
Unfortunately FMOD does not include a Time Compression/Expansion (TCE) plugin, equivalent to the AUTimePitch audio unit I used to adjust the speed of the road surface loop in my previous system. I therefore took a slightly different approach in FMOD, using a combination of slight pitch shifting and low pass filtering to try to achieve the same sense of change in speed:
You can hear the FMOD version and the core audio version here for reference. The FMOD version works OK, especially is it is low in the mix, but it is definitely not as effective as the Core Audio version when accelerating and decelerating to/from stop.
Core Audio Road Noise by jimmcgowan
FMOD Road Noise by jimmcgowan
As expected, the engine and exhaust sounds where the most complicated, and took the most work. I began by simply porting over the method I used in the Core Audio system, which used samples at 1000RPM intervals, using a Varispeed audio unit to adjust speed & pitch, and with volume envelopes as shown here:
Recreating this in FMOD Designer looked like this:
And it sounded terrible!
FMOD Car Engine 1 by jimmcgowan
As you can hear, the multiple samples do not blend together as they did in the Core Audio system, each sample can be heard distinctly as it fades in and out. It seems the pitch shift effect in FMOD does not have the same fidelity as the Varispeed audio unit - the extremes are noticeably unnatural and the rhythm of the samples does not sync as it did previously. It could be that the FMOD pitch shift is trying to change pitch while preserving time, as opposed to just adjusting the sample rate, but I'm not sure. Also FMOD Designer also does not allow effect parameters to be entered as a function of an event parameter, you can only plotting points along the event parameter's range and interpolate between. This means that the envelope curves are only approximations of those from the Core Audio system, which no doubt introduced precision issues.
Given that I would not have the same scope in pitch shifting as I had in the Core Audio system, and that I clearly needed to limit the number of simultaneous playbacks, I created and new structure based on musical tones. With a series of engine samples at RPM speeds that were 1 tone apart, I could minimize the extents of pitch shifting, and have short fades between sequential samples to limit the number of playbacks. A bit of calculation gave me the following values mapping RPM intervals to 1 tone intervals, which my be save some calculator time for anyone else working on vehicles:
Tone RPM Pitch Range Pitch Length ---------------------------------------------------- -1 890.8987 min - 943.8743 943.8743 0 1000.0000 943.8743 - 1059.4631 115.5888 1 1122.4620 1059.4631 - 1189.2071 129.7440 2 1259.9210 1189.2071 - 1334.8399 145.6327 3 1414.2136 1334.8399 - 1498.3071 163.4672 4 1587.4011 1498.3071 - 1681.7928 183.4858 5 1781.7974 1681.7928 - 1887.7486 205.9558 6 2000.0000 1887.7486 - 2118.9262 231.1776 7 2244.9241 2118.9262 - 2378.4142 259.4880 8 2519.8421 2378.4142 - 2669.6797 291.2655 9 2828.4271 2669.6797 - 2996.6142 326.9344 10 3174.8021 2996.6142 - 3363.5857 366.9715 11 3563.5949 3363.5857 - 3775.4973 411.9116 12 4000.0000 3775.4973 - 4237.8524 462.3551 13 4489.8482 4237.8524 - 4756.8285 518.9761 14 5039.6842 4756.8285 - 5339.3594 582.5310 15 5656.8542 5339.3594 - 5993.2283 653.8689 16 6349.6042 5993.2283 - 6727.1713 733.9430 17 7127.1897 6727.1713 - 7550.9945 823.8232 18 8000.0000 7550.9945 - max
Unfortunately I did not have access to a garage with a dynamometer that I could run a car at such precise RPMs, so I had to improvise a little by writing some code that would spin my previous core audio system's engine at these values and dump out an audio file per value. In this way I'm stuck using 'second hand' audio, but for the purposes of this experimentation, it is good enough for now. This gave me 20 samples each for engine and exhaust sound. Sticking these into an FMOD Designer event looked like this:
This sounded really good when previewing in Designer, but once it was hooked up the the game engine, there were audible problems, which you can hear in this clip:
FMOD Car Engine 2 by jimmcgowan
As you can hear there are sudden dynamic jumps and pops as the car revs up and down. As close as I can figure out this seems to be due to the FMOD engine encountering some latency when starting sample playback (despite all the wave data being in RAM). In order to get around this I separated each sample out onto its own layer, and used a volume envelope on each to recreate the fades. However, instead of sample fading out and stopping, each sample fades down to an extremely low level and continues to play throughout, so that they do have to be stopped and restarted by the engine. In FMOD it looked like this:
The audio clip below shows that this solved the problem:
FMOD Car Engine 3 by jimmcgowan
However, this introduced a new problem. By having a total of 40 layers for engine and exhaust sounds I ran out of physical voices on my sound card, and could no longer hear crash sounds. Experimenting with disabling 1 layer at a time and rebuilding, I realized that I needed to keep engine and exhaust below 27 layers total to have enough voices for the full system. To get round this I 'doubled up' layer content as shown here:
I also wanted to make some improvements over the previous system. In particular, when I compared it with some video capture of Need For Speed: Shift (in this post), there was a really effective jump in dynamics applied and disengaged. I added a throttle parameter to the engine & exhaust events in designer and added another volume enveloped driven by this parameter. I thought that FMOD would simply sum multiple volume values for the same event (i.e. event volume is 0dB, param 1 is -3dB, param 2 is -3db, therefore output is -6dB) but this seems not to be the case. As you can hear in the next clip, multiple volume curves seems to confuse the playback engine. But this desired effect can be implemented at a code level by manipulating the input values, so I've done that instead.
FMOD Car Engine 4 by jimmcgowan
In the next few days I'll post on how I got the FMOD API up and running with the Unity game and I'll put together a side-by-side comparison of the FMOD system against the Core Audio system.
My entry for the Waves sound design competition. A full description of how the sounds were made is on the vimeo page.