Overview of Web Audio API

# Web Audio API ## Overview by Gaëtan Renaudeau - @greweb

# Summary * Audio basics * DSP * Visualizing sound * The Web Audio API * AudioNodes...

# Audio Basics

### Compression wave * Air density variation * Longitudinal, like a spring * Propagation: ~ 343 m/s in dry air, 20°C ![](./images/compressions.gif)

### Amplitude * Decibel: Difference of sound pressure ![](./images/amplitude.png)

### Frequency * Vibration speed (in Hertz = pulse per second) * human hearing range: 20 Hz to 20'000 Hz ![](./images/frequency.png)

# Digital Signal Processing

## Sampling * 44100 samples per second

## Quantization ![](./images/quantization.png)

# Visualizing sound

Waveform

Spectrum Analyzer

Spectrogram

# Web Audio API

## Abstract > a high-level JavaScript API for processing and synthesizing audio in web applications. The primary paradigm is of an **audio routing graph**, where a number of **AudioNode** objects are connected together to define the overall audio rendering. The actual processing will primarily take place in the **underlying implementation** (typically optimized Assembly / C / C++ code), but **direct JavaScript** processing and synthesis is also supported. [spec](https://dvcs.w3.org/hg/audio/raw-file/tip/webaudio/specification.html)

## Basic Modular Routing ![](./images/modular-routing1.png)

## Modular Routing ![](./images/modular-routing2.png)

## AudioContext ``` interface AudioContext : EventTarget { readonly attribute AudioDestinationNode destination; readonly attribute float sampleRate; readonly attribute double currentTime; readonly attribute AudioListener listener; AudioBufferSourceNode createBufferSource(); MediaElementAudioSourceNode createMediaElementSource(HTMLMediaElement mediaElement); MediaStreamAudioSourceNode createMediaStreamSource(MediaStream mediaStream); MediaStreamAudioDestinationNode createMediaStreamDestination(); ScriptProcessorNode createScriptProcessor(optional unsigned long bufferSize = 0, optional unsigned long numberOfInputChannels = 2, optional unsigned long numberOfOutputChannels = 2); AnalyserNode createAnalyser(); GainNode createGain(); DelayNode createDelay(optional double maxDelayTime = 1.0); BiquadFilterNode createBiquadFilter(); WaveShaperNode createWaveShaper(); PannerNode createPanner(); ConvolverNode createConvolver(); ChannelSplitterNode createChannelSplitter(optional unsigned long numberOfOutputs = 6); ChannelMergerNode createChannelMerger(optional unsigned long numberOfInputs = 6); DynamicsCompressorNode createDynamicsCompressor(); OscillatorNode createOscillator(); PeriodicWave createPeriodicWave(Float32Array real, Float32Array imag); AudioBuffer createBuffer(unsigned long numberOfChannels, unsigned long length, float sampleRate); void decodeAudioData(ArrayBuffer audioData, DecodeSuccessCallback successCallback, optional DecodeErrorCallback errorCallback); }; ```

## TL;DR. * AudioContext allows to create different **AudioNode**. * \+ constants: *sampleRate*, *currentTime*, *destination*

## different AudioNodes * GainNode * OscillatorNode * BiquadFilterNode * DynamicsCompressorNode * DelayNode * ConvolverNode * WaveShaperNode * PannerNode, ChannelSplitterNode, ChannelMergerNode * AnalyserNode * AudioBufferSourceNode * MediaElementAudioSourceNode, MediaStreamAudioSourceNode, MediaStreamAudioDestinationNode * ScriptProcessorNode

## AudioNode ``` interface AudioNode : EventTarget { void connect(AudioNode destination, optional unsigned long output = 0, optional unsigned long input = 0); void connect(AudioParam destination, optional unsigned long output = 0); void disconnect(optional unsigned long output = 0); readonly attribute AudioContext context; readonly attribute unsigned long numberOfInputs; readonly attribute unsigned long numberOfOutputs; // Channel up-mixing and down-mixing rules for all inputs. attribute unsigned long channelCount; attribute ChannelCountMode channelCountMode; attribute ChannelInterpretation channelInterpretation; }; ```

## TL;DR. * an AudioNode is an abstract interface. * an AudioNode can be connected to an **AudioNode**. * an AudioNode can be connected to an **AudioParam**. * an AudioNode generally have different **AudioParam**s.

## AudioParam ``` interface AudioParam { attribute float value; readonly attribute float defaultValue; void setValueAtTime(float value, double startTime); void linearRampToValueAtTime(float value, double endTime); void exponentialRampToValueAtTime(float value, double endTime); void setTargetAtTime(float target, double startTime, double timeConstant); void setValueCurveAtTime(Float32Array values, double startTime, double duration); void cancelScheduledValues(double startTime); }; ```

## TL;DR. * an AudioParam represents a **value**. * You can set/get it with the **value** field. * You can **schedule** the value (setValueAtTime). * You can **automate** the value. (linearRamp, exponentialRamp)

## Examples of value automation ![](./images/audioparam-automation1.png)

## GainNode The simplest and most used AudioNode. It amplifies/reduces the amplitude of the audio stream. ``` interface GainNode : AudioNode { readonly attribute AudioParam gain; }; ```

## OscillatorNode Fundamental periodic waveform. ``` enum OscillatorType { "sine", "square", "sawtooth", "triangle", "custom" }; interface OscillatorNode : AudioNode { attribute OscillatorType type; readonly attribute AudioParam frequency; // in Hertz readonly attribute AudioParam detune; // in Cents void start(double when); void stop(double when); void setPeriodicWave(PeriodicWave periodicWave); attribute EventHandler onended; }; ```

Oscillator


var osc = ctx.createOscillator();
osc.type = "sine"; // "triangle" or "sawtooth" or "square"
osc.frequency = 440;
osc.start(0);
osc.connect(ctx.destination);

Oscillator Frequency Modulation


var carrier = ctx.createOscillator(); carrier.start(0);
var modulator = ctx.createOscillator(); modulator.start(0);
var modGain = ctx.createGain(); modGain.gain.value = 440;
modulator.connect(modGain);
modGain.connect(carrier.frequency);
carrier.connect(ctx.destination);

Oscillator Frequency Modulation

## ADSR Envelope ![](./images/ADSR_parameter.png)

### Envelope is just automation on a Gain ```javascript function noteEnvelope (gainNode, time, volume, a, d, s, r) { var ctx = gainNode.context; var gain = gainNode.gain; gain.value = 0; gain.cancelScheduledValues(0); gain.setValueAtTime(0, time); gain.linearRampToValueAtTime(volume, time + a); gain.linearRampToValueAtTime(volume * s, time + a + d); return function (t) { gain.cancelScheduledValues(0); gain.setValueAtTime(gain.value, t); gain.linearRampToValueAtTime(0, t + r); return Q.delay(1000*(0.1 + (t+r) - ctx.currentTime)); }; } ```

Envelope + Oscillator

QWERTY keyboard is used by default. try 'ZXCVBNM' and 'QWERTYU' keys. Black keys are also supported and you can change octaves with "." and "/" keys. use ?azerty=1 in the URL if you prefer AZERTY. MIDI keyboard is also supported! (today needs Chrome Canary with flags)

FM Oscillator + 2 envelopes

## AudioBufferSourceNode Play a sample. ``` interface AudioBufferSourceNode : AudioNode { attribute AudioBuffer? buffer; readonly attribute AudioParam playbackRate; attribute boolean loop; attribute double loopStart; attribute double loopEnd; void start(optional double when = 0, optional double offset = 0, optional double duration); void stop(optional double when = 0); attribute EventHandler onended; }; ```

AudioBufferSourceNode

## BiquadFilterNode ``` enum BiquadFilterType { "lowpass", "highpass", "bandpass", "lowshelf", "highshelf", "peaking", "notch", "allpass" }; interface BiquadFilterNode : AudioNode { attribute BiquadFilterType type; readonly attribute AudioParam frequency; // in Hertz readonly attribute AudioParam detune; // in Cents readonly attribute AudioParam Q; // Quality factor readonly attribute AudioParam gain; // in Decibels void getFrequencyResponse(Float32Array frequencyHz, Float32Array magResponse, Float32Array phaseResponse); }; ```

Filter

N.B. The Gain only works with lowshelf, highshelf and peaking. Also it is protected by a Compressor to not break your ears!

FM + Filter Cutoff LFO

## DynamicsCompressorNode A **Compressor** dynamically normalizes the sound: lowers the volume of the loudest parts. raises the volume of the softest parts. ``` interface DynamicsCompressorNode : AudioNode { readonly attribute AudioParam threshold; // in Decibels readonly attribute AudioParam knee; // in Decibels readonly attribute AudioParam ratio; // unit-less readonly attribute AudioParam reduction; // in Decibels readonly attribute AudioParam attack; // in Seconds readonly attribute AudioParam release; // in Seconds }; ```

Compressor

## Making Noise ``` function Noise (ctx) { var bufferSize = 2 * ctx.sampleRate, noiseBuffer = ctx.createBuffer(1, bufferSize, ctx.sampleRate), output = noiseBuffer.getChannelData(0); for (var i = 0; i < bufferSize; i++) { output[i] = Math.random() * 2 - 1; } var whiteNoise = ctx.createBufferSource(); whiteNoise.buffer = noiseBuffer; whiteNoise.loop = true; this.out = whiteNoise; } ``` ![](./images/noise.png)

## Protecting with Filter ``` { // ... var gain = ctx.createGain(); whiteNoise.connect(gain); var filter = ctx.createBiquadFilter(); gain.connect(filter); filter.type = "lowpass"; this.white = whiteNoise; this.gain = gain; this.out = this.filter = filter; } ``` ![](./images/noise_filtered.png)

Noise + Filter

## DelayNode A DelayNode delays input to a given time. ``` interface DelayNode : AudioNode { readonly attribute AudioParam delayTime; }; ```

Delay effect example

"Repeater"

## ConvolverNode > ConvolverNode applies a linear convolution effect given an impulse response. ![](./images/convolution.gif)

![](./images/music.png) ![](./images/impulse.png) ![](./images/music_reverbed.png)

Convolution

You can find some Impulse Response on the internet (e.g. here), or you can record it yourself!

## ChannelSplitterNode ![](./images/channel-splitter.png)

## ChannelMergerNode ![](./images/channel-merger.png)

## Making Stereo Echo Effects ``` // Schema: var input = ctx.createGain(); var left = ctx.createGain(); var right = ctx.createGain(); var delayL = ctx.createDelay(); var delayR = ctx.createDelay(); var delayGainL = ctx.createGain(); var delayGainR = ctx.createGain(); input.connect(left); input.connect(right); left.connect(delayL); left.connect(delayR); delayL.connect(delayGainL); delayR.connect(delayGainR); delayGainL.connect(left); delayGainR.connect(right); var merger = ctx.createChannelMerger(); left.connect(merger, 0, 0); right.connect(merger, 0, 1); // Basic Usage: delayL.delayTime.value = 0.2; // second delayL.delayTime.value = 0.3; // second delayGainL.gain.value = 0.3; delayGainR.gain.value = 0.4; anything.connect(input); merger.connect(ctx.destination); ```

Stereo echo

## WaveShaperNode > WaveShaperNode is an AudioNode processor implementing non-linear distortion effects. > Non-linear waveshaping distortion is commonly used for both subtle non-linear warming, or more obvious distortion effects. Arbitrary non-linear shaping curves may be specified. *(no example yet)*

## ScriptProcessorNode > This interface is an AudioNode which can generate, process, or analyse audio directly using JavaScript.

## ScriptProcessorNode Noise Implementation ``` var node = ctx.createScriptProcessor(1024, 1, 1); node.onaudioprocess = function (e) { var output = e.outputBuffer.getChannelData(0); for (var i = 0; i < output.length; i++) { output[i] = Math.random(); } }; node.connect(ctx.destination); ``` > N.B. It is recommended to use a buffer noise loop approach for better performances. **Indeed, ScriptProcessorNode can be CPU consuming.**

# Final words

## Web Audio API is Awesome! * Audio Nodes are simple but useful * **Audio Modular** is very powerful * Very efficient

## Cons * **Implementing by Hand is quite heavy** * A lot of LoC for doing your stuff * counter-productive for testing / adjusting * solution: making a node editor (Zound goal) * **Today's support** * Chrome Desktop: ok, still missing the `PeriodicWave`. * Chrome Mobile: it works, but not so efficient. * Firefox: only in Aurora, still in WIP

I made a game!

http://greweb.me/2013/09/timelapse/

## Questions? [@greweb](http://twitter.com/greweb)