Piper Audio Streaming Web Worker Example

Summary

I was looking into how to compile VAMP audio analysis plugins into WebAssembly modules when I stumbled upon Piper Audio. The Piper Audio documentation was sparse, which is why I created this minimal example.

Introduction

In Machine Learning, Feature Extraction is the attempt to transfer some form of input data (text, image, audio, etc.) into a, supposedly, more compact and more appropriate format (features) for the task at hand.

The VAMP system defines a standard plugin API for feature extraction from audio signals. The most prominent VAMP host making use of this API to visualise the traits of an audio signal is Sonic Visualiser. Both the VAMP API and Sonic Visualiser have been developed at the center for digital Music (C4DM), a research group of the Queen Mary University of London (QMUL).

Piper Audio is/was a

project of C4DM to explore audio analysis using browser technology, Javascript, and network protocols.

It was introduced in L. Thompson, C. Cannam, and M. Sandler, “Piper: Audio Feature Extraction in Browser and Mobile Applications,” in Proceedings of 3rd Web Audio Conference, London, August 2017, 2017 (Video) and comprises

a protocol schema
C++ support code for Piper including adaptation to Vamp plugin SDK
an adapter to turn Vamp plugins into Piper modules, usually with Emscripten
piper-js, a library for building Piper servers and clients with Javascript
a proof of concept demo called ugly duckling (Source)
build scripts for a set of existing VAMP plugins

The ugly duckling is a rather complex Angular app written in TypeScript (which is great).

The minimal demo at hand uses piper-js and the QM Note Onset Detector to estimate note onset times of an audio file. It is written in Javascript (ES7) and depends only on piper-js.

Code

The code uses webpack to create two bundles:

worker.js, the script executed by a Web Worker and
index.js, which starts the web worker and a client talking to it.

The streaming web worker server

import {
  QMVampPlugins
} from './QMVampPluginsModule'
import {
  EmscriptenService
} from 'piper-js/emscripten'
import {
  PiperStreamingService
} from 'piper-js/streaming'
import {
  WebWorkerStreamingServer
} from 'piper-js/web-worker'

const qmService = new EmscriptenService(QMVampPlugins())
const streamingService = new PiperStreamingService(qmService)
new WebWorkerStreamingServer(self, streamingService)

The streaming web worker client

import { 
  countingIdProvider,
  WebWorkerStreamingClient
} from 'piper-js/web-worker'
import {
  collect
} from 'piper-js/streaming'

const qmPluginsServer = new Worker('worker.bundle.js')
const piperClient = new WebWorkerStreamingClient(qmPluginsServer, countingIdProvider(0))

function extractOnsetFeatures(audioBuffer) {
  const extractionRequest = {
    audioData: [...Array(audioBuffer.numberOfChannels).keys()]
    .map(i => audioBuffer.getChannelData(i)),
    audioFormat: {
      sampleRate: audioBuffer.sampleRate,
      channelCount: audioBuffer.numberOfChannels,
      length: audioBuffer.length
    },
    key: 'qm-vamp-plugins:qm-onsetdetector',
    outputId: 'onsets'
  }

  let percent = 0

  return collect(piperClient.process(extractionRequest), (streamingResponse) => {
    const currentPercent = Math.round(100.0 * streamingResponse.progress.processedBlockCount / streamingResponse.progress.totalBlockCount)

    // reduce the amount of progress updates
    if (currentPercent > percent) {
      percent = currentPercent
      updateProgress(percent)
    }
  })
}

Building

npm install
npm run build
npm run serve

Licence

QMVampPlugins module:

Example code: