GitHub user Matthew-DWM created a discussion: Proposal: Audio Attribute 
Extractor, measuring audio files in a workflow

## Summary

A new operator that reads audio files from a binary column, as File Scan 
produces them, and measures one property of the sound over time: its waveform, 
loudness, noisiness, pitch, onsets or beats. Each file becomes rows of `x` 
(time in seconds), `y` (the value) and `series` (the property's name), so the 
existing chart, aggregate and join operators work on it unchanged.

The idea comes from issue #3762. A prototype of the feature exists at PR #9146 
with a demo video and additional details.

## Motivation

Texera can already load audio files as binary data, and the Hugging Face 
operator can send audio to a hosted model for speech recognition or 
classification. What's missing is a way to look at the sound itself. Today that 
takes a Python UDF that brings its own decoder and signal-processing code.

Some uses:

- Compare a folder of recordings by tempo, loudness or how busy each one is, 
joined with their metadata.
- Follow the pitch of a voice or an instrument over time.
- Find quiet, loud or noisy stretches, for example to check recordings before 
sending them to a speech model.
- Count sound events, such as notes or taps, per file.

## Approach

**For the user.** One operator in Utilities. Pick the audio column (WAV, MP3 or 
FLAC) and one attribute:

| Attribute | `y` |
|---|---|
| `waveform` | the signal |
| `rms` | loudness of each frame |
| `zcr` | zero-crossing rate: high for noise, low for tones |
| `pitch_hz` | pitch, 60 to 1000 Hz |
| `onsets` | 1.0 at each new sound |
| `beats` | 1.0 at each beat |

The other settings are the sample rate, the frame and hop length, a cap on rows 
and seconds per file, keeping the input's other columns (such as the filename), 
and skipping unreadable files instead of stopping.

**How it runs.** Like the other Python-based operators: the Scala descriptor 
checks the settings when the workflow compiles and generates the Python a 
worker runs. The worker decodes each file, mixes it to mono (single audio 
stream), resamples it, measures it and emits a bounded number of rows per file.

**Dependencies.** One new Python package, `miniaudio` (MIT), which brings 
`cffi` (MIT-0) and `pycparser` (BSD-3). NumPy and SciPy are already there. 

## Prototype

#9146 has the operator (about 1,350 lines, most of it the Python the worker 
runs), 68 tests, a docs page and a recording of an example workflow. On my 
machine, a four-minute WAV takes at most about 1.2 s per attribute.

## Open questions

1. Would you find an audio operator useful?
2. Is a new Python dependency acceptable (`miniaudio`, plus `cffi` and 
`pycparser`)?
3. Should the scope of the current PR be reduced and split into multiple PRs?
4. Is use of the librosa-derived code acceptable?


GitHub link: https://github.com/apache/texera/discussions/9276

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to