Toggle Spectrogram Preview for Audio

[Path-A]

Is your feature request related to a problem? Please describe.
Classifying or segmenting audio with only a waveform preview can be time-consuming or difficult, especially with noisy audio data. Some data is more easily segmented by looking at frequency content over time.

Describe the solution you'd like
Include a toggle to preview a spectrogram representation of an audio clip. Some common python libraries to generate these are Librosa or Scipy.signal.

Describe alternatives you've considered
I've manually generated the spectrograms and saved them as images to be used within the image classification labeling tool. The downsides of this are threefold.

1. Labeling audio this way does not allow for temporal segmentation. The user must classify the entire spectrogram, not simply a vertical fraction of it. A user could, in theory, use the image annotation tool, but it would be tedious and the user would need to convert bounding boxes to its corresponding time in the audio clip.
2. The user can no longer listen to the audio clip while viewing the spectrogram image.
3. The user generated spectograms require temporary additional storage requirements.

Additional context
Each user's spectrogram needs may differ, such as their sound of interest being within the low or high frequency areas of the spectrogram. To keep implementation simple, use default spectrogram parameters that generalize well and potentially allow users to zoom in on this general spectrogram. A more robust solution would allow the user to specify a few parameters to generate the spectrogram that they would want. Lastly, I include an example of a log-scaled spectrogram with its accompanying waveform.

[makseq]

@Path-A Thank you for your issue. Are you familiar with React?

[Path-A]

@makseq Unfortunately not, although I've been wanting to learn.

[feddybear]

For future reference, I think the challenge here is more of understanding the multicanvas code of wavesurfer.
I was able to successfully use the spectrogram functions but could only implement it on the older single canvas implementation. Unfortunately, for long audio files, this is impractical because of the need to recalculate and redraw spectra when zooming, etc. If pre-segmentation is done, it becomes more practical. But segmentation of audio (especially for speech technology applications) isn't perfect.
Here's a sample demonstration as reference: LINK
It takes about 12-14 seconds per zoom value on a 3-minute file (using N=512 fft samples).

[makseq]

@feddybear Wow! It's very impressive! Do you have an account on our slack? https://label-studio.slack.com/

[feddybear]

@feddybear Wow! It's very impressive! Do you have an account on our slack? https://label-studio.slack.com/

Hi @makseq yeah I also mentioned this on one of the spectrogram inquiries there. But I'm leaving it to someone more capable, especially in reading the wavesurfer multicanvas codes. Hopefully it's also someone who knows signal processing, as the older implementation of drawing spectrogram on wavesurfer had some really weird canvas settings that didn't make sense (e.g. height of the spectrogram).

[makseq]

@feddybear Let's move to slack. I know DSP, also we'll include our frontend team there.

[makseq]

@feddybear please, tag me again (@makseq). I can't find my mention there.

[Tom-Lu]

I'm also seeking for similar feature, any progress so far?

[makseq]

@Tom-Lu
We have some news from our contributor:
https://github.com/feddybear/label-studio-frontend

I hope we will make this work to the end.

[tpeet]

Would be also very interested in this feature, as it is currently hard to select regions for audio with low SNR

[Path-A]

Has there been any progress regarding development of this feature?

[makseq]

Only if @feddybear has any news.
We are currently focusing on the image / html tagging. Audio updates are planned for next year.

[feddybear]

Sorry, I have yet to integrate the spectrogram-related edits from the previous version to the latest one. Also, kinda occupied with other stuff outside of annotation.

[Selimonder]

Hello,

Has there been any progress regarding the development of this feature?

[mikolajpabiszczak]

I know it's irritating with people asking again & again, but it would be really useful to have this, any news about this?

[makseq]

Thank you for asking. By your activity we prioritize features, so it isn't irritating :-)
@nicholasrq had some progress in Audio Plus Engine, but I heard that we haven't still implemented spectrograms :-( I will draw the attention of our team to this feature request.

[faroit]

@makseq @feddybear also 👍 for spectrograms!

[mcgee0916]

Hello,
Has there been any progress about this feature?

[cspindler]

👍 Yes, spectrogram annotation would be fantastic. But it would be just the start - with the spectrogram view available, the following features would be super useful:

• RectangleLabels in the spectrogram (time and frequency range)
• Audio playback speed control (with and without time-stretching), especially slowing down to 0.1 of original speed.
• Choices of frequency scaling (lin, log, mel, bark)
• Zoom (time and frequency)

I'm not familiar with React at all, but that could change.

We're building a pipeline to annotate bats in high-frequency audio recordings, based on batdetect2. They also have a labelling UI that checks many of the boxes UI-wise, except the handling of large amount of tasks, users, storage backends etc. - all the golden labelstud.io features.

[paulpeyret-biophonia]

Hello,

I would love to see this feature for spectrogram annotation with sound playback. There is a huge demand from all the bioacoustics and eco-acoustics community (who are still working on desktop app like audacity and raven for annotations).
@cspindler I think you well described the need.

I understand that spectrogram calculation speed is a bottleneck here for the zooming feature inside the spectrogram.
Maybe these libs can help get descent speeds.
libAudioFlux/audioFlux#22

Hopping this feature will come soon.
Cheers

[samvelkoch]

Up for specs in audio labeling

[sajarin]

/jira create

Workflow run
Jira issue TRIAG-527 is created

[DK2895]

Hi guys (@sajarin @makseq @feddybear),

Has there been any further progress on this feature? Thanks

[isaac-jordan]

+1 on wanting these spectrogram tooling. The lack of it is forcing us to consider using other tools for audio labelling.

[l4j3b]

@isaac-jordan Would you mind sharing the other tools you are considering please? I am in the same situation.

[isaac-jordan]

@isaac-jordan Would you mind sharing the other tools you are considering please? I am in the same situation.

Two front-runners for us are https://github.com/mbsantiago/whombat and https://www.wildlifeacoustics.com/products/kaleidoscope

[shemerey]

btw the repo even contains web/node_modules/wavesurfer.js/src/plugin/spectrogram/fft.js I thought that we can enable the plugin and use it, but unfortunately it looks like wavesurfer.js is not used any more.

[xixinzhang]

It's helpful but seems hard

[DK2895]

Hi there - is there any news on this or whether it is being worked on for a future release?

[JulienBrn]

Spectrogram view for audio labeling would be very helpful. We are currently using Label Studio for keypoint labeling, but we now need to annotate audio and this can not be done without a spectrogram view... we are considering other options, but we would prefer to stick with label studio.

[xvbai0317]

I seem to have accomplished this, not sure if that's the kind of mel spectrogram people want.

[jonnor]

@xvbai0317 looks good! Are you able to share the code / a branch with your changes? So that others can maybe try it out

[DK2895]

Nice work @xvbai0317, looks awesome! For features, basing them off of Audacity's built in capabilities and parameters would be useful

[xvbai0317]

It is my pleasure to help you with my work, and I will submit the finished code to a branch later. @jonnor @DK2895

[cloudmark]

Hey @xvbai0317 , I’ve been working on the spectrogram feature and came across this issue, looks like we're tackling the same thing. Just wondering if you're still planning to share a branch with your work? Happy to help out if there's anything I can do, maybe we can get it over the line together.

[cloudmark]

I'm happy to announce that the spectrogram visualization feature has been implemented and is now available in PR #7400:
#7400

The implementation includes everything from the original feature request plus additional capabilities:

• Real-time spectrogram visualization
• Multiple color schemes with live preview
• Configurable FFT parameters (64-2048 samples)
• Mel-scale frequency mapping
• Adjustable dB range (-80 to -10 dB)
• Different windowing functions (Blackman, Hann, Hamming)

I've recorded a short video demonstrating these features in action:
Video Demonstration

What the video shows:
The video demonstrates the new spectrogram feature within the Label Studio interface. You can see:

• The spectrogram appearing below the audio waveform.
• Synchronized playback, with the cursor moving across both the waveform and the spectrogram.
• How zooming interacts with both views.
• Live updates as various configuration options are changed through the settings panel, including:
  • FFT window size adjustments
  • Different color schemes being applied
  • Switching between windowing functions
  • Toggling the Mel scale on/off
  • Modifying the dB range for amplitude scaling

You can try it by adding spectrogram="true" to your AudioPlus component:

<AudioPlus
  name="audio"
  value="$audio"
  hotkey="space"
  sync="group_a"
  defaultscale="1"
  defaultzoom="2"
  zoom="true"
  spectrogram="true"
/>