Audio & Speech Models
Audio AI spans speech recognition (transcription), text-to-speech (generation), speaker identification, sound event detection, music generation, and audio understanding. Modern models like OpenAI Whisper, ElevenLabs TTS, and AudioCraft have dramatically improved quality and accessibility of audio AI.
The field has converged towards unified architectures that process multiple audio tasks: Whisper does transcription, translation, and language identification; Meta's AudioCraft handles music generation, sound generation, and compression. Multimodal models increasingly process audio alongside text and images.
OpenAI Whisper is a general-purpose speech recognition model trained on 680,000 hours of multilingual data. It supports transcription (speech to text in the same language), translation (speech to English text), and language identification. Whisper handles background noise, accents, and diverse audio quality better than previous models.
| 1 | # Whisper — transcription via API |
| 2 | from openai import OpenAI |
| 3 | |
| 4 | client = OpenAI() |
| 5 | |
| 6 | # Audio transcription (same language) |
| 7 | response = client.audio.transcriptions.create( |
| 8 | model="whisper-1", |
| 9 | file=open("meeting.mp3", "rb"), |
| 10 | response_format="verbose_json", |
| 11 | timestamp_granularities=["segment"], |
| 12 | ) |
| 13 | |
| 14 | print(response.text) |
| 15 | # Each segment has start/end timestamps |
| 16 | for segment in response.segments: |
| 17 | print(f"[{segment.start:.1f}s -> {segment.end:.1f}s] {segment.text}") |
| 18 | |
| 19 | # Translation to English |
| 20 | response = client.audio.translations.create( |
| 21 | model="whisper-1", |
| 22 | file=open("spanish_audio.mp3", "rb"), |
| 23 | ) |
| 24 | |
| 25 | # Whisper local inference (open-source model) |
| 26 | import whisper |
| 27 | |
| 28 | model = whisper.load_model("large-v3") |
| 29 | result = model.transcribe( |
| 30 | "audio.mp3", |
| 31 | language="en", |
| 32 | temperature=0.0, |
| 33 | word_timestamps=True, |
| 34 | ) |
| 35 | |
| 36 | print(result["text"]) |
| 37 | for seg in result["segments"]: |
| 38 | print(f"{seg['start']:.1f}s: {seg['text']}") |
info
Modern TTS models produce natural, expressive speech with controllable pace, pitch, and emotion. ElevenLabs offers the most realistic voices with voice cloning from short samples. OpenAI TTS provides six built-in voices with two quality tiers. Open-source options include Coqui TTS and Bark.
| 1 | # OpenAI TTS |
| 2 | from openai import OpenAI |
| 3 | import io |
| 4 | |
| 5 | client = OpenAI() |
| 6 | |
| 7 | response = client.audio.speech.create( |
| 8 | model="tts-1-hd", # High quality |
| 9 | voice="nova", # alloy, echo, fable, nova, onyx, shimmer |
| 10 | input="Hello, welcome to the audio AI guide. Today we'll explore speech synthesis.", |
| 11 | speed=1.0, |
| 12 | ) |
| 13 | |
| 14 | # Stream to file |
| 15 | response.stream_to_file("output.mp3") |
| 16 | |
| 17 | # ElevenLabs TTS |
| 18 | import elevenlabs |
| 19 | from elevenlabs.client import ElevenLabs |
| 20 | |
| 21 | client = ElevenLabs(api_key="your-key") |
| 22 | |
| 23 | # Generate speech with a premade voice |
| 24 | audio = client.generate( |
| 25 | text="This is generated speech from ElevenLabs.", |
| 26 | voice="Rachel", |
| 27 | model="eleven_multilingual_v2", |
| 28 | ) |
| 29 | |
| 30 | # Voice cloning from audio sample |
| 31 | voice = client.clone( |
| 32 | name="Custom Voice", |
| 33 | files=["sample.mp3"], |
| 34 | description="Natural speaking voice", |
| 35 | ) |
| 36 | audio = client.generate( |
| 37 | text="This is a cloned voice speaking.", |
| 38 | voice=voice.voice_id, |
| 39 | ) |
| 40 | |
| 41 | # Play audio |
| 42 | elevenlabs.play(audio) |
Sound event detection identifies non-speech audio events (glass breaking, dog barking, car engine). AudioCraft (MusicGen + AudioGen) from Meta generates music and sound effects from text descriptions. These models use audio tokenization and transformer architectures similar to text generation.
| 1 | # AudioCraft — music and sound generation |
| 2 | from audiocraft.models import MusicGen, AudioGen |
| 3 | |
| 4 | # Music generation |
| 5 | music_model = MusicGen.get_pretrained("melody") |
| 6 | music_model.set_generation_params(duration=8) |
| 7 | |
| 8 | descriptions = [ |
| 9 | "upbeat electronic dance music with heavy bass", |
| 10 | "calm classical piano piece with strings", |
| 11 | ] |
| 12 | |
| 13 | wav = music_model.generate(descriptions) |
| 14 | # wav shape: [batch, channels, samples] |
| 15 | |
| 16 | # Sound effect generation |
| 17 | sound_model = AudioGen.get_pretrained("facebook/audiogen-medium") |
| 18 | sound_model.set_generation_params(duration=5) |
| 19 | |
| 20 | sounds = sound_model.generate([ |
| 21 | "dog barking in the distance", |
| 22 | "car engine starting and driving away", |
| 23 | ]) |
| 24 | |
| 25 | # Sound event classification |
| 26 | from transformers import AutoModelForAudioClassification |
| 27 | import torchaudio |
| 28 | |
| 29 | model = AutoModelForAudioClassification.from_pretrained( |
| 30 | "MIT/ast-finetuned-audioset-10-10-0.2" |
| 31 | ) |
| 32 | waveform, sample_rate = torchaudio.load("sound.wav") |
| 33 | inputs = processor(waveform, sampling_rate=sample_rate, return_tensors="pt") |
| 34 | outputs = model(**inputs) |
| 35 | predicted_class_id = outputs.logits.argmax(-1).item() |
| 36 | label = model.config.id2label[predicted_class_id] |
- Whisper provides state-of-the-art transcription with multilingual support and speaker detection
- TTS models (ElevenLabs, OpenAI) produce natural speech with voice cloning capabilities
- AudioCraft generates music and sound effects from text descriptions
- Pre-process audio to 16kHz mono WAV for optimal recognition accuracy
- Combine ASR + TTS for voice chatbots and real-time translation pipelines
Community
Get help on Slack, Discord or VIP
Stuck on a guide? Join the community and ask.