music-assistant-server

1# Streams Controller Architecture
2
3This document provides an overview of the Music Assistant Streams Controller architecture, including audio buffering, streaming pipeline, and smart fades.
4
5## Table of Contents
6
7- [Overview](#overview)
8- [Network Architecture](#network-architecture)
9- [Inbound Audio](#inbound-audio)
10- [Core Components](#core-components)
11- [AudioBuffer](#audiobuffer)
12- [StreamsAudio](#streamsaudio)
13- [Streaming Pipeline](#streaming-pipeline)
14- [Analyze Callbacks](#analyze-callbacks)
15- [Smart Fades](#smart-fades)
16- [Audio Overlay](#audio-overlay)
17- [Stream Types](#stream-types)
18- [Configuration](#configuration)
19
20## Overview
21
22The Streams Controller is a core controller that manages all audio streaming to players. It provides:
23- HTTP streaming endpoints for players on the local network
24- Audio buffering with configurable memory usage
25- Volume normalization (dynamic, measurement-based, and fixed gain)
26- Smart crossfading between tracks
27- Flow mode for continuous queue playback
28- Audio overlay: a looping sound effect (e.g. rain) mixed into queue playback
29- Announcement and plugin source streaming
30- Ahead-of-time audio analysis (loudness, beat detection) via buffer callbacks
31
32## Network Architecture
33
34The streams controller runs its own dedicated HTTP-only webserver on a separate port (default 8097), independent of the main webserver/API. This design is intentional:
35
36- **No SSL/TLS**: Many audio players (especially embedded devices) have limited resources and struggle with SSL handshakes. Since the stream server only runs on the internal network, encryption is unnecessary.
37- **No authentication**: Players need to access streams without credentials. Instead, stream URLs include a **session ID** that is validated on each request to prevent stale or invalid stream attempts.
38- **Separate port**: Keeps audio streaming isolated from the API, allowing independent scaling and configuration.
39
40## Inbound Audio
41
42Live announcements (`live_announcements.py`) are the one path where audio travels *into* the stream server rather than out of it: a client pushes raw PCM while a user speaks, and it is played on a player as an ordinary announcement.
43
44This splits across both webservers, because neither can do the job alone:
45
46- The **inbound** half is a WebSocket on the main webserver. Audio from a client is a privileged action, so it needs the authentication and the SSL support that the stream server deliberately does not have. Browsers additionally require a secure context to reach a microphone at all, which only the main webserver can offer.
47- The **outbound** half is an ordinary stream server route serving the buffered speech as a WAV. The announcement renderer only ever pulls its audio from a URL, so exposing the clip as one keeps live announcements on exactly the same path as every other announcement.
48
49The announcement is dispatched only once the clip is complete, not while it is still being spoken. Players that announce natively need the whole clip up front: AirPlay renders it to a file and schedules a single synchronized instant across every group member from its exact duration, and Sonos needs the duration to know how long the clip runs. Handing them a clip that is still growing gives one player type a head start and truncates another, so every player gets the same finished clip instead.
50
51A session is identified by an unguessable id that appears only in the stream URL, and it is dropped as soon as the announcement has been played.
52
53## Core Components
54
55```
56controllers/streams/
57  __init__.py          - Package init, exports StreamsController
58  controller.py        - StreamsController: HTTP endpoints, public streaming API
59  audio.py             - StreamsAudio: audio processing, stream acquisition, DSP/filters
60  audio_buffer.py      - AudioBuffer: in-memory PCM audio buffering with seek support
61  constants.py         - Shared constants (buffer sizes, config keys)
62  ogg_handler.py       - Chained OGG stream stitching for radio
63  smart_fades/         - Smart crossfade detection and mixing
64    analyzer.py        - Beat analysis for smart fade detection
65    fades.py           - Fade curve generation
66    mixer.py           - Crossfade mixing logic
67```
68
69Supporting modules in `helpers/`:
70- `helpers/audio.py` - Generic audio utilities (PCM helpers, format conversions, silence stripping)
71- `helpers/ffmpeg.py` - FFmpeg process management
72
73## AudioBuffer
74
75`AudioBuffer` is the primary interface for all buffered audio streaming. It stores **raw decoded PCM audio** (no filters applied) and serves as the single source of truth for audio data.
76
77### Design Principles
78
791. **Always-on buffering**: Every queue stream (tracks and radio) goes through an AudioBuffer
802. **Raw PCM only**: The buffer stores decoded audio in original sample rate and bit depth. Filters (volume normalization, playback speed, etc.) are applied when reading via `get_stream()`
813. **Pre-initialization**: Buffers are created and start filling before the player requests the stream, ensuring immediate playback start
824. **Buffer reuse**: Existing valid buffers are reused for seek operations and reconnections
835. **Smart seeking**: Forward seeks within 20 seconds of buffered data wait for the producer; larger seeks trigger a re-fetch at the seek position
84
85### Buffer Modes
86
87- **SEEKABLE** (tracks): Maintains a deque of 1-second PCM chunks with seek support. Old chunks are discarded when the buffer reaches max size
88- **ROLLING** (radio/non-seekable): Short FIFO buffer (~15 seconds) where the consumer pops chunks sequentially
89
90### Key Methods
91
92- `AudioBuffer.get_buffer()` - Static factory that creates or reuses a buffer. Reads config, determines mode, starts the analysis reader, starts filling
93- `AudioBuffer.get_stream()` - Get processed audio with optional filters/resampling applied
94- `AudioBuffer.get_raw_stream()` - Get unprocessed raw PCM audio (playback consumer)
95- `AudioBuffer.read_chunk_for_analysis()` - Read one chunk for a passive analysis reader without mutating the buffer; raises when the chunk has been evicted (reader fell behind)
96- `AudioBuffer.fill()` - Start filling from an async generator of PCM chunks
97- `AudioBuffer.ready` - Event set when enough chunks are buffered past the seek point (threshold-based)
98
99### Buffer Lifecycle
100
101```
1021. _load_item() fetches stream details, creates buffer with wait_ready=True
1032. Buffer starts filling from get_media_stream() in background
1043. Analysis (loudness, smart fades) reads the same buffer in parallel, at lower priority
1054. Player requests stream -> get_queue_item_stream() calls buffer.get_stream()
1065. 60s before the end of the source stream: prepare_next_audio_buffer() pre-fills next track
1076. _cleanup_stale_queue_buffers() clears old buffers to free memory
108```
109
110### Error Handling
111
112- Producer errors are captured and surfaced when consumers try to read
113- Consumers can drain remaining buffered data before the error surfaces at EOF
114- Errors bubble up as `AudioError` through the streaming chain
115
116## StreamsAudio
117
118`StreamsAudio` is the audio processing sub-controller, initialized as `self.audio` on the StreamsController. It handles all audio-related logic that needs access to the MusicAssistant instance:
119
120- **Stream acquisition**: `get_media_stream`, `get_stream_details`, radio/HTTP/file stream helpers
121- **Queue streaming**: `get_queue_item_stream`, `get_queue_item_stream_with_smartfade`, `get_queue_flow_stream`
122- **Format selection**: `get_output_format`, `select_pcm_format`, `select_flow_format`
123- **DSP and output plans**: `get_player_output_plan`, `get_player_dsp_details`, `get_stream_dsp_details`
124- **Crossfade management**: `crossfade_allowed`, `clear_crossfade_handover`
125- **Loudness analysis**: `attach_loudness_analyzer` (via buffer callbacks)
126
127`AudioProcessingManager`, initialized as `self.audio_processing` on the
128StreamsController, combines queue processing and per-player output plans into complete
129`AudioProcessingChain` snapshots attached to `StreamDetails`.
130
131## Streaming Pipeline
132
133```
134Music Provider -> get_media_stream() -> FFmpeg (decode to raw PCM)
135    -> AudioBuffer (raw PCM storage, analyze callbacks run here)
136    -> buffer.get_stream() -> Optional: FFmpeg (volume normalization, speed, fade-in)
137    -> Optional: Smart Fades (crossfade mixing between tracks)
138    -> FFmpeg (encode to output format with player-specific DSP)
139    -> HTTP Response / Direct PCM stream
140```
141
142### Stream Entry Points
143
1441. **HTTP endpoints** (`serve_queue_item_stream`, `serve_queue_flow_stream`): Used by players that consume HTTP streams (Chromecast, DLNA, Sonos, etc.)
1452. **Direct PCM** (`get_stream`): Used by player providers that consume raw PCM directly (AirPlay, Sendspin, etc.)
146
147## Analyze Callbacks
148
149AudioBuffer supports registering chunk callbacks that receive raw PCM data as it flows into the buffer. This enables ahead-of-time analysis without re-streaming:
150
151### Loudness Measurement
152- Attached automatically when a new buffer is created (tracks and radio)
153- Feeds up to 2 minutes of PCM into an FFmpeg `ebur128` process
154- Result stored for future volume normalization (avoids dynamic mode overhead)
155
156### Smart Fades Beat Analysis
157- Attached automatically for music tracks (MediaType.TRACK only, not podcasts/audiobooks)
158- Collects first 45 seconds (intro) and last 45 seconds (outro) of audio
159- Triggers librosa beat detection in a background thread
160- Results cached for crossfade timing decisions
161
162Both analyzers check for existing measurements before starting, avoiding redundant work.
163
164## Smart Fades
165
166The smart fades system provides intelligent crossfading between tracks:
167
168- **Smart Crossfade**: Analyzes audio beats to detect natural fade points
169- **Standard Crossfade**: Fixed-duration overlap crossfade with silence stripping
170- Operates in both flow mode (continuous stream) and per-item mode (gapless playback)
171
172## Audio Overlay
173
174The audio overlay is a per-queue feature (configured via `player_queues/overlay`) that mixes a
175looping sound effect — any `sound_effect` media item offered by a provider — into the queue's
176audio stream:
177
178- Mixing happens once per queue stream (ffmpeg `amix`, overlay looped via `-stream_loop -1`),
179  so all (synced) players consuming the stream hear the identical mix.
180- An active overlay forces flow mode: the overlay must play continuously across track
181  boundaries, which is impossible with per-item stream requests. Radio is the exception —
182  it always plays as a single long-lived stream and is wrapped per-request instead.
183- The internal PCM format is upgraded to F32 (like crossfade/DSP) for clipping-free headroom.
184- Failures degrade gracefully: when the overlay source can not be resolved, playback simply
185  continues without overlay; when the overlay input dies mid-stream, ffmpeg keeps passing
186  the main audio. Music playback is never interrupted by the overlay.
187- Note: audio already sitting in a player's (pre)buffer is unaffected by overlay changes,
188  which is why the queue controller restarts playback on an audible change. For the same
189  reason a seek can momentarily shift the overlay position — acceptable for ambient content.
190
191## Stream Types
192
193| Type | AudioBuffer | Description |
194|------|-------------|-------------|
195| Queue tracks | Yes (SEEKABLE) | Regular track playback with full buffering |
196| Radio streams | Yes (ROLLING) | Short rolling buffer, non-seekable |
197| Announcements | Yes (SEEKABLE) | Short one-off audio (TTS), rendered once and shared by all consumers |
198| Plugin sources | No | Real-time audio (microphone, aux), streamed directly |
199
200## Configuration
201
202Key configuration entries (in streams controller config):
203
204| Key | Type | Default | Description |
205|-----|------|---------|-------------|
206| `buffer_size` | String | Memory-dependent (`maximum` >=8GB, `balanced` >=4GB, `minimal` <4GB) | Audio buffer size preset |
207| `volume_normalization_radio` | String | `fallback_dynamic` | Normalization mode for radio |
208| `volume_normalization_tracks` | String | `fallback_dynamic` | Normalization mode for tracks |
209| `allow_crossfade_same_album` | Boolean | `false` | Whether to crossfade consecutive album tracks |
210