music-assistant-server

1# Streams Controller Architecture
2
3This document provides an overview of the Music Assistant Streams Controller architecture, including audio buffering, streaming pipeline, and smart fades.
4
5## Table of Contents
6
7- [Overview](#overview)
8- [Network Architecture](#network-architecture)
9- [Core Components](#core-components)
10- [AudioBuffer](#audiobuffer)
11- [StreamsAudio](#streamsaudio)
12- [Streaming Pipeline](#streaming-pipeline)
13- [Analyze Callbacks](#analyze-callbacks)
14- [Smart Fades](#smart-fades)
15- [Audio Overlay](#audio-overlay)
16- [Stream Types](#stream-types)
17- [Configuration](#configuration)
18
19## Overview
20
21The Streams Controller is a core controller that manages all audio streaming to players. It provides:
22- HTTP streaming endpoints for players on the local network
23- Audio buffering with configurable memory usage
24- Volume normalization (dynamic, measurement-based, and fixed gain)
25- Smart crossfading between tracks
26- Flow mode for continuous queue playback
27- Audio overlay: a looping sound effect (e.g. rain) mixed into queue playback
28- Announcement and plugin source streaming
29- Ahead-of-time audio analysis (loudness, beat detection) via buffer callbacks
30
31## Network Architecture
32
33The streams controller runs its own dedicated HTTP-only webserver on a separate port (default 8097), independent of the main webserver/API. This design is intentional:
34
35- **No SSL/TLS**: Many audio players (especially embedded devices) have limited resources and struggle with SSL handshakes. Since the stream server only runs on the internal network, encryption is unnecessary.
36- **No authentication**: Players need to access streams without credentials. Instead, stream URLs include a **session ID** that is validated on each request to prevent stale or invalid stream attempts.
37- **Separate port**: Keeps audio streaming isolated from the API, allowing independent scaling and configuration.
38
39## Core Components
40
41```
42controllers/streams/
43  __init__.py          - Package init, exports StreamsController
44  controller.py        - StreamsController: HTTP endpoints, public streaming API
45  audio.py             - StreamsAudio: audio processing, stream acquisition, DSP/filters
46  audio_buffer.py      - AudioBuffer: in-memory PCM audio buffering with seek support
47  constants.py         - Shared constants (buffer sizes, config keys)
48  ogg_handler.py       - Chained OGG stream stitching for radio
49  smart_fades/         - Smart crossfade detection and mixing
50    analyzer.py        - Beat analysis for smart fade detection
51    fades.py           - Fade curve generation
52    mixer.py           - Crossfade mixing logic
53```
54
55Supporting modules in `helpers/`:
56- `helpers/audio.py` - Generic audio utilities (PCM helpers, format conversions, silence stripping)
57- `helpers/ffmpeg.py` - FFmpeg process management
58
59## AudioBuffer
60
61`AudioBuffer` is the primary interface for all buffered audio streaming. It stores **raw decoded PCM audio** (no filters applied) and serves as the single source of truth for audio data.
62
63### Design Principles
64
651. **Always-on buffering**: Every queue stream (tracks and radio) goes through an AudioBuffer
662. **Raw PCM only**: The buffer stores decoded audio in original sample rate and bit depth. Filters (volume normalization, playback speed, etc.) are applied when reading via `get_stream()`
673. **Pre-initialization**: Buffers are created and start filling before the player requests the stream, ensuring immediate playback start
684. **Buffer reuse**: Existing valid buffers are reused for seek operations and reconnections
695. **Smart seeking**: Forward seeks within 20 seconds of buffered data wait for the producer; larger seeks trigger a re-fetch at the seek position
70
71### Buffer Modes
72
73- **SEEKABLE** (tracks): Maintains a deque of 1-second PCM chunks with seek support. Old chunks are discarded when the buffer reaches max size
74- **ROLLING** (radio/non-seekable): Short FIFO buffer (~15 seconds) where the consumer pops chunks sequentially
75
76### Key Methods
77
78- `AudioBuffer.get_buffer()` - Static factory that creates or reuses a buffer. Reads config, determines mode, starts the analysis reader, starts filling
79- `AudioBuffer.get_stream()` - Get processed audio with optional filters/resampling applied
80- `AudioBuffer.get_raw_stream()` - Get unprocessed raw PCM audio (playback consumer)
81- `AudioBuffer.read_chunk_for_analysis()` - Read one chunk for a passive analysis reader without mutating the buffer; raises when the chunk has been evicted (reader fell behind)
82- `AudioBuffer.fill()` - Start filling from an async generator of PCM chunks
83- `AudioBuffer.ready` - Event set when enough chunks are buffered past the seek point (threshold-based)
84
85### Buffer Lifecycle
86
87```
881. _load_item() fetches stream details, creates buffer with wait_ready=True
892. Buffer starts filling from get_media_stream() in background
903. Analysis (loudness, smart fades) reads the same buffer in parallel, at lower priority
914. Player requests stream -> get_queue_item_stream() calls buffer.get_stream()
925. ~30s before track end: prepare_next_audio_buffer() pre-fills next track
936. _cleanup_stale_queue_buffers() clears old buffers to free memory
94```
95
96### Error Handling
97
98- Producer errors are captured and surfaced when consumers try to read
99- Consumers can drain remaining buffered data before the error surfaces at EOF
100- Errors bubble up as `AudioError` through the streaming chain
101
102## StreamsAudio
103
104`StreamsAudio` is the audio processing sub-controller, initialized as `self.audio` on the StreamsController. It handles all audio-related logic that needs access to the MusicAssistant instance:
105
106- **Stream acquisition**: `get_media_stream`, `get_stream_details`, radio/HTTP/file stream helpers
107- **Queue streaming**: `get_queue_item_stream`, `get_queue_item_stream_with_smartfade`, `get_queue_flow_stream`
108- **Format selection**: `get_output_format`, `select_pcm_format`, `select_flow_format`
109- **DSP and output plans**: `get_player_output_plan`, `get_player_dsp_details`, `get_stream_dsp_details`
110- **Crossfade management**: `crossfade_allowed`, `clear_crossfade_data`
111- **Loudness analysis**: `attach_loudness_analyzer` (via buffer callbacks)
112
113`AudioProcessingManager`, initialized as `self.audio_processing` on the
114StreamsController, combines queue processing and per-player output plans into complete
115`AudioProcessingChain` snapshots attached to `StreamDetails`.
116
117## Streaming Pipeline
118
119```
120Music Provider -> get_media_stream() -> FFmpeg (decode to raw PCM)
121    -> AudioBuffer (raw PCM storage, analyze callbacks run here)
122    -> buffer.get_stream() -> Optional: FFmpeg (volume normalization, speed, fade-in)
123    -> Optional: Smart Fades (crossfade mixing between tracks)
124    -> FFmpeg (encode to output format with player-specific DSP)
125    -> HTTP Response / Direct PCM stream
126```
127
128### Stream Entry Points
129
1301. **HTTP endpoints** (`serve_queue_item_stream`, `serve_queue_flow_stream`): Used by players that consume HTTP streams (Chromecast, DLNA, Sonos, etc.)
1312. **Direct PCM** (`get_stream`): Used by player providers that consume raw PCM directly (AirPlay, Sendspin, etc.)
132
133## Analyze Callbacks
134
135AudioBuffer supports registering chunk callbacks that receive raw PCM data as it flows into the buffer. This enables ahead-of-time analysis without re-streaming:
136
137### Loudness Measurement
138- Attached automatically when a new buffer is created (tracks and radio)
139- Feeds up to 2 minutes of PCM into an FFmpeg `ebur128` process
140- Result stored for future volume normalization (avoids dynamic mode overhead)
141
142### Smart Fades Beat Analysis
143- Attached automatically for music tracks (MediaType.TRACK only, not podcasts/audiobooks)
144- Collects first 45 seconds (intro) and last 45 seconds (outro) of audio
145- Triggers librosa beat detection in a background thread
146- Results cached for crossfade timing decisions
147
148Both analyzers check for existing measurements before starting, avoiding redundant work.
149
150## Smart Fades
151
152The smart fades system provides intelligent crossfading between tracks:
153
154- **Smart Crossfade**: Analyzes audio beats to detect natural fade points
155- **Standard Crossfade**: Fixed-duration overlap crossfade with silence stripping
156- Operates in both flow mode (continuous stream) and per-item mode (gapless playback)
157
158## Audio Overlay
159
160The audio overlay is a per-queue feature (configured via `player_queues/overlay`) that mixes a
161looping sound effect — any `sound_effect` media item offered by a provider — into the queue's
162audio stream:
163
164- Mixing happens once per queue stream (ffmpeg `amix`, overlay looped via `-stream_loop -1`),
165  so all (synced) players consuming the stream hear the identical mix.
166- An active overlay forces flow mode: the overlay must play continuously across track
167  boundaries, which is impossible with per-item stream requests. Radio is the exception —
168  it always plays as a single long-lived stream and is wrapped per-request instead.
169- The internal PCM format is upgraded to F32 (like crossfade/DSP) for clipping-free headroom.
170- Failures degrade gracefully: when the overlay source can not be resolved, playback simply
171  continues without overlay; when the overlay input dies mid-stream, ffmpeg keeps passing
172  the main audio. Music playback is never interrupted by the overlay.
173- Note: audio already sitting in a player's (pre)buffer is unaffected by overlay changes,
174  which is why the queue controller restarts playback on an audible change. For the same
175  reason a seek can momentarily shift the overlay position — acceptable for ambient content.
176
177## Stream Types
178
179| Type | AudioBuffer | Description |
180|------|-------------|-------------|
181| Queue tracks | Yes (SEEKABLE) | Regular track playback with full buffering |
182| Radio streams | Yes (ROLLING) | Short rolling buffer, non-seekable |
183| Announcements | Yes (SEEKABLE) | Short one-off audio (TTS), rendered once and shared by all consumers |
184| Plugin sources | No | Real-time audio (microphone, aux), streamed directly |
185
186## Configuration
187
188Key configuration entries (in streams controller config):
189
190| Key | Type | Default | Description |
191|-----|------|---------|-------------|
192| `buffer_size` | String | Memory-dependent (`maximum` >=8GB, `balanced` >=4GB, `minimal` <4GB) | Audio buffer size preset |
193| `volume_normalization_radio` | String | `fallback_dynamic` | Normalization mode for radio |
194| `volume_normalization_tracks` | String | `fallback_dynamic` | Normalization mode for tracks |
195| `allow_crossfade_same_album` | Boolean | `false` | Whether to crossfade consecutive album tracks |
196