Getting Started

GitHub
Core
Cloud
Azure
OpenAI
ElevenLabs
Typecast
Microsoft.Extensions.AI
Whisper (Linux)

Frameworks

.NET

.NET MAUI

Operating Systems

Android

iOS

Windows

Web

Linux

Shiny.Speech provides a unified API for speech-to-text, text-to-speech, audio capture, and audio playback across Android, iOS, Windows, Browser (Blazor WebAssembly), and Linux — with pluggable cloud providers for Azure AI Speech, OpenAI, ElevenLabs, and Typecast.

Linux

Linux audio ships in the separate Shiny.Audio.Linux package (PulseAudio/PipeWire with an ALSA fallback) — register it with AddLinuxAudio() before the other Add* calls. There is no native Linux speech engine, so pair it with a cloud provider. See Audio for details.

Features

• Event-based speech-to-text with Start/Stop lifecycle — multiple subscribers supported
• Built-in keyword detection via SpeechRecognitionOptions.Keywords and the KeywordHeard event
• Platform-native text-to-speech with voice selection, rate, pitch, and volume control
• Raw audio capture from the device microphone (16kHz, 16-bit, mono PCM)
• Voice processing on capture — echo cancellation, noise suppression, and automatic gain control via AudioProcessingOptions (cancels TTS bleed for barge-in)
• Audio playback for MP3 streams
• Pluggable cloud provider architecture — swap between on-device and cloud STT/TTS
• Azure AI Speech integration (STT + TTS) with SSML prosody control
• OpenAI integration (STT + TTS) powered by Whisper and GPT-4o models
• ElevenLabs integration — Scribe speech-to-text and multilingual text-to-speech
• Typecast integration — AI voice text-to-speech (TTS only) via the typecast-csharp SDK
• On-device Whisper speech-to-text on Linux (incl. Raspberry Pi) — fully offline, no cloud account or network
• Microsoft.Extensions.AI adapter — expose providers as ISpeechToTextClient / ITextToSpeechClient
• Runtime credential changes — provider config objects are mutable singletons; rotate API keys / region / model without re-registration
• State tracking — IsListening (STT), IsSpeaking (TTS), IsPlaying (audio)
• VU meter signal (outgoing) — AudioLevelChanged event on ITextToSpeechService and IAudioPlayer emits a normalized 0.0–1.0 RMS level during playback; IsPlayerAnalysisSupported reports availability per platform
• VU meter signal (incoming) — InputLevelChanged event on ISpeechToTextService (gated by IsInputAnalysisSupported), IAudioSource (every platform), and IAudioMonitor emits the same normalized 0.0–1.0 level from the microphone
• Permission management via AccessState and RequestAccess()
• Convenience extension methods: ListenUntilSilence, StatementAfterKeyword, WaitListenForKeywords, ListenForKeywords
• CarPlay compatible — iOS audio routes through the car’s mic/speakers automatically when CarPlay is active

Packages

Package	Purpose
Shiny.Speech	Core library — platform-native STT, TTS, audio capture, and playback
Shiny.Speech.Cloud	Cloud provider abstractions (included transitively by Azure/ElevenLabs)
Shiny.Speech.Azure	Azure AI Speech provider (STT + TTS)
Shiny.Speech.OpenAI	OpenAI provider (STT + TTS) — Whisper, GPT-4o Transcribe, GPT-4o Mini TTS
Shiny.Speech.ElevenLabs	ElevenLabs provider — Scribe STT + TTS
Shiny.Speech.Typecast	Typecast provider (TTS only) — AI voice via the typecast-csharp SDK
Shiny.Speech.MicrosoftAI	Microsoft.Extensions.AI adapter — ISpeechToTextClient / ITextToSpeechClient
Shiny.Speech.Linux.Whisper	On-device offline STT on Linux via Whisper/whisper.cpp (STT only)

Setup

MAUIBlazor

Changed files

• MyApp/
  • ▸Platforms
    • ▸Android
      • AndroidManifest.xml
    • ▸iOS
      • Info.plist
      • PrivacyInfo.xcprivacy
  • MauiProgram.cs
  • MyApp.csproj

Platforms/Android/AndroidManifest.xml

AndroidManifest.xmlCopy

1<?xml version="1.0" encoding="utf-8"?>2<manifest xmlns:android="http://schemas.android.com/apk/res/android">3	<application android:allowBackup="true" android:icon="@mipmap/appicon" android:roundIcon="@mipmap/appicon_round" android:supportsRtl="true">4	</application>5	<uses-permission android:name="android.permission.RECEIVE_BOOT_COMPLETED"/>6	<uses-permission android:name="android.permission.BATTERY_STATS" />7	<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />8	<uses-permission android:name="android.permission.INTERNET" />9	<uses-permission android:name="android.permission.RECORD_AUDIO" />10	<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />11</manifest>

UseShiny() is required

Native speech and audio delegate Android permission handling and activity tracking to Shiny.Core’s AndroidPlatform. Reference the Shiny.Hosting.Maui package and call .UseShiny() on your MauiAppBuilder before building:

builder
    .UseMauiApp<App>()
    .UseShiny();

builder.Services.AddSpeechServices();

Without .UseShiny(), RequestAccess() throws a TimeoutException on Android because no AndroidPlatform is registered to receive the permission-result callback. AccessState also comes from Shiny.Core now (namespace Shiny).

Platform Notes

Android Speech Recognition

Android’s native speech recognition engine works in short recording segments — it automatically stops listening after a period of silence and must restart for the next segment. This creates brief pauses during continuous listening that are noticeable compared to iOS, which streams continuously.

If smooth, uninterrupted speech recognition is important for your app, use the ElevenLabs cloud provider, which transcribes captured audio without relying on the native engine’s start/stop behavior.

builder.Services.AddElevenLabsSpeechToText("your-api-key");

Avoid Azure on Android: the Azure AI Speech provider does not currently support Android’s 16 KB memory page size. On devices that use 16 KB pages (Android 15+ on supported hardware), the Azure native libraries fail to load — prefer ElevenLabs on Android.

Platform Permissions

Android — Add to AndroidManifest.xml:

<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />

MODIFY_AUDIO_SETTINGS is required for the TTS audio-level Visualizer and for the native STT beep suppression.

Nothing else is needed. Shiny.Speech ships its own Android library manifest carrying the <queries> declaration that makes the platform recognition service visible on API 30+, and it merges into your app automatically:

<queries>
    <intent><action android:name="android.speech.RecognitionService" /></intent>
</queries>

Without it, package visibility filtering hides the recognition service, IsSupported is false and RequestAccess() returns AccessState.NotSupported before the microphone permission is ever requested. Permissions stay yours to declare — RECORD_AUDIO is a dangerous permission and would otherwise surface in the store listing of an app that only uses text-to-speech.

iOS — Add to Info.plist:

<key>NSSpeechRecognitionUsageDescription</key>
<string>This app uses speech recognition</string>
<key>NSMicrophoneUsageDescription</key>
<string>This app uses the microphone for speech recognition</string>

Windows — Add the Microphone capability to your Package.appxmanifest:

<Capabilities>
    <DeviceCapability Name="microphone" />
</Capabilities>

Browser (Blazor WebAssembly) — No manifest changes and no <script> tag needed. The browser prompts for microphone access automatically, and the JS interop module ships inside the Shiny.Audio package as a static web asset (_content/Shiny.Audio/shiny-audio.js), loaded on demand via JSHost.ImportAsync. Just reference the NuGet package — don’t copy the JS into wwwroot.

IAudioSource captures raw PCM audio in the browser using the Web Audio API (getUserMedia + ScriptProcessorNode), downsampled to 16kHz 16-bit mono.

Quick Example

public class MyViewModel
{
    readonly ISpeechToTextService _stt;
    readonly ITextToSpeechService _tts;

    public MyViewModel(ISpeechToTextService stt, ITextToSpeechService tts)
    {
        _stt = stt;
        _tts = tts;
    }

    async Task ListenAndRespond(CancellationToken ct)
    {
        // 1. Request permission
        var access = await _stt.RequestAccess();
        if (access != AccessState.Available)
            return;

        // 2. Listen until the user stops speaking
        var text = await _stt.ListenUntilSilence(
            new SpeechRecognitionOptions
            {
                Culture = CultureInfo.GetCultureInfo("en-US"),
                SilenceTimeout = TimeSpan.FromSeconds(3)
            },
            ct
        );

        if (text != null)
        {
            // 3. Speak the result back
            await _tts.SpeakAsync($"You said: {text}");
        }
    }

    async Task EventBasedListening()
    {
        // Subscribe to events — multiple classes can subscribe simultaneously
        _stt.ResultReceived += (s, result) =>
            Console.WriteLine($"[{(result.IsFinal ? "FINAL" : "partial")}] {result.Text}");

        _stt.KeywordHeard += (s, keyword) =>
            Console.WriteLine($"Keyword: {keyword}");

        // Start listening with keyword detection
        await _stt.Start(new SpeechRecognitionOptions
        {
            Keywords = ["Yes", "No", "Maybe"]
        });

        // Later: stop
        await _stt.Stop();
    }

    async Task WakeWordExample(CancellationToken ct)
    {
        // "Hey Computer, do something" → returns "do something"
        var command = await _stt.StatementAfterKeyword(["Hey Computer"], cancellationToken: ct);

        if (command != null)
            await _tts.SpeakAsync($"You asked: {command}");
    }

    async Task KeywordExample(CancellationToken ct)
    {
        // Wait for a keyword (with optional timeout)
        await _tts.SpeakAsync("Do you agree? Say yes, no, or maybe.");
        var answer = await _stt.WaitListenForKeywords(
            ["Yes", "No", "Maybe"],
            timeout: TimeSpan.FromSeconds(30),
            cancellationToken: ct
        );
    }
}

Continuous Sessions

Start() opens the microphone and keeps it open. Both native recognizers are single-utterance underneath, so the service re-arms them for you after every final result — you start once and stay listening until you call Stop().

Start once, not once per phrase

Stopping and starting per question tears down the audio engine and loses whatever is said while it comes back up. Keywords is fixed at Start(), so when the set changes per turn, match ResultReceived yourself rather than restarting the session.

Transient failures in that loop — the mic taken by another capture, a busy recognizer, a dropped network round trip — do not end the session. They are reported through Error and the session re-arms behind a backoff of 250ms doubling to a 4 second ceiling, reset the moment a result comes back. After SpeechRetryPolicy.MaxConsecutiveFailures (5) failures in a row the session stops itself rather than looping, so IsListening never reports true for a session that has quietly died. Errors that retrying cannot fix — a missing permission, an unsupported language — stop immediately.

On-device recognition

PreferOnDevice asks for recognition with no network round trip and no session length cap, which is what long continuous sessions and offline use want:

await stt.Start(new SpeechRecognitionOptions { PreferOnDevice = true });

It is best-effort on every platform. iOS sets RequiresOnDeviceRecognition when the locale supports it; Android uses the on-device recognizer when the device has one installed (API 31+) and otherwise falls back to the system recognizer with the offline hint set. A device without local recognition stays on the network path rather than failing.

VU Meters (Audio Levels)

Both directions are metered on the same normalized 0.0–1.0 scale — a “speaking” bar for what comes out of the speaker and a “listening” bar for what goes into the mic.

Outgoing — playback / TTS

ITextToSpeechService and IAudioPlayer raise AudioLevelChanged while audio is playing. Check IsPlayerAnalysisSupported before binding UI — it’s false where metering isn’t available (Windows native TTS, Browser).

Incoming — microphone

ISpeechToTextService raises InputLevelChanged while listening; check IsInputAnalysisSupported. The lower-level IAudioSource.InputLevelChanged (raw capture, supported everywhere, no flag) and IAudioMonitor.InputLevelChanged (live mic-to-output) emit the same signal.

public partial class VoiceViewModel : ObservableObject
{
    [ObservableProperty] double speakingLevel;    // 0.0 .. 1.0
    [ObservableProperty] double listeningLevel;

    public bool IsSpeakingVuSupported { get; }
    public bool IsListeningVuSupported { get; }

    public VoiceViewModel(ITextToSpeechService tts, ISpeechToTextService stt)
    {
        IsSpeakingVuSupported = tts.IsPlayerAnalysisSupported;
        IsListeningVuSupported = stt.IsInputAnalysisSupported;

        tts.AudioLevelChanged += (_, level) =>
            MainThread.BeginInvokeOnMainThread(() => SpeakingLevel = level);

        stt.InputLevelChanged += (_, level) =>
            MainThread.BeginInvokeOnMainThread(() => ListeningLevel = level);
    }
}

<ProgressBar Progress="{Binding SpeakingLevel}"
             IsVisible="{Binding IsSpeakingVuSupported}" />
<ProgressBar Progress="{Binding ListeningLevel}"
             IsVisible="{Binding IsListeningVuSupported}" />

Surface	iOS / macOS	Android	Windows	Browser	Linux
Native TTS (ITextToSpeechService)	✅ — AVAudioEngine + player-node tap	✅ — OnAudioAvailable RMS	❌	❌	n/a
Cloud TTS (CloudTextToSpeech)	✅ — forwarded from IAudioPlayer	✅ — forwarded from IAudioPlayer	❌	❌	✅ — forwarded from IAudioPlayer
Generic playback (IAudioPlayer)	✅ — AVAudioPlayer.MeteringEnabled	✅ — Visualizer on session	❌	❌	✅ — metered from the decoded PCM
Cloud STT (CloudSpeechToText)	✅ — forwarded from IAudioSource	✅ — forwarded from IAudioSource	✅ — forwarded from IAudioSource	✅ — forwarded from IAudioSource	✅ — forwarded from IAudioSource
Native STT (ISpeechToTextService)	✅ — recognizer input-node tap	✅ — OnRmsChanged	❌	❌	n/a
Capture (IAudioSource)	✅	✅	✅	✅	✅
Monitor (IAudioMonitor)	✅	✅	n/a	n/a	✅

Cloud recognition meters the IAudioSource feeding the provider, so a mic meter works on every platform. Native recognition depends on what the OS exposes: Apple taps the recognizer’s own input node, Android forwards SpeechRecognizer.OnRmsChanged, while the Windows and Web Speech recognizers own the mic and surface no level at all. Linux has no native recognizer to meter at all — but its player decodes to PCM in managed code, so playback metering works there where it doesn’t on Windows or Browser.

On iOS / macOS the native TTS path routes AVSpeechSynthesizer through AVAudioEngine + AVAudioPlayerNode so a tap can compute RMS. The engine is created lazily and kept warm across utterances — first-utterance latency adds roughly 50–150 ms; subsequent calls are indistinguishable from the legacy direct path.

Level events are raised off the UI thread (capture-side ones are throttled to ~20/sec, peak-held in between) — marshal before mutating bound properties, and reset your bound value to 0 when playback or listening ends so the meter drains.

Voice Processing (Noise Suppression & Echo Cancellation)

Microphone capture can request platform voice-processing effects to strip background noise and, critically, to cancel your text-to-speech output from the mic so it isn’t re-captured while the mic is open (barge-in). Configure it via AudioProcessingOptions — either directly on IAudioSource.StartCaptureAsync(...) or through SpeechRecognitionOptions.AudioProcessing, which is honored wherever the capture belongs to Shiny: every cloud provider (they record through IAudioSource) and the native iOS / Mac Catalyst / macOS recognizer, which owns its own AVAudioEngine. Leaving it null keeps the default those paths have always used — the full VoiceChat chain — rather than raw capture.

Not available on Android

Native recognition on Android runs in the platform’s own out-of-process SpeechRecognizer service, which opens the microphone itself and exposes no voice-processing controls — there is no capture session to apply this to, and setting it logs a warning. Use a cloud provider if the recognition path needs these effects on Android.

await stt.Start(new SpeechRecognitionOptions
{
    Culture = CultureInfo.GetCultureInfo("en-US"),
    AudioProcessing = AudioProcessingOptions.VoiceChat   // AEC + noise suppression + AGC
});

// or, capturing raw audio directly:
var stream = await audioSource.StartCaptureAsync(new AudioProcessingOptions
{
    EchoCancellation = true,      // subtracts speaker/TTS output from the mic signal
    NoiseSuppression = true,      // attenuates steady background noise
    AutomaticGainControl = true   // normalizes capture level
});

Each flag is best-effort and maps to native voice processing:

Effect	iOS / macOS	Android	Windows	Browser
Echo Cancellation	✅ Voice-Processing I/O	✅ AcousticEchoCanceler + VoiceCommunication	⚠️ best-effort (Communications pipeline)	✅ WebRTC AEC3
Noise Suppression	✅ (bundled)	✅ NoiseSuppressor	⚠️ best-effort	✅
Automatic Gain Control	✅ (bundled)	✅ AutomaticGainControl	⚠️ best-effort	✅

• Apple bundles all three into a single Voice-Processing I/O unit — enabling any flag enables the whole chain (they can’t be toggled independently).
• Android effect availability is device/driver dependent; unavailable effects are skipped. Requesting echo cancellation also routes capture through VoiceCommunication.
• Windows AudioGraph exposes no per-effect control; requesting any effect selects the Communications capture category, which engages driver-provided AEC/NS when present.
• These are OS/hardware cancellers referencing the real speaker feed, so any device audio (not just library-played TTS) is cancelled.
• Native on-device ISpeechToTextService implementations manage their own microphone and are unaffected by this setting — it applies to IAudioSource capture (cloud providers, raw capture).

Changing Credentials at Runtime

Provider config objects are mutable singletons, so you can change the API key (or region / model / voice) at any time — after the user pastes a new key in a settings screen, for example — and the provider picks it up on its next call. No re-registration required. Hold your config instance, or resolve it from DI:

var config = new AzureSpeechConfig { SubscriptionKey = "initial-key", Region = "eastus" };
builder.Services.AddAzureSpeech(config);

// ...later, e.g. after the user rotates the key in settings:
config.SubscriptionKey = "rotated-key";        // next Speak/recognize uses it

// Or resolve it from the container if you didn't keep a reference:
serviceProvider.GetRequiredService<TypecastConfig>().ApiKey = "new-key";

Providers that cache an SDK/HTTP client (ElevenLabs, Typecast) transparently rebuild it when the key changes; Azure and OpenAI read the config on every call.

Samples

• Sample MAUI App
• Sample Blazor WASM App

AI Coding Assistant

Claude CodeGitHub Copilot

Step 1 — Add the marketplace:

claude plugin marketplace add shinyorg/skills

Step 2 — Install plugins:

claude plugin install shiny-client@shiny

BLE, GPS, Jobs, Notifications, Push, HTTP Transfers, OBD, Music, Health, DataSync — iOS, Android, Windows, MacOS, Linux, Web

claude plugin install shiny-maui@shiny

Shell, Contact Store

claude plugin install controls@shiny

TableView, BottomSheet, PillView, ImageViewer, Scheduler, Markdown, Mermaid Diagrams — MAUI and Blazor

claude plugin install shiny-mediator@shiny

Mediator/CQRS with middleware and source generators

claude plugin install shiny-data@shiny

DocumentDB and Spatial data libraries

claude plugin install shiny-aspire@shiny

Orleans and Gluetun Aspire integrations

claude plugin install shiny-extensions@shiny

DI, Stores, Reflector, Localization, Hosting modules

Step 1 — Add the marketplace:

copilot plugin marketplace add https://github.com/shinyorg/skills

Step 2 — Install plugins:

copilot plugin install shiny-client@shiny

BLE, GPS, Jobs, Notifications, Push, HTTP Transfers, OBD, Music, Health, DataSync — iOS, Android, Windows, MacOS, Linux, Web

copilot plugin install shiny-maui@shiny

Shell, Contact Store

copilot plugin install controls@shiny

TableView, BottomSheet, PillView, ImageViewer, Scheduler, Markdown, Mermaid Diagrams — MAUI and Blazor

copilot plugin install shiny-mediator@shiny

Mediator/CQRS with middleware and source generators

copilot plugin install shiny-data@shiny

DocumentDB and Spatial data libraries

copilot plugin install shiny-aspire@shiny

Orleans and Gluetun Aspire integrations

copilot plugin install shiny-extensions@shiny

DI, Stores, Reflector, Localization, Hosting modules

View Skills Repository

Links

• GitHub Repository