Speech

The speech bridge recognizes speech on the device, streams dictation as events, speaks text and lists voices.

Install and register

dotnet add package Shiny.AppDeviceBridge.Speech

builder.UseAppDeviceBridge(
    bridge => bridge.AddSpeechBridge(),
    webApp => { /* … */ }
);

Client	C#	TypeScript
ISpeechBridge	AddSpeechBridgeClient()	SpeechBridge

Details

Built on Shiny.Speech, which is still a prerelease package.

Topic	Details
Recognizing once	recognize listens until a pause and returns { "text": … }. It gives up after timeoutMs: 15 s by default, 60 s at most. text is null if nothing was heard.
Dictation	POST listener keeps the microphone open and streams speech.partial and speech.result events until you delete it. It needs a listener for one of them to start (409 without) and stops once neither has one, so a page that goes away can’t leave the microphone on. speech.ended says why it stopped: Stopped, PageClosed or Error.
One microphone	a recognition and a listener can’t run at once. The second gets 409 microphone_busy.
Speaking	speak waits until the text has been spoken, unless you pass "wait": false, which returns 204 straight away and raises speech.spoken when done. A new utterance interrupts the current one. Text is capped at 4,000 characters.
Platforms	Linux has no OS speech engine, so its endpoints return 501. To use a cloud provider or Whisper there, pass o => o.RegisterSpeechServices = false and register your own ISpeechToTextService / ITextToSpeechService.
Platform setup	Android needs RECORD_AUDIO, plus a <queries> entry for android.intent.action.TTS_SERVICE. Apple platforms need NSSpeechRecognitionUsageDescription and NSMicrophoneUsageDescription, plus the com.apple.security.device.audio-input entitlement for sandboxed apps.

From the page

recognize is one utterance in, text out. A listener keeps dictating and streams partial and final results as events — listen to onPartial or onResult first, or starting it answers 409.

import { SpeechBridge } from "@shinyorg/appdevicebridge";

const speech = new SpeechBridge();
await speech.requestAccess();

const { text } = await speech.recognize({ culture: "en-US", silenceTimeoutMs: 1500 });

await speech.onPartial(r => input.value = r.text);
await speech.onResult(r => submit(r.text));
await speech.startListener({ culture: "en-US", keywords: ["stop"] });

await speech.speak({ text: "Order received", rate: 1.0, wait: true });

Requests

GET/_bridge/speech/statusWhat is supported, and whether the microphone or the speaker is in use.POST/_bridge/speech/accessRequests microphone and speech recognition permission.POST/_bridge/speech/recognizeListens until the speaker goes quiet and returns what was heard.GET/_bridge/speech/listenerThe running dictation, or null when there is none.POST/_bridge/speech/listenerStarts dictation.DELETE/_bridge/speech/listenerStops dictation.POST/_bridge/speech/speakSpeaks text, interrupting anything already being spoken.DELETE/_bridge/speech/speakStops speaking.GET/_bridge/speech/voicesThe voices installed, optionally for one culture such as en-US.GET/_bridge/speech/culturesThe cultures voices exist for.

GET/_bridge/speech/status#

What is supported, and whether the microphone or the speaker is in use.

C#ISpeechBridge.GetStatusAsyncTSSpeechBridge.getStatus

Response

200 SpeechStatus

SpeechStatus · 5 fields

recognitionSupported	boolean
synthesisSupported	boolean
listening	boolean
recognizing	boolean
speaking	boolean

POST/_bridge/speech/access#

Requests microphone and speech recognition permission.

C#ISpeechBridge.RequestAccessAsyncTSSpeechBridge.requestAccess

Response

200 SpeechAccessResult

SpeechAccessResult · 1 field

access

AccessState

POST/_bridge/speech/recognize#

Listens until the speaker goes quiet and returns what was heard. text is null when nothing was heard before the timeout.

C#ISpeechBridge.RecognizeAsyncTSSpeechBridge.recognize

Body

application/json · RecognizeRequest

RecognizeRequest · 5 fields

culture	string | null	A culture name such as en-US; the device's when null.
silenceTimeoutMs	number | null	How much quiet ends it, 500–10,000 ms.
timeoutMs	number | null	The longest it waits in all, 1,000–60,000 ms; 15 seconds when null.
preferOnDevice	boolean | null	Keep audio on the device where the platform allows.
keywords	string[] | null	At most 32, each at most 64 characters.

Response

200 RecognizeResult

RecognizeResult · 1 field

text

string | null

What was heard; null when nothing was.

GET/_bridge/speech/listener#

The running dictation, or null when there is none.

C#ISpeechBridge.GetListenerAsyncTSSpeechBridge.getListener

Response

200 SpeechListener · or 204 when there is none

SpeechListener · 2 fields

culture	string | null
keywords	string[] | null

POST/_bridge/speech/listener#

Starts dictation. Results arrive through onPartial, onResult and onKeyword, so listen first: without a listener for onPartial or onResult it fails with 409. It ends when stopped or once neither has a listener left.

C#ISpeechBridge.StartListenerAsyncTSSpeechBridge.startListener

Body

application/json · SpeechListenerRequest

SpeechListenerRequest · 4 fields

culture	string | null	A culture name such as en-US; the device's when null.
silenceTimeoutMs	number | null	How much quiet ends a phrase, 500–10,000 ms.
preferOnDevice	boolean | null	Keep audio on the device where the platform allows.
keywords	string[] | null	Words that raise onKeyword. At most 32, each at most 64 characters.

Response

200 SpeechListener

SpeechListener · 2 fields

culture	string | null
keywords	string[] | null

Errors

409Conflict

DELETE/_bridge/speech/listener#

Stops dictation.

C#ISpeechBridge.StopListenerAsyncTSSpeechBridge.stopListener

Response

204 No content

POST/_bridge/speech/speak#

Speaks text, interrupting anything already being spoken. With wait it returns once speech ends; without, it returns null straight away and onSpoken says when it ended.

C#ISpeechBridge.SpeakAsyncTSSpeechBridge.speak

Body

application/json · SpeakRequest

SpeakRequest · 7 fields

text	string	At most 4,000 characters.
culture	string | null	A culture name such as en-US.
voice	string | null	A voice's id.
pitch	number | null	0.5–2.
rate	number | null	0.25–4.
volume	number | null	0–1.
wait	boolean	Return once speech ends rather than straight away.

Response

200 SpeakResult · or 204 when there is none

SpeakResult · 1 field

completed

boolean

False when it was interrupted.

DELETE/_bridge/speech/speak#

Stops speaking.

C#ISpeechBridge.StopSpeakingAsyncTSSpeechBridge.stopSpeaking

Response

204 No content

GET/_bridge/speech/voices#

The voices installed, optionally for one culture such as en-US.

C#ISpeechBridge.GetVoicesAsyncTSSpeechBridge.getVoices

Parameter	In	Type
culture	query	string | null	optional

Response

200 SpeechVoice[]

SpeechVoice · 3 fields

id	string
name	string
culture	string | null

GET/_bridge/speech/cultures#

The cultures voices exist for.

C#ISpeechBridge.GetCulturesAsyncTSSpeechBridge.getCultures

Response

200 string[]

Events

EVENTspeech.partial#

Dictation's best guess so far, replaced as the speaker goes on.

C#ISpeechBridge.OnPartialAsyncTSSpeechBridge.onPartial

Payload

SpeechRecognized

text	string
confidence	number | null	0–1, where the platform reports one.

EVENTspeech.result#

A finished phrase from dictation.

C#ISpeechBridge.OnResultAsyncTSSpeechBridge.onResult

Payload

SpeechRecognized

text	string
confidence	number | null	0–1, where the platform reports one.

EVENTspeech.keyword#

One of the dictation's keywords was heard.

C#ISpeechBridge.OnKeywordAsyncTSSpeechBridge.onKeyword

Payload

SpeechKeyword

keyword

string

EVENTspeech.ended#

Dictation ended.

C#ISpeechBridge.OnEndedAsyncTSSpeechBridge.onEnded

Payload

SpeechEnded

reason

SpeechEndReason

EVENTspeech.spoken#

An utterance finished or was interrupted.

C#ISpeechBridge.OnSpokenAsyncTSSpeechBridge.onSpoken

Payload

SpeakResult

completed

boolean

False when it was interrupted.

EVENTspeech.error#

Recognition or synthesis failed.

C#ISpeechBridge.OnErrorAsyncTSSpeechBridge.onError

Payload

SpeechError

source	SpeechErrorSource
message	string

Types

JSON property names are camelCase and enums travel as their names. A ? marks a property the payload may omit;| null one that may be null.

SpeechStatusobject#

recognitionSupported	boolean
synthesisSupported	boolean
listening	boolean
recognizing	boolean
speaking	boolean

SpeechAccessResultobject#

access

AccessState

AccessStateenum#

Where a permission stands. Reported by every bridge that needs one; the names match Shiny's own.

"Unknown""NotSupported""NotSetup""Disabled""Restricted""Denied""Available"

RecognizeRequestobject#

culture?	string | null	A culture name such as en-US; the device's when null.
silenceTimeoutMs?	number | null	How much quiet ends it, 500–10,000 ms.
timeoutMs?	number | null	The longest it waits in all, 1,000–60,000 ms; 15 seconds when null.
preferOnDevice?	boolean | null	Keep audio on the device where the platform allows.
keywords?	string[] | null	At most 32, each at most 64 characters.

RecognizeResultobject#

text

string | null

What was heard; null when nothing was.

SpeechListenerobject#

culture	string | null
keywords	string[] | null

SpeechListenerRequestobject#

culture?	string | null	A culture name such as en-US; the device's when null.
silenceTimeoutMs?	number | null	How much quiet ends a phrase, 500–10,000 ms.
preferOnDevice?	boolean | null	Keep audio on the device where the platform allows.
keywords?	string[] | null	Words that raise onKeyword. At most 32, each at most 64 characters.

SpeakRequestobject#

text	string	At most 4,000 characters.
culture?	string | null	A culture name such as en-US.
voice?	string | null	A voice's id.
pitch?	number | null	0.5–2.
rate?	number | null	0.25–4.
volume?	number | null	0–1.
wait?	boolean	Return once speech ends rather than straight away.

SpeakResultobject#

completed

boolean

False when it was interrupted.

SpeechVoiceobject#

id	string
name	string
culture	string | null

SpeechRecognizedobject#

text	string
confidence	number | null	0–1, where the platform reports one.

SpeechKeywordobject#

keyword

string

SpeechEndedobject#

reason

SpeechEndReason

SpeechEndReasonenum#

Why dictation ended.

"Stopped""PageClosed""Error"

SpeechErrorobject#

source	SpeechErrorSource
message	string

SpeechErrorSourceenum#

"Recognition""Synthesis"

Security

These routes sit under AppDeviceBridgePolicies.Bridges like every other bridge: callers on this device by default. See Security.