Add robius-speech, a new crate for platform-native speech-to-text input - #25
Merged
Conversation
…nput
`robius-speech` streams speech-to-text from the OS's own recognizer, with no
model downloads and no third-party services.
A recognition "session" returns multiple partial transcripts as they stream in
and a single final transcript from a user's "utterances" (chunks of spoken words).
It also provides microphone levels for a sound level meter or waveform animation.
Backends:
* macOS/iOS: SFSpeechRecognizer + AVAudioEngine, via a Swift bridge
* Android: SpeechRecognizer, via Java compiled to an embedded DEX
* Windows: SAPI dictation, chosen over WinRT because it works in a plain
unpackaged desktop app
* Linux, etc: unavailable, just a stub whose `is_supported()` returns false
Permissions are automatically requested by the crate on every platform,
just like all other robius crates, so your app doesn't need any
platform-specific code.
…field Transcripts are only half the job: a text field wants "replace these bytes with this text", partials have to revise the current utterance in place, dictated words need a space between them and whatever the user typed, and users keep typing and clicking mid-sentence. Every app that uses this crate would end up writing that same bookkeeping. Dictation owns it, with no dependency on any UI toolkit. Feed it each transcript and apply the Replacement it hands back; tell it when the user is about to change the field (interrupt) and what the field holds afterwards (settle), and it re-anchors at the caret adding only the words that aren't on screen yet, so nothing is lost or repeated. Each Replacement also says whether it continues an earlier one, so a run of revisions can share one undo step.
A caret key that doesn't move the caret (End at the end, ArrowRight at the end) or a click right on the caret is warned about as an edit before anyone can know it will change nothing. settle() then kept the interruption, swallowed every further revision of the utterance, and once its final arrived offered only the words beyond what was shown, so a recognizer that rewrote "I scream" as "Ice cream please" lost both the correction and the new word. settle() now checks the field first: if it holds exactly what dictation left in it, the interruption is called off and the utterance carries on, revisions included. Also covers a partial fed during an edit with a test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
robius-speechstreams speech-to-text from the OS's own recognizer, with no model downloads and no third-party services.A recognition "session" returns multiple partial transcripts as they stream in and a single final transcript from a user's "utterances" (chunks of spoken words). It also provides microphone levels for a sound level meter or waveform animation.
Backends:
unpackaged desktop app
is_supported()returns falsePermissions are automatically requested by the crate on every platform, just like all other robius crates, so your app doesn't need any platform-specific code.