SEPIA
Replies: 14 comments 29 replies
-
|
Hi Florian, Thanks for writing up this proposal 🙂 Interoperability between our projects would be great, so I'll do what I can to help. Using websockets and PCM audio as you described sounds good to me. This should allow almost anything to participate as an audio source 👍 Have you considered modeling the API after Home Assistant's WebSocket API? They have an "auth" step very similar to what you described. Their event structure also seems pretty simple, something like: {
"id": 5,
"type":"event",
"event":{
"data":{},
"event_type":"onaudiostart"
}
}
Adding some kind of "session id" into
The Web Speech API result structure seems reasonable to me. Extra properties could always be added for things like:
TrainingI think this is where things are going to get complicated. For the non-dictation case, something like Kaldi Active Grammars would be great to be able to use from a client. The author of Jaco Assistant (Daniel) has also proposed a Markdown format for sharing skills between assistants (riddle example). This includes:
Using rhasspy-nlu, I can transform these Markdown files into n-gram language models or directly into finite-state transducers for Kaldi. Since most of our STT engines use n-gram models or FSTs, we might consider this as a basic training format. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Michael,
But it is not clear to me how to apply that approach to the requirement here, having by example audio transcript client-server requests that I imagine as one-shot (stateless) websockets requests containing the audio binary buffer and metadata/properties (userId, authorization, language, mode=transcript, etc.).
Yes, my question in the above point is how to add "properties" in a raw websocket? using a serialization protocol as BSON/protocolo Buffer?
Yes, for non-dictation cases! If I'm not wrong, Vosk enables adaptive grammars using exactly the Kaldi format. Apparently the grammar (e.g. as a list of expected words/sentences) could be one of the properties of the client request, but please not that the these sentences are related to the specific "stateful" conversational context. So you want that a Dialog Manager (DM) store dynamically this data in a "user session" data storage, and the ASR could use this data when transcribing. That's interesting for me. Connecting the DM to the ASR, could simple, by example on my dialog manager, NaifJs; BTW, in very simple cases (command&control assistants), you can asimilate the The "bad" news is that ASR accuracy depends from the DM state.
I have to study the Daniel's Jaco interesting framework! |
Beta Was this translation helpful? Give feedback.
-
|
Hi Michael, thanks for stopping by 😃
I have not but I will definitely take a look. Compatibility to HA is probably not the worst idea ^^.
This can be done for sure, but my concern was more about the underlying STT engine's capability to transcribe multiple streams at the same time ... if the machine has enough resources anyway ^^. I think Nickolay from Vosk said that this is no problem, so I guess I'm just going to try and see :-)
👍
I would love to see that. I think this needs language models that are trained in a specific way or? As far as I remember there was only one model on the Vosk page that mentions it specifically.
This is probably something I could adapt for SEPIA. There is a similar format in SEPIA that's used for the Teach-UI and all the sentences trained with it can be loaded from the server via a REST endpoint. I've implemented this actually for the purpose of language model training 🙂 . I refined this later to JSON objects stored in the database with a more complex structure but the idea remains the same and I always wanted to further improve it to replace Well the training part is certainly a very interesting topic but maybe a bit out of scope for version 1 of the STT server 😅 . |
Beta Was this translation helpful? Give feedback.
-
|
Hi Florian, this is an interesting idea. Recently I did update Jaco's STT framework Scribosermo with new models. The network has now a better performance than the DeepSpeech network I did use before, but is as fast as it. The repository provides pretrained models in English, German, French and Spanish, which are exported into tflite-format for easy usage. Today I also did add support for streaming the audio input. Greetings, |
Beta Was this translation helpful? Give feedback.
-
|
Streaming example can be found here: Link |
Beta Was this translation helpful? Give feedback.
-
|
Multiple sessions should be possible, you just would need to handle the buffers and decoder objects for each session. Depending on the size of the language model, you might be able to handle about 2-3 session on a Raspi4 at the sime time. |
Beta Was this translation helpful? Give feedback.
-
|
Great @DanBmh it certainly sounds very interesting 😎 |
Beta Was this translation helpful? Give feedback.
-
|
Yes, that's possible. Example is here: https://gitlab.com/Jaco-Assistant/Scribosermo/-/tree/master/extras/misc#testing-transcription-on-raspberry-pi. Installation is very easy, see provided Containerfile. I did test it on 32bit Raspi, transcription runs faster than realtime (~0.7x) with the full English language model (and even faster with small customized LMs). |
Beta Was this translation helpful? Give feedback.
-
|
If I read your |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, you can use the prebuilt wheel for armv7, it has the same version as the latest releases from DeepSpeech project. For the aarch64 version check out DeepSpeech's forum, maybe someone already did build it, I don't have one for this as well. |
Beta Was this translation helpful? Give feedback.
-
|
I can't find the aarch64 version anywhere on the web :-( Guess I just have to wait until I can prepare another 32bit installation. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Florian, everyone!
My experiments with offline ASRs
My simplistic/temporary comparisons:
Temporary conclusion: I vote vosk :)
BTW, Chapeau to Michael dissemination and for Rhasspy prject and also chapeau to your works, Florian!!
Websockets? I tend to agree about the fact websocket are a possible winner, nevertheless, considering that your clients are, if I well understood, mobile phones web app or native apps, I want to share a possible warning using raw websockets. Questions: are your clients every possible device? I'm asking because, in my modest experience, every time I used raw websockets I had a lot of communication issues, by example you have to manage re-connections / errors with your own protocol, compatibility, data structure when mixing binary (audio) and others data types, etc.. So on a client-server architectures where clients are remote web apps (on the browser) I recently preferred to use using socket.IO. By example, this is an architecture I build (see slide 21): See a comparison of features here: Another problem I see with websockets is the fact that maybe you want client submit an "object", a data structure that not only containing the audio binary data streaming, but also some attributes, e.g. the language model (or the language), the grammar, ad ID, the data format, etc. If so, you have to send this object with a binary serialization protocol on top of websockets... like protocolol buffers, etc. BTW socketIO seems to be able to send objects containing binary/blobs, see: BTW, pseudocode of a server that ingest audio data using socket.IO: To be honest I'm not sure that a websocket/socketio is the best interface for a server interface that has to be "not-hardcoded". I have to think about it.
Sentence-based (message-based) transcript vs streaming-based transcript In the long term, I tend to agree with you that streaming is "better" to cover all possible application scenarios (dictation). My consideration is that it's maybe not so useful in the context of to manage back and forth turns (conversations!) on a voice assistant. In theory It's the Dialog Manager (DM) in charge to reply to "partial" (streamed) sentences, but it's really hard to build a dialog manager able to be "smart" to reply to user with not-concluded (partial) sentences. So in general I'd go for a message based approach in cases of conversational systems. BTW, streaming processing could be instead useful for vertical applications where the assistant has to detect and react as-soon-as-possible to the user (security at work in hazardous contexts, etc.) but please note that in this case the DM probably has to process emotional/ton metadata and currently we don't have (mainstream) ASR technology that supply this kind of audio/speech/non-verbal info interpretation.
Compressed versus uncompressed - audio transfer In my previous software I did the uncompression task server-side, converting an OPUS FILE to a WAV FILE using ffmpeg. Stupid maybe but at the end ffmpeg process run pretty fast.
Yes for a specific dictation application,
Not necessarily. Using Vosk or DeeSpeech (Coqui) you can load model(s) in memory ONCE at the server startup time. Of course with this choice, at runtime clients could ask a transcript on a subset of possible languages (or models). BTW splitting model creation from transcript task is the basic key to optimize run-time transcript latencies.
Yes, a more note about websockets and the global architecture:
See above: Vosk doesn't block, DeepSpeech blocks!
BTW, on the client request I'd optionally add, if ASR anable the feature, the daptative grammars" attribute. See an example using Vosk: https://github.com/solyarisoftware/voskJs/tree/master/examples#transcript-http-server related to the proposed architecture here:
I'd go for a JSON / data structure. Hope to be useful. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Giorgio, thanks for all your thoughts and insights 🙂
I think most of your observations align with mine. Of cause (as you mentioned) the model plays a big role but this effects all systems more or less in the same way (big model = more RAM = longer processing time). I usually favor the smaller models because I think in any case its still necessary to adapt the LM or use adaptive grammar to get the required accuracy. Since you can use the Teach-UI in SEPIA to define new commands (and obtain a list of those sentences via API) I hope we can use them to train the LM and gradually optimize the assistant for the own use cases. The tricky part here is to generate the dictionary entries for unknown words automatically (@synesthesiam has some experience with that, I think we can use the same scripts that do the job for TTS). Currently my plan is to do the first integration using Vosk but try to keep a modular structure wherever possible to add Coqui and Scribosermo later. Some reasons for Vosk are mentioned above but there are a few more:
Thanks 😃
Clients can be everywhere. SEPIA's default client-server connection is based on WebSockets (that's the chat-server component) and I never had major problems with it. I mean if you have a bad connection you will have problems in any case, especially in a voice assistant where you need something like 3s response time at most. If the connection drops it will buffer and try to reconnect and then exponentially back-off. To my understanding this is the recommended default method and its not too hard to implement. Far more complicated is the question what the client is supposed to do in the meantime since you should block certain interactions while the reconnect process is happening. It took quite some tweaking and refinements to get this right, but this is independent of the actual implementation of the connection itself.
All configuration options like language model, grammar and so on are handled in the welcome event when the connection is authenticated and established. From that point on you will stream audio bytes until the session is done. Actually the language model will probably only be switchable via a HTTP endpoint since it simply takes too long to load models on the fly (one server = one model at a time, at least for Raspberry Pi - see below for further discussion).
This is indeed an interesting point, but so far the most complicated thing to send was a simple array buffer and that can be as easy as
I'm not sure if we are on the same page here. The scope of this server is only to transcribe audio into text. Partial results are usually a cosmetic feature indicating to the user that something is actually happening and its going in the right/wrong direction. Dialog management is done after the final result and not by this server/client.
Actually it is (see beta realease of my new audio lib for SEPIA, documentation will follow at some point ^^). Chrome and Safari can do it, but what they do is using an integrated resampler with unknown properties and performance. The new SEPIA web-audio library uses a WASM compiled Speex resampler with 10 levels of quality to resample the audio buffer in realtime or optionally a FIR based resampler with static properties.
Good question and the answer is I'm not sure 😋 .
I didn't know that and it might be interesting to play with this! For most of my RPis its probably not really an option but a 8 GB version might be able to handle it 🤔
I understand why you like Socket.io and I might be biased here by my personal taste for "as close to the core as possible if its not too hard to handle" ^^. The main reason why I don't want Socket.io is that you get an additional dependency and message overhead that you probably only need to make your life a little bit easier. Plain WebSocket is something that almost any Programming language and even some micro controllers can handle and the message size is reduced to the absolute minimum. With Socket.io you have to rely on given client implementations for your language. I quickly checked Python support and saw that the library is 4 years old! Other languages might not even have one at all. In addition most of the Socket.io features are not relevant for this use-case (HTTP polling fallback, broadcasting features, name-spaces, etc.) and messages are larger.
I wouldn't call it evil to send a WAV file but this usually implies that your whole process looks like this: record the full file, send the full file, wait for result. In my opinion the only way to get a good response time is to stream while recording and transcribe as soon as you see the first chunk. If your complete voice command is 3s this method is already at minimum 3s faster since you don't wait for the user to finish speaking.
YES! Thanks for bringing this up and the example 😃
Yes, thats kind of required. I was rather asking myself what the best JSON format is ;-). Web Speech API for example is kind of awkward somehow, it roughly looks like this (0, 1, ... being alternative transcriptions): ... but for some reason a bunch of smart people decided that this is a good format ^^. I'm still evaluating if they were right :-p Ok so this has become quite a long text 😬 but I hope I could clarify some things or at least make clear why I have certain opinions about topics you mentioned 🙂 . cu, |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for clarifications feedback,
Well, more RAM doesn't means longer latencies, in my simplistic tests (with Vosk). I got these latencies
I agree, with some notes:
That's an interesting "machine learning" approach (I guess used by google and all ASR by big players) ;-) and yes it's an interesting topic, with a single note: it's a long-processing task not feasible for realtime requests, immo it's a kind of backend / batch task and, worst, it maybe requires human annotation/supervision...
Yes! With a warning: to increase performances I'd avoid to use containers on small computers as server (you mentioned a Rpi4). TBV.
I see. I'm biased on a different approach / architecture maybe, where I see the ASR as a backend component used "internally" on a multilayer/multi/engine system: But instead it seems that you are thinking on a "flat services" like this: Is that correct?
That's a possible solution! your ASR:
It makes sense for me, but I'm still perplexed, because this does NOT allow to modify request parameters dynamically for each atomic request. By example a transcript request can't specify at run-time a state_id (e.g. an adaptive grammar to be associated to the speech).
Yes, standard websockets are perfect if you send just one type of data (for us here the audio as PCM buffer), my doubt is about how to manage with websockets a complex object (type data structure) containing the binary audio data and some attributes (metadata, say strings) related and contained on the same binary websocket message.
Now is not clear to me where/who is managed the dialog management. Could you please clarify? Thanks!
Weird. I tested months ago, on updated Chrome/Firefox browsers and on recent Linux/Android, using Mediarecorder API and I was NOT enable to get anything else from OPUS encoding @ 48Khz. I'll do more tests asap.
That's could be a way, TBV.
Good points.
I don't want to convince about the fact Socket.io is "better" and I see your points.
You are right, just consider an important usual requirement: in my case/but I believe that's very general, a (multi-user = server) voice assistant has to track end user speech++ requests (and assistant voice++ responses) as log FILES. Taht's for tracking/monitoring. So you generally want to save audio files. Nevertheless I admit that to optimize latencies, file storage could be a parallel task, and passing through disk give slowest performances.
What do you mean when you say "complete voice command? it's the elapsed time from the end-user begin of speech to the end of assistant response? Cpuld you clarify? UX requirements must drive technical feasibility :-) Thanks |
Beta Was this translation helpful? Give feedback.
-
Thanks, giorgio! I'll keep going until they stop me ;)
Standard websockets differentiate between text (UTF-8) and binary data. The python websockets library, for example, returns a @fquirin, if you're going to use Vosk for your initial test, I'd like to create a similar STT server for Coqui STT. How do you want to handle the authentication? I had planned to just mirror what Home Assistant does, essentially:
After that, the client should send a start message to configure things; something like: { "type": "startsession", "siteId": "...", ... }and then the client starts streaming audio. Responses from the server would have the same Thoughts? |
Beta Was this translation helpful? Give feedback.
-
Yes for me. Let me think about how to implement this: Option 1: Websocket server ingest "self-consistent" websockets requestsAs far as I understood, standard websockets allow text OR binary payloads. We need binary setting to transfer audio (PCM/OPUS), avoiding to convert binary in text. So, if I understand your statement, the client message could be made by 3 parts
As proposed here: https://stackoverflow.com/a/38022867/1786393 Option 2: HTTP/JSON API server "spawn" websockets
In this hypothesis (option2), the "negotiation" between clients and server concerns not only authentications, but also other attributes/metadata, as language, model name, state, etc. This complicates a bit message exchanges, not just at start up time but at run time, if by example client would change settings (language, grammar, etc.) on the fly for single messages. My fear is about the syncronizationbetween HTTP/JSON "commands" requests/reply and the websockets data flow. On the other hand, in option 1 requests are self-consistents with the cons that the websocket here are a "custom" binary interface with drawbacks (e.g. client.server binary formats compatibility, etc)
Please consider my notes about the apparent NO multi-tasking architecture of Coqui STT (that's so far essentially the same of DeepSpeech, see my previous comment: #112 (comment) So Vosk recognizer spawn threads for each request, "by design", whereas Coqui STT doesn't. You have to build a multi-pocess/multi-thread yourself. |
Beta Was this translation helpful? Give feedback.
-
Wait @solyarisoftware , are you saying you got faster transcription with the large model on the SAME! machine? 😲 ... I gotta test this as well! 🤔
I was a bit concerned about this as well but so far I haven't seen notable differences and in one test the container was even faster! 🤷♂️ 😅 I will keep an eye on it but containers are what people are asking for ^^ (and it really saves a lot of pain :-p).
That's about correct ... well actually the SEPIA assist server currently also manages the connection to the TTS server but I want to offer a 3rd interface (client, SEPIA, 3rd party API) with a direct connection to a MaryTTS compatible API as well (e.g. Larynx/Open TTS by Michael) just to give users all options. SEPIA was always about total customization so I guess it makes sense.
Why not? It kind of works identical to the Web Speech API: idle -> connect to ASR server with grammar option -> stream -> transcribe -> final result -> disconnect -> idle. That's about as atomic as it gets I guess :-)
In the case of SEPIA its basically the client but in theory it can be the assist or chat server as well with a bit of code refactoring. The SEPIA assist server was initially built to work behind a load balancer so state management would have been hard to synchronize. Anyway, this STT server is just responsible for speech to text. Its only one of the puzzle pieces and its designed to work independently. Think of the Web Speech API again ;-).
I remember commenting it somewhere and before I rebuilt the audio lib I assumed it wasn't possible but it works on Chrome/Chromium (probably Edge) and Safari. I'm not sure if they updated it or if I just did it the right way this time ^^. But even though it works I prefer the manual resampling, it gives you more control over performance and quality and you can combine it with the PCM 16bit integer conversion required for WAV encoding. I'm planning to but a demo of the audio lib online but I'm still changing parts all the time. Its pretty cool though, you can chain all the modules and build a complete pipeline of audio recording tools required for speech 😁
I was thinking about this as kind of an opt-in feature for users to donate their recordings. You can always dump the stream to a file after the stream. This is what I'm doing in the old server actually :-)
Ah, in this case I mean audio-input-start to audio-input-end :-) "Command" is confusing here "input" might be the better word.
Thats the plan @synesthesiam 🙂
It would be great to have both. The plan was to keep everything the same up to the point where we feed the chunks into the recognizer. I've seen the necessary code for Vosk (
That is pretty similar (almost identical) to the way SEPIA authenticates to it's chat server so I agree we can take it.
Actually we could send this together with the answer to the auth. request to save one ping-pong 🤔 .
You mean The weird thing about the Web Speech API results is that they have: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks Florian for clarifications.
Yes, I confirm my previous statement, but I did STT of few test audio files: These are recognized with 100% accuracy, and about Recognizer latency (left out the initial/one-off model load in RAM)
It's to be verified with a vast set of speech files. TBD.
BTW, I remember that Nicolay (Vosk author) discouraged to use dockers on a RPi, but I never tryied.
In this way, if I well understand:
That's correct?
I see. In the case your STT accept (dialog state dependent) grammars you need to connect STT with a dialog manager. That could be done also client side. In this scenario
I'd prefer to manage STT-DM logics, server-side, but i'm biased by a conversation-based multilayer architecture :)
On a mobile phone? Android/iOS?
Great for flexibility. |
Beta Was this translation helpful? Give feedback.
-
This is really interesting .. and confusing somehow 🙈 . What are the specs of your machine? (CPU, RAM) Is this accelerated by the GPU?
Not quite. Check out the order of events in my latest post 🙂 . Hope that clarifies it.
For SEPIA its a historical development ^^. The first version (ILA) was a Java application that did everything in one program, then I switched to client-server architecture where the client did STT, then I replaced this STT with another self-hosted server 😁 . I'm sure there are pros and cons for all the possible architectures ;-).
Mobile and Desktop.
Yes. This will be something I'd like to improve as soon as the WAV version is running. |
Beta Was this translation helpful? Give feedback.
-
It's a stupid HP laptop (stupid because i'm experiencing strange core usage that I believe are caused by cpuppower saving not controllable settings): I don't think Vosk-API use GPU for the run-time recognizer. TBV. As said in previous comment, my tests are absolutely superficial because I used just few English language sentences, but it seems to me that's the behavior is "stable". More seriously, we need to test different "language models" (of differen ASR technologies) with a possible common shared metrics (language, accuracy, latency) and common test data (words/sentences).
I'm still confused, I admit.
Thanks |
Beta Was this translation helpful? Give feedback.
-
I will need to make some tests on the RPi4 with live speech and then we'll get some more data. If its true in general its actually awesome :-D but this would mean the language model is not limiting the speed which seems counter-intuitive assuming the engine has to search in a much larger graph for the best match 🤔
Yes, but the HTTP endpoint is basically only for general server settings and info. At least in the old STT server its used to switch the model which is a static, global settings (this is debatable if you have a large machine that can hold multiple models in memory and run at the same time). It will also be used to train language models :-).
Not really, this is done in the |
Beta Was this translation helpful? Give feedback.
-
I had assumed I'd just build a Python server separate for this. Part of the reasoning is that I plan to add wake word detection to my server as well. When configured, this would first pass the audio chunks into a wake word recognizer, then into STT once a detection has occurred. Without wake word detection configured, it would just be an STT server.
Let's just stick with the way SEPIA does it then 🙂 Do you have a doc describing that?
Sure, that's fine with me. What do you propose as the format?
Yeah, that seems reasonable to me. For results, though, I propose we have: {
"type": "result",
"result": {
"isFinal": true,
"transcript": "..."
},
"resultIndex": -1,
"results": [...]
}where the "result" is the best result and the "results" list is empty unless an n-best list is supported/requested. I don't see the point in making everyone dig through an object tree just to get the transcript. |
Beta Was this translation helpful? Give feedback.
-
Ok I think it makes sense to keep this separate because it requires a long running constant connection and audio stream which is a bit out of my initial scope (SEPIA does VAD and WW in the client). It is interesting though ^^.
The message format is a bit more complex between SEPIA client <-> chat server, but maybe the SEPIA client <-> CLEXI (something I built to augment the client) connection is a good reference. I've slightly adjusted the format for our use-case:
If authentication failed server sends message
What do you think? 🙂 |
Beta Was this translation helpful? Give feedback.
-
Just a side note on this: If you are running the WW-service on the same server as the STT-service, you could think about using the STT-service for WW-detection as well. I did run a short experiment on this with Jaco and the smallest Scribosermo model in which I found that this can work better than Porcupine's models for specific phrases. Downside was that it only did work well for a few phrases which are commonly used like "i don't know", even though it is theoretically possible to use any text as wake word. And you might have a slightly higher power consumption, because the model is larger than a model tuned for WW-detection only. |
Beta Was this translation helpful? Give feedback.
-
|
Too add my own thoughts is that a TTS audio stream is all that is needed from TTS as audio delivery frameworks such as squeezelite, snapcast & airplay already exist are already maintained and likely to do a better job already than some sort of streaming service currently from scratch. You don't need a streaming audio STT you just need the metadata to a audio multiplexer can direct to the right mixer. If you look at available services they seem to split into room/zones and corresponding channels and shouldn't embed specific technology that will exclude any but just have a middleware bridge to connect to all. You just need bridge ware that can provide templated metadata translation that contains a number of services for delivery of a audio stream that map the destinations. Zones/Rooms are sessions and channel is a device that require mapping services through the conversion of parameters through templated mappings. Compression/codec prob depends on the service you are feeding to what its expecting at the mixer and not end delivery but guess that should be part of the template and service install. If a VoiceAI system has there own embedded audio system then that is just another service to be added and templated, even if source and destination 'moniker' is the same which is unlikely with 3rd party services. |
Beta Was this translation helpful? Give feedback.
-
|
I'm sorry, but I'm not sure if I can follow your arguments 😅 .
Are you suggesting to run another server that has the only purpose of receiving and distributing audio streams? It surely sounds interesting but what is the advantage here for STT? I fear that it increases complexity a lot and you pull a bunch of dependencies into your project that might not available cross-paltform in the hope to safe you some headaches of handling audio buffers. In addition 3rd party audio libraries often don't give you the low-level access to your audio buffer that you need for e.g. client side processing.
I'm not completely against it but imagine you have a website with a voice assistant (e.g. the SEPIA client ^^) and your own STT server. Now you already disassemble the microphone input audio to the audio-buffer level because you need it to do e.g. wake-word and VAD. Then you simply send this audio-buffer to the server in chunks and thats it ;-). I wouldn't even call that an "audio system" at least not whats going on betwenn client and server ^^. |
Beta Was this translation helpful? Give feedback.
-
|
Yes basically. The only streams from STT are concurrent which even then is more the exception and the streams are very simple but its the meta data and mapping to those systems so that in a 'room / zone they can be easily implemented as why use the inferior or have additional audio capability when high quality is already there. Also connecting and controlling media systems has huge benefits for Voice Ai but there are clear partitions in a voice AI system and the audio system is one and it is separate. There isn't an advantage for TTS as its job is to create Text To Speech whilst audio distribution is another job. You just need a simple socket or pipe and a zone and channel like metadata to describe delivery but audio delivery of any type is something else job. Its not even much of a service as its just a bridging service likely running off mapping templates to allow interoperability to many. The whole point that it isn't in STT is that there are no dependencies and should be a contained server in its own right even though relatively simple where its mapping and delivering audio streams. |
Beta Was this translation helpful? Give feedback.
-
|
So looking at the current draft what exactly would you change? And what would be the benefit? Let's leave TTS out for now since this discussion is about the generalized interface for the STT server. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah its just that I see strong parallels with audio in and audio out hence SST / TTS and for me there is a before and after that aggregates sources via zones/rooms to devices. Again so there are no dependencies from mic device / KWS there is an intermediary that doesn't have specific protocols but again acts as middle ware and maps metadata and converts audio and provides base operations to feed a ASR and can act as a basic audio queue. You need look at what libs are available and work at the lowest common denominator whicjh is likely esp32 and have that as a 'starter' connection rather than preference for any specific lib you may of used. AMR-WB or codec choice really needs to be a consideration and raw PCMs such as 16Khz wavs over WiFi in any form of concurrency quickly build up and its like 2 decades since the Ipod sort of made the codec obigatory. If SST can aggree on a fixed MFCC alg than like Google Lyra with the heavy resynthesis it provides an audio stream of the highest compression and still can be compressed further with simple gzip methods. I am a big fan of distributed array mics which are just multiples in a zone and the simple function of using the stream in the zone with the best KW hit is often not implemented. ASR shouldn't be dealing with streams just like TTS and there should be a middleware interoperable platform that can work with many in a multi-homed inout that doesn't dictate use apart from the plugin that connects to the ASR of use. |
Beta Was this translation helpful? Give feedback.
-
|
If it doesn't make sense here is where I replied to a suggestion in the Rhasspy community and it sets a better context of explanation. As if you take the commonality of input and output look at the problems of interoperability and length of life. https://community.rhasspy.org/t/almond-rhasspy/2809/2?u=rolyan_trauts |
Beta Was this translation helpful? Give feedback.
-
|
Hi everybody @synesthesiam , @solyarisoftware , @DanBmh Everything is in the 'dev' branch right now: https://github.com/SEPIA-Framework/sepia-stt-server/tree/dev |
Beta Was this translation helpful? Give feedback.
-
|
Awesome! Ty 😁 |
Beta Was this translation helpful? Give feedback.
-
|
Hey @synesthesiam thanks to your pre-compiled files I've managed to update my old language model adaptation repo (Zamia fork) to create custom, small-domain LMs (or larger if you have a big corpus) that are based on Zamia Speech models and work in Vosk 😎 : https://github.com/fquirin/kaldi-adapt-lm . |
Beta Was this translation helpful? Give feedback.
-
|
@fquirin I have a pre-trained Phonetisaurus g2p model for Zamia here: https://github.com/synesthesiam/en-us_kaldi-zamia/ For converting back and forth between phoneme sets, I've recently added support for this to gruut-ipa. It hasn't been thoroughly tested yet, but maybe this could be an opportunity 🙂 |
Beta Was this translation helpful? Give feedback.
-
|
Looks interesting :-). So if I have the Zamia has some IPA dicts in the original form as well: https://github.com/gooofy/zamia-speech/tree/master/data/src/dicts [EDIT] Guenter has some Python scripts as well, e.g.: xsampa2ip have you seen them? |
Beta Was this translation helpful? Give feedback.
-
I've created an Issue about it in your repo :-) : rhasspy/gruut-ipa#2 |
Beta Was this translation helpful? Give feedback.
-
|
@synesthesiam , @DanBmh Thanks to @domcross I've made some progress with Scribosermo as well and managed to build a test-system on aarch64 (Raspberry Pi4) and am64 🥳 . I've collected all my Scribosermo knowledge here: https://github.com/fquirin/scribosermo-stt-setup @DanBmh There is some confusion on my side regarding the Scribosermo model checkpoints. Can the German QuartzNet actually be used in an open-source project like SEPIA or are the Nvidia terms-of-use prohibiting that? 🤔 |
Beta Was this translation helpful? Give feedback.
-
|
Great to read 👍 Regarding your question about the model license, I think there should be no problem if you use them in an open source project, but I'm not completely sure. I didn't fully understand Nvidia's terms-of-use and if they still apply after training over the weights, because you get a different model then. |
Beta Was this translation helpful? Give feedback.
-
That was exactly the point I didn't understand either 😕 |
Beta Was this translation helpful? Give feedback.
-
|
The API docs have been updated to describe the general process flow and the 'welcome' event (authentication, model/language selection, mode, grammar, etc.), just in case someone was interesting in building a Python or Node.js client ;-) 😁 ... . |
Beta Was this translation helpful? Give feedback.
-
|
I've updated the STT-Server repository with the BETA version of a Python client library and demo script that demonstrates STT on the command line 🙂 . |
Beta Was this translation helpful? Give feedback.
-
Hi everybody 🙂
I've been planning to update the SEPIA STT server for a while now but due to technical challenges and time constraints I haven't really managed to make any progress since 2019.
Now it is 2021 and I feel like the circumstances have improved a lot (except maybe the time constraints 😅 ). On the one hand I've managed to build a new web-audio library that simplifies audio processing in web-based/mobile clients (soon to be released). On the other hand speech recognition tools like Vosk or Coqui have made it a lot easier to install and run the STT engines on different platforms and to update the language models. Besides that Michael from Rhasspy is doing a great job of piecing together a lot of open-source voice projects and resources that fly around in the web to things like the Larynx TTS server (and much more).
To cut a long story short I'm going to rebuild the STT server more or less from scratch and I want to use the chance to make it as compatible as possible to all the other great projects out there. That's why I'd like to invite you to a little discussion before I start 🙂 .
Here are my ideas/requirements:
Open questions:
Rough description of the client-server communication (draft):
I'm building this specifically to work together with the SEPIA web-based cross-platform client but I'd be very happy if the server can become useful for many other open-source projects! 🤖 💬
Any comments, questions and ideas are greatly appreciated! 🙂
cu,
Florian
Beta Was this translation helpful? Give feedback.
All reactions