...
Speech Assistant is a micro-frontend application, where all ui code resides. The UI code is built in React using Carbon Design System.
Vakyansh is used as Speech to Text engine.
speech-recognition-open-api-proxy converts a socket connection to GRPC call, so that a user can continuously audio stream the message from browser.
speech-recognition-open-api receives the audio stream and return the converted text back.
The two Vakyansh applications are packaged in Speech Assistant Package, to be used with Bahmni docker
Speech Assistant is bundled in Bahmni apps and is deployed available at https://speech.mybahmni.in/
Setting up Speech Assistant Feature with Bahmni:
Setting up Vakyansh:
Clone Speech Assistant Package
Download models by running download_models.sh in the scripts folder.
Run docker-compose up -d. It will bring two containers: vakyansh-api and vakyansh-proxy.
Setting up Speech Assistant Frontend:
As Speech Assistant is a microfrontend, so to use it, corresponding bundle.js needs to be included in the script tag.
bundle.js can be availed either from this url, which contains the latest bundle, or while building the app present in the repository.
Directions:
Currently the language model been used by Vakyansh is normal English model. To understand medical words, model needs to be trained with the relevant vocabulary. Trained medical model then can be used by Vakyansh to get the proper text back.
Vakyansh api works better when deployed in a GPU machine. One instance of api can easily serve upto 10 concurrent audio connections. If it needs to be increased further, the api needs to be scaled.
If NLP library is applied to understand the meaning from the sentence, then the usecase could be extended to other consultation tasks like Medications, Symptoms, etc.