Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] [UI] Add voice dialog #2285

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

GiviMAD
Copy link
Member

@GiviMAD GiviMAD commented Jan 26, 2024

This is a WIP PR for the issue #2275

The PR uses these audio WebSocket which is still in review openhab/openhab-core#4032 so you need to use the org.openhab.core.io.websocket and org.openhab.core.voice bundles from there if you want to test it.

It is built on top of the webpack 5 upgrade PR #2267, because on webpack 4 you need to use loaders and I didn't want to generate more work to do in a future upgrade. The actual PR changes are in the last commit.

Changes added:

Screenshot 2024-01-26 at 17 12 13

I have added that panel at the end of the "help & about" page (it should only be displayed if the necessary APIs are available), it contains the following options:

  • "Enable Voice Dialog": When enabled the dialog execution will start after the first user event on the page (needed to start the microphone stream).
  • "Connection ID": And auto generated code that is used to register the audio components. Ex: the id of the registered sink will look like "pcm::ui-52-68::sink".
  • "Selected Keyword": Rustpotter config, choose rustpotter model file. At the moment the file is selected using an input type file and persisted using an IndexedDB instance.
  • "Keyword Threshold": Rustpotter config, threshold in range 0 - 1.
  • "Keyword Min Detections": Rustpotter config, min number of positive scores to emit the detection.
  • "Listening Item": Dialog configuration, Item to toggle while listening the command.
  • "Location Item": Dialog configuration, used by the Standard language processor to improve the Item selection.

Modifying any of the options triggers a page reload.

The dialog will use the voice services configured at "Settings / Voice". Once the dialog is running it can be triggered using the voice if the rustpotter model was provided, or using the listenAndAnswer action. For example from the oh console by running voice listenandanswer --source pcm::ui-52-68::source --sink pcm::ui-52-68::sink.

For now I have placed some console warning in the events emitted about the dialog state.

I wanted to open the PR in case you can give me some feedback about the integration of the feature or in case someone wants to dig into the implementation.

Best regards.

@GiviMAD GiviMAD requested a review from a team as a code owner January 26, 2024 17:49
@GiviMAD GiviMAD force-pushed the feature/voice_dialog branch 2 times, most recently from 89bd614 to 355c5fc Compare January 31, 2024 19:31
Copy link

relativeci bot commented Jan 31, 2024

#2674 Bundle Size — 11.05MiB (+0.65%).

25d363c(current) vs d16fe3a main#2671(baseline)

Warning

Bundle contains 2 duplicate packages – View duplicate packages

Warning

Bundle introduced one new package: reentrant-lock – View changed packages

Bundle metrics  Change 7 changes Regression 3 regressions
                 Current
#2674
     Baseline
#2671
Regression  Initial JS 1.9MiB(+0.11%) 1.9MiB
No change  Initial CSS 577.21KiB 577.21KiB
Change  Cache Invalidation 18.3% 18.28%
Change  Chunks 229(+0.88%) 227
Change  Assets 254(+1.6%) 250
Change  Modules 2960(+0.34%) 2950
Regression  Duplicate Modules 155(+0.65%) 154
No change  Duplicate Code 1.8% 1.8%
Regression  Packages 99(+1.02%) 98
No change  Duplicate Packages 2 2
Bundle size by type  Change 2 changes Regression 2 regressions
                 Current
#2674
     Baseline
#2671
Regression  JS 9.26MiB (+0.78%) 9.19MiB
Regression  CSS 867.09KiB (~+0.01%) 867.02KiB
No change  Fonts 526.1KiB 526.1KiB
No change  Media 295.6KiB 295.6KiB
No change  IMG 140.74KiB 140.74KiB
No change  HTML 1.38KiB 1.38KiB
No change  Other 871B 871B

Bundle analysis reportBranch GiviMAD:feature/voice_dialogProject dashboard


Generated by RelativeCIDocumentationReport issue

@florian-h05 florian-h05 added enhancement New feature or request main ui Main UI awaiting other PR Depends on another PR labels Feb 28, 2024
@GiviMAD GiviMAD force-pushed the feature/voice_dialog branch from 355c5fc to 0091917 Compare January 8, 2025 23:01
@GiviMAD GiviMAD force-pushed the feature/voice_dialog branch 2 times, most recently from 624b940 to ce0a9e6 Compare January 8, 2025 23:10
Signed-off-by: Miguel Álvarez Díez <miguelwork92@gmail.com>
@GiviMAD GiviMAD force-pushed the feature/voice_dialog branch from ce0a9e6 to 25d363c Compare January 10, 2025 00:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting other PR Depends on another PR enhancement New feature or request main ui Main UI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants