Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Web UI to support querying from browsers. #250

Merged
merged 64 commits into from
Feb 9, 2024

Conversation

junhaoliao
Copy link
Member

@junhaoliao junhaoliao commented Jan 26, 2024

Description

  1. Add component WebUI to support querying from browser clients.
  2. Add Taskfile configuration webui and depedency configuration nodejs.
  3. Update CLP package util & config scripts.
  4. Update CLP Config template to include WebUI address override options.

Validation performed

Build and Load + Stats Visualization

  1. Loaded CLP Docker image.
  2. Built package under project root with task command and observed all tasks to complete without critical error.
  3. Start CLP with <BUILD_DIR>/clp-package/sbin/start-clp.sh and observed that all components started properly.
  4. Visited http://localhost:4000 on the running Linux host and observed the web page to load without error messages in the browser debugger console. To be exact, the access was done via an SSH port-forwarding channel since the test was done on another Windows machine.
  5. Compressed example logs hadoop-24hrs/logs/i-00c90a0f with <BUILD_DIR>/clp-package/sbin/compress.sh and observed no error.
  6. Observed the compression stats (e.g. space savings, begin & end timestamps, files & messages count) to show up in the WebUI in 5 seconds.
  7. Stopped component webui via command <BUILD_DIR>/clp-package/sbin/start-clp.sh webui and observed messages showing the component has been stopped.
  8. Modify <BUILD_DIR>/clp-package/etc/clp-config.yml to specify custom port 9999 for webui. e.g.,
    webui:
      host: "localhost"
      port: 9999

Query

  1. Built and loaded CLP package with WebUI. Visited WebUI address in Microsoft Edge Version 121.0.2277.81 and observed page to load and show query User Interface.
  2. Performed a search with search string FsDatasetAsyncDiskService and Time Range from Unix epoch to date which returned 172 results. Toggle browser search by Ctrl + F with the same search string and observed every result entry has matching part highlighted. With MongoDB Compass, verified that the results count (172) matched total document count for the search job results collection in MongoDB.
  3. Repeat Step 2 with Time Range being the "Last 15 Minutes" by selecting the option in "Presets" dropdown.
  4. Without refreshing page, enter another query with search string fsdatasetasyncdiskservice and observed a "No results found" message presented in the UI, which is expected because the search is not case-insensitive.
  5. Opened anther tab. In the new tab search with DirectoryScanner, and as the query is progressing, quickly performed another search with FsDatasetAsyncDiskService. Observed no race condition using the browser highlight method described in Step 2.
  6. Validated other features including:
    1. Expand / Collapse left Menu. The setting is persistent over page refreshes.
    2. Expand / Collapse Search Control panel. The setting is persistent over page refreshes.
    3. Change "Max lines per result". The setting is persistent over page refreshes.
    4. Cancel a running search.
    5. Clear results.
    6. Trigger lazy loading by scrolling to bottom of the page, when query results are more than 10.

Logging

  1. Built and loaded CLP package with WebUI.
  2. Ran <BUILD_DIR>/clp-package/sbin/start-clp.sh results_cache to stop the MongoDB "Results Cache", which is a dependency of the WebUI.
  3. Visited http://localhost:4000 and observed the web page to keep displaying a loading animation because the Meteor.js server is unable to connect to the MongoDB to retrieve user information.
  4. Inspected log files webui_error*.log and webui*.log inside <BUILD_DIR>/clp-package/var/log/<host>/webui and found the operation sequences and errors getting logged properly.

@junhaoliao junhaoliao deleted the branch y-scope:main January 26, 2024 17:52
@junhaoliao junhaoliao closed this Jan 26, 2024
@junhaoliao junhaoliao deleted the main branch January 26, 2024 17:52
@junhaoliao junhaoliao restored the main branch January 26, 2024 17:52
@junhaoliao junhaoliao reopened this Jan 26, 2024
@junhaoliao
Copy link
Member Author

PR was closed due to manual error. Reopening now.

Please continue to review.

@junhaoliao junhaoliao requested review from kirkrodrigues, wraymo, gibber9809 and haiqi96 and removed request for kirkrodrigues January 26, 2024 17:55
Copy link
Member

@kirkrodrigues kirkrodrigues left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An incomplete review, but something to get started with. You'll also want to merge main in at some point since there've been some large changes.

Taskfile.yml Outdated Show resolved Hide resolved
components/webui/README.md Outdated Show resolved Hide resolved
components/webui/README.md Outdated Show resolved Hide resolved
components/webui/README.md Outdated Show resolved Hide resolved
components/webui/README.md Outdated Show resolved Hide resolved
components/webui/imports/api/search/collections.js Outdated Show resolved Hide resolved
components/webui/imports/api/search/collections.js Outdated Show resolved Hide resolved
components/webui/imports/ui/SearchView/datetime.js Outdated Show resolved Hide resolved
components/webui/imports/ui/SearchView/datetime.js Outdated Show resolved Hide resolved
Taskfile.yml Outdated Show resolved Hide resolved
components/webui/README.md Outdated Show resolved Hide resolved
@junhaoliao junhaoliao requested review from kirkrodrigues and removed request for kirkrodrigues February 8, 2024 05:03
kirkrodrigues
kirkrodrigues previously approved these changes Feb 9, 2024
Copy link
Member

@kirkrodrigues kirkrodrigues left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit msg: Add Web UI to support querying from browsers. (#250)?

Taskfile.yml Outdated Show resolved Hide resolved
Taskfile.yml Show resolved Hide resolved
Taskfile.yml Show resolved Hide resolved
Copy link
Member Author

@junhaoliao junhaoliao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good on my end. Kindly please review and merge.

@kirkrodrigues kirkrodrigues changed the title Add Web UI to support querying from browser clients. Add Web UI to support querying from browsers. Feb 9, 2024
@kirkrodrigues kirkrodrigues merged commit d569137 into y-scope:main Feb 9, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants