New release: 2.0.0a (pre-alpha)

[hayabhay]

There was some delay in working on this due to other commitments but moving forward, this will be more actively maintained.

In terms of priorities, there will a strong bias towards a self-hosted UI and models with the goal of making it easy to search and reason over multi-media content from both the web and local files. To make this easier and also allow other folk to add customizations, this code base has now been rewritten (mostly) and split into two parts:

• A small Python utility package called frogbase that contains all the backend logic for the UI. This can be used as a standalone package or integrated into other applications.
• A slimmer Streamlit UI that provides a thin wrapper around the frogbase package built purely with self-hosted applications in mind.

Featuress

• More content sources & formats

  • The use of pytube has been replaced with yt_dlp. This unlocks content download from a broad range of media platforms like YouTube (channels, playlists, videos), TikTok, Vimeo etc. (full list)
  • Local files can now be ingested from a directory instead of just a single file.

  Sources can now be added in as a list of urls and/or local file paths.

• Semantic Search

  • The search functionality now includes semantic search over transcript contents instead of a simple substring search.
    This is done using sentence-transformers and hnswlib

• Updated Streamlit UI

  • The UI now includes the concept of Libraries to further organize media downloads. Libraries are simply subdirectories within the main data directory.
  • Filter & search functionality have been simplified and made more intuitive.