planning: Jan and Cortex's Extension Framework

[louis-jan]

Problem Statement

Objective

To scale Jan and enable the addition of new features without modifying the core app codebase. This roadmap aims to strengthen Jan's core framework, making it more robust for public use by both the community and core developers. New features will not be limited to bundles; users can opt-in to use them through various methods such as installing from GitHub repositories or extension stores.

Some concerns related to the update:

• How to handle remote provider extensions? Such as:
  • Users can view the latest supported models from the provider.
  • The extension provider can add additional configurations, such as manually entering a model and URL for OpenAI-compatible extensions.
• Bring Ichigo into Jan
• Build web-search feature
• Better RAG
• Code playground
• Jan Console (logs, running playground console)
• Jan Mobile

All of these are great tests for the extension framework update.

More importantly, it should allow Jan to support more platforms, including mobile.

Feature Idea

What are the parts of the update?

Extensions interact with the core API and events to add features, perform tasks, update app data, and enhance app UI components.

We've laid the groundwork for extensible app logic implementation but still need the UI Extension framework. That would be great if an extension can work with the UI API to extend Jan's related features such as:

• Registering a form on the settings page to define an OpenAI model and its endpoint.
• Declaring input shortcuts or types, like audio recording indicators.
• Declaring input settings, such as toggling web search on or off.
• Registering a tool tab on the right panel. From there, it can register windows/forms/buttons. So that it can provide a code playground, web/code review panels.
• Streaming console logs to the app's bottom system bar panel to register a console window tab so users can debug their playground/extensions.
• Adding extension ribbons for users to select and show detail panels.

Core API Update

There are also core APIs/events updates that could improve functionality, such as current app and conversation states:

• Getting the current code block so when users select a code block message, it streams to the Code Playground or similar.
• Getting the selected model or running a downloaded model to perform tasks.
• Intercepting LLM requests to provide extra information like retrieval/web search.

[freelerobot]

Tasklist

• discussions: Remote API Extension #3505
• roadmap: Jan has revamped Remote Engines (e.g. OpenAI, Anthropic etc) #3786
• planning: Jan's path to cortex.cpp? #3690 (Inference Engine Extension)
• planning: Jan UI Extensions Framework #2718
• planning: Jan Theme Extensions API #3600
• feat: Separate OpenAI compatible server support for "local server" to keep using OpenAI in parallel #3214
• feat: support a local custom cortex engine #3764

Tracking all the sub-issues above, since this is a bit of a mega issue

[dan-menlo]

What are the key "Core Modules"?

• Settings (e.g. Settings)
• Threads (e.g. Parameters)
• Hub (e.g. Provider would register models in Hub)
• Extensions Page
• Providers Page

[louis-jan]

Core modules

There are certain core modules that can be extended through an extensible framework.

1. Providers:
There are local and remote providers that can be incorporated, offering models and settings. It can bundle the engine server or route requests to cloud endpoints.

The provider should register supported parameters for each registered model.

2. Parameters & Tools:
Extensions can register tools, which could include embedding, retrieval, function calling, or code interpretation. There could be an extensible user interface on the right panel.

Scenarios:
Users select the "Tools" tab, then they can choose an embedding model for the retrieval tool, or pick a model for the chat with specified parameters. The kind of settings are now quite hard-coded, making it difficult to adapt to new changes. It would be great if the providers can register these settings.

3. Input
Multi-modal input types should be extensible; there should be a universal design where incorporating new input type support wouldn't disrupt the input layout (e.g., iMessage).

Scenarios:
Extensions register a new input type for a specific model, such as audio input. It can construct a React component that interacts with Native APIs, like Microphone, visualizing waveforms, encoding, returning semantic tokens, and building message inputs.

Extension can be replaced to offer a new outlook (visualization) or a new encoder.

4. Settings
Every extension can register settings, allowing users to configure options such as an API Key.

These settings should be removed once the extension is uninstalled.

5. Hub
Each extension can register models, which will be categorized and displayed in the Model Hub. There was no idea on how it can visualize the hub, for now it's just to provide data. Extensions should not impact the shared UI page of them, such as the Extension Store and Model Hub.

6. Conversation
Extensions should be capable of querying conversation data, visualizing some of its functions such as a code playground.
getCurrentCodeBlock, getCurrentMessage, getCurrentFile...

In short:

• Threads
  • Threads list & messages (Data API)
  • Parameters, Tools (Data API + UI)
  • Input (multi-modal - UI)
• Settings
  • Settings Form (Data API + UI)
• Hub
  • Model list (Data API)
• Extensions Page (Data API)
• Providers Page

[imtuyethan]

@dan-homebrew