[BETA FEATURE] Chunked Prompts - Prompt OpenAI with text way beyond the input limits

[chhoumann]

What do you do if you have some text you want to have summarized? Easy, ask ChatGPT to do it.

Now, what if you have a large body of text you want summarized?
Do you ask ChatGPT to do so by repeatedly copying over parts of the text?
That's tedious, and we don't do that here.

Let me introduce Chunked Prompts¹ for QuickAdd.

This will chunk your text into pieces that fit into the input window for the various models. The feature inserts these chunks into your predefined chunk and makes requests to the OpenAI API for each of them.
It's even able to merge multiple chunks together if there's space.

This is incredibly useful for summarizing or transforming large bodies of text. Here are some of the things I've done with it:

• Summarized my book notes that couldn't fit in ChatGPT
• Made flashcards from 200-page PDF contents²

I'm kind of on the fence about whether it should be a feature for QuickAdd, so I've made a script that does the same to get feedback.
It's a very specific feature, whereas I like the plugin having more primitive features that can be used as building blocks.

Setup

First, you'll need the associated script. Here's how you can use scripts in macros (video instructions):

You include the script by creating it as a file in your vault, e.g. chunked-prompt.js and then pasting the script into the file.
Then it'll appear in the scripts dropdown as an option you can add.

Find the script below:

const INPUT_FORMAT = "Input format";
const PROMPT_TEMPLATE = "Prompt template";
const MODEL = "Model";
const VARIABLE_NAME = "Result variable name";
const SHOW_ASSISTANT_MESSAGES = "Show assistant messages";
const SYSTEM_PROMPT = "System prompt";
const CHUNK_SEPARATOR = "Chunk separator";
const CHUNK_JOINER = "Chunk joiner";
const SHOULD_MERGE = "Should merge";

module.exports = {
    entry: run,
    settings: {
        name: `Chunked Prompt`,
        author: `Christian Bager Bach Houmann`,
        options: {
            [INPUT_FORMAT]: {
                type: "format",
                defaultValue: "{{selected}}",
                placeholder: "Enter your input format",
            },
            [PROMPT_TEMPLATE]: {
                type: "format",
                defaultValue: "{{value:chunk}}",
                placeholder: "Enter your prompt template",
            },
            [MODEL]: {
                type: "dropdown",
                defaultValue: "gpt-3.5-turbo",
                options: [
                    "gpt-3.5-turbo",
                    "gpt-3.5-turbo-16k",
                    "gpt-4",
                    "gpt-4-32k",
                    "gpt-4-1106-preview",
                    "text-davinci-003",
                ],
            },
            [VARIABLE_NAME]: {
                type: "text",
                defaultValue: "output",
                placeholder: "Enter result variable name",
            },
            [SHOW_ASSISTANT_MESSAGES]: {
                type: "checkbox",
                defaultValue: true,
            },
            [SYSTEM_PROMPT]: {
                type: "format",
                defaultValue: "",
                placeholder: "Enter system prompt",
            },
            [CHUNK_SEPARATOR]: {
                type: "text",
                defaultValue: "",
                placeholder: "Enter chunk separator",
            },
            [CHUNK_JOINER]: {
                type: "text",
                defaultValue: "",
                placeholder: "Enter chunk joiner",
            },
            [SHOULD_MERGE]: {
                type: "checkbox",
                defaultValue: false,
            },
        },
    },
};

async function run(params, settings) {
    const API = params.quickAddApi;

    const input = await API.format(settings[INPUT_FORMAT], params.variables);
    if (!input) {
        new Notice(`No text input. Stopping.`);
        return;
    }

    const variables = Object.assign({}, params.variables);

    const out = await API.ai.chunkedPrompt(
        input,
        settings[PROMPT_TEMPLATE],
        settings[MODEL],
        {
            variableName: settings[VARIABLE_NAME],
            showAssistantMessages: settings[SHOW_ASSISTANT_MESSAGES],
            systemPrompt: await API.format(settings[SYSTEM_PROMPT], params.variables),
            chunkSeparator: new RegExp(settings[CHUNK_SEPARATOR]),
            chunkJoiner: settings[CHUNK_JOINER],
            shouldMerge: settings[SHOULD_MERGE],
        },
        variables
    );

    Object.assign(params.variables, out);
}

You can now create a macro with the script. As an example, you can copy my setup. The steps should be as follows:

1. The script
2. A Capture

The script should have the following settings:

Here are explanations for each of the script settings:

• The input format is what determines what is sent to OpenAI. If it's {{selected}}, it'll use the selected text. You could also hook it into another macro and use the value from some prior step, e.g. the book summarizer or YouTube summarizer from here.
• Prompt Template is as expected. This is the template used for each prompt that's sent to OpenAI. The important part to know here is that the template is used for each chunk, so it's important to have {{value:chunk}}, so QuickAdd can insert the chunk where desired.
• Model is which OpenAI model to use
• Result variable name is the output variable name
• Show assistant messages is whether to display progress notifications
• System prompt is as expected.
• Chunk separator is the regular expression used to separate text into chunks. '\n' is a fine start. It'll separate each new line into a new chunk. You should use the Should merge setting when doing that.
• Chunk joiner is how the chunks are joined back together.
• Should merge will fill up a chunked prompt with smaller chunks until it fits about the size that's acceptable for input & output. This means that, instead of just sending a ton of chunks with only one line, it'll merge them together into one merged chunk with multiple lines.

Using my settings will allow you to select a bunch of text, which it'll send to OpenAI in chunks. The resulting text gets put back together & inserted into the active file.

The Capture settings are rather simple - you just send the output to the active file:

Be aware that your Obsidian might freeze after you've selected the text. It's just chunking - it'll unfreeze momentarily.

Footnotes

1. Name pending. ↩

2. I extracted the text. It can't parse PDFs. ↩

[Gourav1910]

What do you do if you have some text you want to have summarized? Easy, ask ChatGPT to do it.

Now, what if you have a large body of text you want summarized? Do you ask ChatGPT to do so by repeatedly copying over parts of the text? That's tedious, and we don't do that here.

Let me introduce Chunked Prompts1 for QuickAdd.

This will chunk your text into pieces that fit into the input window for the various models. The feature inserts these chunks into your predefined chunk and makes requests to the OpenAI API for each of them. It's even able to merge multiple chunks together if there's space.

This is incredibly useful for summarizing or transforming large bodies of text. Here are some of the things I've done with it:

• Summarized my book notes that couldn't fit in ChatGPT
• Made flashcards from 200-page PDF contents2

I'm kind of on the fence about whether it should be a feature for QuickAdd, so I've made a script that does the same to get feedback. It's a very specific feature, whereas I like the plugin having more primitive features that can be used as building blocks.

Setup

First, you'll need the associated script. Here's how you can use scripts in macros (video instructions):

You include the script by creating it as a file in your vault, e.g. chunked-prompt.js and then pasting the script into the file. Then it'll appear in the scripts dropdown as an option you can add.

Find the script below:

const INPUT_FORMAT = "Input format";
const PROMPT_TEMPLATE = "Prompt template";
const MODEL = "Model";
const VARIABLE_NAME = "Result variable name";
const SHOW_ASSISTANT_MESSAGES = "Show assistant messages";
const SYSTEM_PROMPT = "System prompt";
const CHUNK_SEPARATOR = "Chunk separator";
const CHUNK_JOINER = "Chunk joiner";
const SHOULD_MERGE = "Should merge";

module.exports = {
    entry: run,
    settings: {
        name: `Chunked Prompt`,
        author: `Christian Bager Bach Houmann`,
        options: {
            [INPUT_FORMAT]: {
                type: "format",
                defaultValue: "{{selected}}",
                placeholder: "Enter your input format",
            },
            [PROMPT_TEMPLATE]: {
                type: "format",
                defaultValue: "{{value:chunk}}",
                placeholder: "Enter your prompt template",
            },
            [MODEL]: {
                type: "dropdown",
                defaultValue: "gpt-3.5-turbo",
                options: [
                    "gpt-3.5-turbo",
                    "gpt-3.5-turbo-16k",
                    "gpt-4",
                    "gpt-4-32k",
                    "text-davinci-003",
                ],
            },
            [VARIABLE_NAME]: {
                type: "text",
                defaultValue: "output",
                placeholder: "Enter result variable name",
            },
            [SHOW_ASSISTANT_MESSAGES]: {
                type: "checkbox",
                defaultValue: true,
            },
            [SYSTEM_PROMPT]: {
                type: "format",
                defaultValue: "",
                placeholder: "Enter system prompt",
            },
            [CHUNK_SEPARATOR]: {
                type: "text",
                defaultValue: "",
                placeholder: "Enter chunk separator",
            },
            [CHUNK_JOINER]: {
                type: "text",
                defaultValue: "",
                placeholder: "Enter chunk joiner",
            },
            [SHOULD_MERGE]: {
                type: "checkbox",
                defaultValue: false,
            },
        },
    },
};

async function run(params, settings) {
    const API = params.quickAddApi;

    const input = await API.format(settings[INPUT_FORMAT], params.variables);
    if (!input) {
        new Notice(`No text input. Stopping.`);
        return;
    }

    const out = await API.ai.chunkedPrompt(
        input,
        settings[PROMPT_TEMPLATE],
        settings[MODEL],
        {
            variableName: settings[VARIABLE_NAME],
            showAssistantMessages: settings[SHOW_ASSISTANT_MESSAGES],
            systemPrompt: await API.format(settings[SYSTEM_PROMPT], params.variables),
            chunkSeparator: new RegExp(settings[CHUNK_SEPARATOR]),
            chunkJoiner: settings[CHUNK_JOINER],
            shouldMerge: settings[SHOULD_MERGE],
        }
    );

    Object.assign(params.variables, out);
}

You can now create a macro with the script. As an example, you can copy my setup. The steps should be as follows:

1. The script
2. A Capture

The script should have the following settings:

Here are explanations for each of the script settings:

• The input format is what determines what is sent to OpenAI. If it's {{selected}}, it'll use the selected text. You could also hook it into another macro and use the value from some prior step, e.g. the book summarizer or YouTube summarizer from here.
• Prompt Template is as expected. This is the template used for each prompt that's sent to OpenAI. The important part to know here is that the template is used for each chunk, so it's important to have {{value:chunk}}, so QuickAdd can insert the chunk where desired.
• Model is which OpenAI model to use
• Result variable name is the output variable name
• Show assistant messages is whether to display progress notifications
• System prompt is as expected.
• Chunk separator is the regular expression used to separate text into chunks. '\n' is a fine start. It'll separate each new line into a new chunk. You should use the Should merge setting when doing that.
• Chunk joiner is how the chunks are joined back together.
• Should merge will fill up a chunked prompt with smaller chunks until it fits about the size that's acceptable for input & output. This means that, instead of just sending a ton of chunks with only one line, it'll merge them together into one merged chunk with multiple lines.

Using my settings will allow you to select a bunch of text, which it'll send to OpenAI in chunks. The resulting text gets put back together & inserted into the active file.

The Capture settings are rather simple - you just send the output to the active file:

Be aware that your Obsidian might freeze after you've selected the text. It's just chunking - it'll unfreeze momentarily.

Footnotes

1. Name pending. ↩
2. I extracted the text. It can't parse PDFs. ↩

Hi does it require openai api key? because i don't have credit card to create a paid account in openai.

[chhoumann]

You do need an OpenAI account, yes. This is a paid workflow.

Hoping NLP & LLM research enables local inference - running the model on your computer - relatively soon.

[Redster1]

This is great! I got it working! Thanks for your Youtube video on QuickAdd. I was having an odd behavior with macro not being recognized, but you dealt with the same issue when you were running a user script, so I was able to solve it.

Not seeing a video of house this is used, is the expected behavior when you run the script that you will paste it into "chunk?" I thought it was going to work so that it would simply chunk the "selected" text, but a modal appears titled chunk and I paste the selected text in there.

Either way, this is exciting. Thanks for making this.

[schartkoff]

Hi Christian, I'm excited to try to get this working! One question: following the directions above, I get a "chunk" modal that looks like it's waiting for input while the "Assistant is chunking. Creating prompt chunks with text and prompt template" is also showing. What am I supposed to do here? And is there a way to not have this pop up?

[chhoumann]

Hey! This is because you need to use the {{value:chunk}} syntax in the chunk format settings.
Like so:

[chhoumann]

Released new update that allows variables to be given & used, e.g. like title and author here:

[Triquetra]

@chhoumann I'm recently running into some sort of limitation using this with gpt-4-1106-preview. The model returns a truncated answer. Switching to gpt-3.5-turbo-1106 returns a complete response using the same prompt and input. Based on the processing notification, it seemed that the chunked prompt script sent the lengthy input in 2 batches to gpt-3.5-turbo-1106, but only a single large prompt with gpt-4-1106-preview.

[chhoumann]

Hi @Triquetra, is the truncation visible by sentences cutting off or because you expected more in the output? Trying to diagnose. Haven't seen any issues myself.

The amount of chunks being sent depends on the context size of the models. Given that gpt-4-1106-preview has a much larger context size than 3.5-turbo, it's able to fit the text in a single chunk.

[Triquetra]

A sentence was cut off, but also the task (correcting a provided text) was incomplete with gpt-4-1106 missing a large portion of the end of the provided text. I'll try to test more this week if I have time.