Adds support for memory and prompt passing to a ChatConversationalAgent #374

jacoblee93 · 2023-03-18T07:12:59Z

This PR adds a new method for initializing agent executors, initializeAgentExecutorWithOptions, which takes aTool[] parameter and a BaseLanguageModel like the existing initializeAgentExecutor, but also takes an options object to support additional parameters without breaking existing code.

This new method also adds BufferMemory to the executor when the agent type is set as chat-conversational-react-description.

We'll now be able to pass a prompt into a ChatConversationalAgent and have built in memory like this:

import { initializeAgentExecutorWithOptions } from "langchain/agents";
const model = new OpenAIChat({
  openAIApiKey: process.env.OPENAI_API_KEY,
  modelName: process.env.OPENAI_MODEL_NAME
});
const tools = [new Calculator()];
const executor = await initializeAgentExecutorWithOptions(
  tools,
  model,
  {
    agentType: "chat-conversational-react-description",
    verbose: true,
    prompt: `Answer the following questions as best you can, but speaking as a pirate might speak.`
  }
);

const result0 = await executor.call({input: "How's your day going?"});

// result0.output
// Arr, me day be goin' well, matey! What be ye needin' help with?

const result1 = await executor.call({input: "What is 9 raised to the second power?"});

// result1.output
// Arr, 9 raised to the second power be 81, matey!

const result2 = await executor.call({ input: "What is that result times 2?" });

// result2.output
// Arr, the result of 9 raised to the second power times 2 be 162, matey!

hwchase17 · 2023-03-19T02:04:04Z

the refactor to use Options is done to allow for more optional params in a more elegant way?

If thats the case, this should be the interface we promote, correct? if so, maybe lets update the examples to reflect that

hwchase17 · 2023-03-19T02:05:08Z

langchain/src/agents/chat_convo/index.ts

-      humanMessage = SUFFIX,
-      outputParser = new AgentOutputParser(),
-    } = args ?? {};
+    const systemMessage =


why is the prefix/suffix helpful here? its just same as system/human right? seems confusing to have two with same name

The comment block above that line may be wrong (maybe copy-pasted from elsewhere?) but they implied that args should contain:

* @param args.suffix - String to put after the list of tools. * @param args.prefix - String to put before the list of tools.

I'm a bit new to Langchain's/prompt terminology in general so prefix/suffix seemed clearer but I can update the comment block and change the parameter names to systemMessage and humanMessage instead if that's preferred. Will update it tomorrow!

Now that I've been playing with LangChain for a bit longer, I feel like we should deprecate these systemMessage and humanMessage properties to be in line with the other agents, or deprecate prefix on suffix on the other agents to avoid confusion here. I changed it back to restart a discussion.

jacoblee93 · 2023-03-19T02:14:31Z

the refactor to use Options is done to allow for more optional params in a more elegant way?

If thats the case, this should be the interface we promote, correct? if so, maybe lets update the examples to reflect that

Yes, the existing method had the following signature:

export const initializeAgentExecutor = async (
  tools: Tool[],
  llm: BaseLanguageModel,
  agentType = "zero-shot-react-description",
  verbose = false,
  callbackManager: CallbackManager = getCallbackManager()
): Promise<AgentExecutor>

And tacking on more parameters at the end seemed like the wrong approach. I can update this PR with some docs updates tomorrow!

GautierT · 2023-03-19T20:03:28Z

Hi ! I'm looking for a way to build a chat who can use tools with a memory in a database.
Seems like this PR can help !
But how to handle the memory when multiple user connect to the same API but each user have his "own" memory ?
Maybe we should be able to set the user id has the memoryKey ?

Thanks !

…e arguments to createPrompt method in ChatConversationalAgent

…thod

jacoblee93 · 2023-03-19T22:25:20Z

@hwchase17 I've updated the docs and changed the parameter names as requested, and found out how to get rid of the ugly .memory hack. Turned out the AgentExecutor subclass wasn't passing arguments along to the BaseChain superclass:
https://github.com/hwchase17/langchainjs/pull/374/files#diff-59a2c3ddfc3c7e38be163dd2b5f2a48f76da2b51370b98755cf6a24a2d290c70R42

@GautierT I also updated the initializeAgentExecutorWithOptions method to allow you to pass in a custom memory arg to override the default settings - does that fit your use case?

https://github.com/hwchase17/langchainjs/pull/374/files#diff-7d801aff4f2d252eed87091afa8bb799fa2fde06e1cf8315078c8d3ce86a5e49R52

jacoblee93 · 2023-03-19T22:35:22Z

Ok, merged in main as well. @hwchase17 PTAL when you get some time!

I'm also very open to suggestions if someone has a better name than initializeAgentExecutorWithOptions

jacoblee93 · 2023-03-20T00:55:32Z

The formatting check is passing on my end - is it possible to rerun it? Might be a flake or have run on a previous commit.

jacobrosenthal · 2023-03-20T01:00:02Z

If you look close it says you need to run prettier on

docs:format:check: cache miss, executing 01977ba0ac428b4d
langchain:format:check: src/agents/chat_convo/index.ts

jacoblee93 · 2023-03-20T01:11:31Z

If you look close it says you need to run prettier on
docs:format:check: cache miss, executing 01977ba0ac428b4d
langchain:format:check: src/agents/chat_convo/index.ts

I was trying it with yarn format:check as specified in CONTRIBUTING.md but didn't see any warnings or alerts - could definitely be something wrong with my setup though:

jacobrosenthal · 2023-03-20T01:55:01Z

If I do yarn format:check in your branch which is what that test runs, it shows the error and do a yarn format it does change it

examples/src/agents/chat_convo_with_tracing.ts

agola11

Thanks @jacoblee93! Overall lgtm, just a small comment and need to fix formatting

…feature_custom_prompt_agent

jacoblee93 · 2023-03-20T17:03:55Z

Thanks @jacoblee93! Overall lgtm, just a small comment and need to fix formatting

Ah, yes! Fixed. Also fixed a typo in the Zapier docs page I noticed.

If I do yarn format:check in your branch which is what that test runs, it shows the error and do a yarn format it does change it

Strangely, pushing this unrelated docs fix triggered the CI again and it passed this time without issue 🤷.

agola11 · 2023-03-20T20:57:00Z

Thanks for adding this in @jacoblee93. I spoke with @nfcampos and @hwchase17 and we think it makes the most sense to hold off on merging this in for a few days -- we're actively making some changes to the callback manager which will affect this PR. Also, it would be good to deprecate the existing method to avoid having multiple ways to construct an agent. We can take over from here. Appreciate it!

jacoblee93 · 2023-03-21T00:33:57Z

Ok, sounds good!

nfcampos · 2023-03-27T15:39:03Z

langchain/src/agents/initialize.ts

+  tools: Tool[],
+  llm: BaseLanguageModel,
+  options: {
+    agentType?: string;


Add returnIntermediateSteps

nfcampos · 2023-03-27T15:40:03Z

langchain/src/agents/initialize.ts

+          prefix: options.prompt,
+        }),
+        tools,
+        returnIntermediateSteps: true,


Make the default false

…feature_custom_prompt_agent

…ions options, change prompt option to prefix, remove memory default

vercel · 2023-03-28T02:12:05Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchainjs-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 17, 2023 4:23pm

vercel · 2023-03-28T02:12:07Z

Someone is attempting to deploy a commit to the LangChain Team on Vercel.

A member of the Team first needs to authorize it.

jacoblee93 · 2023-03-28T02:13:30Z

langchain/src/agents/chat_convo/index.ts

  systemMessage?: string;
-  /** String to put before the list of tools. */
+  /** DEPRECATED: String to put before the list of tools. */
  humanMessage?: string;


I opted to deprecate these properties to be in line with the other agents - all prompt args should probably all inherit from the same type anyway, no?

So these have different arg names because they are designed to work with chat models, instead of string LLMs as the other agents. Hence the different arg names. Thoughts?

Ah ok understood, thanks for the context. What would you think of subclassing this new type of agent into a ChatAgent? We could also make memoryKey configurable maybe to remove the hidden chat_history dependency?

I'll revert it to the way it was, we could also scope that/some other solution to a future PR.

Added a bit more discussion here: #374 (comment)

jacoblee93 · 2023-03-28T02:16:25Z

langchain/src/agents/initialize.ts

+        returnIntermediateSteps,
+        verbose,
+        callbackManager,
+        memory: options.memory,


@nfcampos I removed the memory default here to avoid a special case, but don't love the fact that we need to initialize memory with specific options like memoryKey: "chat_history". Would it make sense to make a new wrapper memory class for use with this agent with defaults that work with it?

Or even just change the default memory key in BufferMemory to chat_history?

This agent is "special" in that it actually is designed to work with memory, the other ones aren't. I agree the chat_history hidden requirement is bad, but we do need to default here to creating a memory

Gotcha. I'll add it back in - I'll throw an error if a user attempts to pass memory into the initializeAgentExecutorWithOptions method for other agent types in that case.

I also changed the options input syntax for initializeAgentExecutorWithOptions to take a promptArgs option that will just get passed through directly to fromLLMAndTools, so we don't need to worry about conflating systemMessage and prefix - the user can look at the specific agent they're using and figure out which one is appropriate.

Longer term, subclassing at least the prompt args could make sense to me so we could do something like:

promptArgs?: AgentCreatePromptArgs | ChatAgentCreatePromptArgs;

ChatAgentCreatePromptArgs could also take a memory key too and decouple the hidden reliance on a memoryKey set to chat_history.

…ent types

…feature_custom_prompt_agent

… accepted by the agent specfied

nfcampos · 2023-04-17T16:11:09Z

@jacoblee93 I've updated this to reflect latest changes in main, this will be merged now. Thanks a lot for this, this makes things a lot better

jacoblee93 · 2023-04-17T16:16:33Z

Sweet let's go! Thanks for the review + fixups @nfcampos

…nt (langchain-ai#374) * Use OpenAIChat class to initialize gpt-4 models * Formatting * Adds support for passing a prompt to a ChatConversationalAgent * Formatting * Properly set memory when passed into AgentExecutor constructor, rename arguments to createPrompt method in ChatConversationalAgent * Formatting * Update docs and tests to use new intializeAgentExecutorWithOptions method * Formatting * Update docs * Add returnIntermediateSteps, suffix to initializeAgentExecutorWithOptions options, change prompt option to prefix, remove memory default * Remove test timeout * Revert renames, throw error if memory is passed in for unsupported agent types * Fix import * Do some tsc wizardry to make the initialise function only accept args accepted by the agent specfied * Lint * Update examples added since * Fix int tests --------- Co-authored-by: Jacob Lee <jacob@autocode.com> Co-authored-by: Nuno Campos <nuno@boringbits.io>

Jacob Lee added 4 commits March 17, 2023 13:28

Use OpenAIChat class to initialize gpt-4 models

56b3677

Formatting

c5fc36f

Adds support for passing a prompt to a ChatConversationalAgent

78df0ca

Formatting

a82b75c

jacoblee93 changed the title ~~Adds support for passing a prompt to a ChatConversationalAgent~~ Adds support for memory and passing a prompt to a ChatConversationalAgent Mar 18, 2023

jacoblee93 changed the title ~~Adds support for memory and passing a prompt to a ChatConversationalAgent~~ Adds support for memory and prompt passing to a ChatConversationalAgent Mar 18, 2023

hwchase17 reviewed Mar 19, 2023

View reviewed changes

Jacob Lee added 3 commits March 19, 2023 15:01

Properly set memory when passed into AgentExecutor constructor, renam…

bde7350

…e arguments to createPrompt method in ChatConversationalAgent

Formatting

d198eb8

Update docs and tests to use new intializeAgentExecutorWithOptions me…

403c22e

…thod

Jacob Lee added 2 commits March 19, 2023 15:30

Merge in main

ec606d2

Formatting

f03cf7b

agola11 reviewed Mar 20, 2023

View reviewed changes

examples/src/agents/chat_convo_with_tracing.ts Outdated Show resolved Hide resolved

agola11 reviewed Mar 20, 2023

View reviewed changes

Jacob Lee added 2 commits March 20, 2023 09:43

Merge branch 'main' of https://github.com/hwchase17/langchainjs into …

9d7f891

…feature_custom_prompt_agent

Update docs

ccaa4ef

nfcampos reviewed Mar 27, 2023

View reviewed changes

Jacob Lee added 2 commits March 27, 2023 17:59

Merge branch 'main' of https://github.com/hwchase17/langchainjs into …

607f9ef

…feature_custom_prompt_agent

Add returnIntermediateSteps, suffix to initializeAgentExecutorWithOpt…

9352f69

…ions options, change prompt option to prefix, remove memory default

jacoblee93 commented Mar 28, 2023

View reviewed changes

vercel bot deployed to Preview – langchainjs-docs March 28, 2023 02:14 View deployment

jacoblee93 commented Mar 28, 2023

View reviewed changes

Remove test timeout

07c1306

vercel bot had a problem deploying to Preview – langchainjs-docs March 28, 2023 02:18 Failure

Jacob Lee added 2 commits March 28, 2023 14:29

Revert renames, throw error if memory is passed in for unsupported ag…

3d0fbc9

…ent types

Merge branch 'main' of https://github.com/hwchase17/langchainjs into …

622ea77

…feature_custom_prompt_agent

vercel bot deployed to Preview – langchainjs-docs March 28, 2023 21:32 View deployment

nfcampos added 2 commits April 17, 2023 16:21

Merge branch 'main' into pr/jacoblee93/374

a092fd2

Fix import

6b68c5b

vercel bot deployed to Preview April 17, 2023 15:28 View deployment

Do some tsc wizardry to make the initialise function only accept args…

db79b9a

… accepted by the agent specfied

vercel bot deployed to Preview April 17, 2023 16:04 View deployment

nfcampos added 3 commits April 17, 2023 17:06

Merge branch 'main' into pr/jacoblee93/374

2ac4f4c

Lint

800e517

Update examples added since

27f376d

nfcampos mentioned this pull request Apr 17, 2023

Allow prompt customization for initializeAgentExecutor #195

Closed

vercel bot deployed to Preview April 17, 2023 16:15 View deployment

Fix int tests

ac7ae1f

vercel bot deployed to Preview April 17, 2023 16:23 View deployment

nfcampos merged commit 14f7719 into langchain-ai:main Apr 17, 2023

dmechea mentioned this pull request Apr 19, 2023

Init prompt not being used by agent in initializeAgentExecutorWithOptions #891

Closed

jacoblee93 added a commit that referenced this pull request Oct 9, 2024

Handle explicit undefined values passed into serde (#374)

5f8a645

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds support for memory and prompt passing to a ChatConversationalAgent #374

Adds support for memory and prompt passing to a ChatConversationalAgent #374

jacoblee93 commented Mar 18, 2023 •

edited

Loading

hwchase17 commented Mar 19, 2023

hwchase17 Mar 19, 2023

jacoblee93 Mar 19, 2023 •

edited

Loading

jacoblee93 Mar 28, 2023 •

edited

Loading

jacoblee93 commented Mar 19, 2023

GautierT commented Mar 19, 2023

jacoblee93 commented Mar 19, 2023 •

edited

Loading

jacoblee93 commented Mar 19, 2023 •

edited

Loading

jacoblee93 commented Mar 20, 2023

jacobrosenthal commented Mar 20, 2023

jacoblee93 commented Mar 20, 2023

jacobrosenthal commented Mar 20, 2023

agola11 left a comment

jacoblee93 commented Mar 20, 2023

agola11 commented Mar 20, 2023

jacoblee93 commented Mar 21, 2023

nfcampos Mar 27, 2023

nfcampos Mar 27, 2023

vercel bot commented Mar 28, 2023 •

edited

Loading

vercel bot commented Mar 28, 2023

jacoblee93 Mar 28, 2023

nfcampos Mar 28, 2023

jacoblee93 Mar 28, 2023 •

edited

Loading

jacoblee93 Mar 28, 2023

jacoblee93 Mar 28, 2023 •

edited

Loading

nfcampos Mar 28, 2023

jacoblee93 Mar 28, 2023

jacoblee93 Mar 28, 2023

nfcampos commented Apr 17, 2023

jacoblee93 commented Apr 17, 2023

Adds support for memory and prompt passing to a ChatConversationalAgent #374

Adds support for memory and prompt passing to a ChatConversationalAgent #374

Conversation

jacoblee93 commented Mar 18, 2023 • edited Loading

hwchase17 commented Mar 19, 2023

Choose a reason for hiding this comment

jacoblee93 Mar 19, 2023 • edited Loading

Choose a reason for hiding this comment

jacoblee93 Mar 28, 2023 • edited Loading

Choose a reason for hiding this comment

jacoblee93 commented Mar 19, 2023

GautierT commented Mar 19, 2023

jacoblee93 commented Mar 19, 2023 • edited Loading

jacoblee93 commented Mar 19, 2023 • edited Loading

jacoblee93 commented Mar 20, 2023

jacobrosenthal commented Mar 20, 2023

jacoblee93 commented Mar 20, 2023

jacobrosenthal commented Mar 20, 2023

agola11 left a comment

Choose a reason for hiding this comment

jacoblee93 commented Mar 20, 2023

agola11 commented Mar 20, 2023

jacoblee93 commented Mar 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vercel bot commented Mar 28, 2023 • edited Loading

vercel bot commented Mar 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jacoblee93 Mar 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jacoblee93 Mar 28, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nfcampos commented Apr 17, 2023

jacoblee93 commented Apr 17, 2023

jacoblee93 commented Mar 18, 2023 •

edited

Loading

jacoblee93 Mar 19, 2023 •

edited

Loading

jacoblee93 Mar 28, 2023 •

edited

Loading

jacoblee93 commented Mar 19, 2023 •

edited

Loading

jacoblee93 commented Mar 19, 2023 •

edited

Loading

vercel bot commented Mar 28, 2023 •

edited

Loading

jacoblee93 Mar 28, 2023 •

edited

Loading

jacoblee93 Mar 28, 2023 •

edited

Loading