anthropic: input_json_delta tokens should not be passed to handleLLMNewToken

[danielleontiev]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain.js documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain.js rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

Introduced here: #6179 (comment), the Anthropic chat model with the tools would pass input_json_delta tokens to handleLLMNewToken resulting in the unwanted tokens being propagated to the callbacks.

import { ChatAnthropic } from "@langchain/anthropic";
import { HumanMessage } from "@langchain/core/messages";
import { tool } from "@langchain/core/tools";
import { z } from "zod";

const weatherTool = tool(({ location }) => "The weather in " + location + " is sunny", {
  name: "get_current_weather",
  description: "Get the current weather for a location",
  schema: z.object({
    location: z.string(),
  }),
});

const newTokenCallback = { handleLLMNewToken(token: string) { console.log("New token: " + token) } };

const chat = new ChatAnthropic({ model: "claude-3-5-sonnet-20240620", streaming: true });
const chatWithTools = chat.bindTools([weatherTool]);

const messages = [
  new HumanMessage("What's the weather in Paris?"),
];

console.log(await chatWithTools.invoke(messages, { callbacks: [newTokenCallback] }));

Output

New token: Certainly
New token: !
New token:  I
New token:  can help
New token:  you fin
New token: d out
New token:  the current weather in
New token:  Paris.
New token:  To
New token:  get this
New token:  information, I'll
New token:  use
New token:  the
New token:  get
New token: _current_weather function
New token: .
New token:  Let
New token:  me fetch
New token:  that
New token:  data
New token:  for
New token:  you.
New token: {"location":
New token:  "P
New token: aris"}
AIMessageChunk {
  "id": "msg_01DdYShYboNJmRuUboYtFkJo",
  "content": [
    {
      "index": 0,
      "type": "text",
      "text": "Certainly! I can help you find out the current weather in Paris. To get this information, I'll use the get_current_weathe
r function. Let me fetch that data for you."
    },
    {
      "index": 1,
      "type": "tool_use",
      "id": "toolu_013m574E44V7kBZtyEzWFgFv",
      "name": "get_current_weather",
      "input": "{\"location\": \"Paris\"}"
    }
  ],
  "additional_kwargs": {
    "id": "msg_01DdYShYboNJmRuUboYtFkJo",
    "type": "message",
    "role": "assistant",
    "model": "claude-3-5-sonnet-20240620",
    "stop_reason": "tool_use",
    "stop_sequence": null
  },
  "response_metadata": {},
  "tool_calls": [
    {
      "name": "get_current_weather",
      "args": {
        "location": "Paris"
      },
      "id": "toolu_013m574E44V7kBZtyEzWFgFv",
      "type": "tool_call"
    }
  ],
  "tool_call_chunks": [
    {
      "args": "{\"location\": \"Paris\"}",
      "id": "toolu_013m574E44V7kBZtyEzWFgFv",
      "name": "get_current_weather",
      "index": 1,
      "type": "tool_call_chunk"
    }
  ],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 392,
    "output_tokens": 95,
    "total_tokens": 487
  }
}

The last three tokens are wrong because they are part of Anthropic's input_json_delta:

New token: {"location":
New token:  "P
New token: aris"}

For example, OpenAI behaves differently

import { ChatOpenAI } from "@langchain/openai";

const chat = new ChatOpenAI({ model: "gpt-4o", streaming: true });

Output

New token: 
New token: 
New token: 
New token: 
New token: 
New token: 
New token: 
AIMessageChunk {
  "id": "chatcmpl-AEQLZY0XohZRecOzovcUdPxDQUNZg",
  "content": "",
  "additional_kwargs": {
    "tool_calls": [
      {
        "index": 0,
        "id": "call_vHoeJADcXYCVHtbo0XkAU2cB",
        "type": "function",
        "function": "[Object]"
      }
    ]
  },
  "response_metadata": {
    "estimatedTokenUsage": {
      "promptTokens": 13,
      "completionTokens": 0,
      "totalTokens": 13
    },
    "prompt": 0,
    "completion": 0,
    "finish_reason": "tool_calls",
    "system_fingerprint": "fp_e5e4913e83",
    "usage": {
      "prompt_tokens": 52,
      "completion_tokens": 15,
      "total_tokens": 67,
      "prompt_tokens_details": {
        "cached_tokens": 0
      },
      "completion_tokens_details": {
        "reasoning_tokens": 0
      }
    }
  },
  "tool_calls": [
    {
      "name": "get_current_weather",
      "args": {
        "location": "Paris"
      },
      "id": "call_vHoeJADcXYCVHtbo0XkAU2cB",
      "type": "tool_call"
    }
  ],
  "tool_call_chunks": [
    {
      "name": "get_current_weather",
      "args": "{\"location\":\"Paris\"}",
      "id": "call_vHoeJADcXYCVHtbo0XkAU2cB",
      "index": 0,
      "type": "tool_call_chunk"
    }
  ],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 52,
    "output_tokens": 15,
    "total_tokens": 67
  }
}

Interestingly enough, the behavior is not reproduced if using .streamEvents on the chat model, because arguments tokens are returned as part of input_json_delta objects, but is reproduced when using .streamEvents from the LangGraph agent, because content is returned as simple string in that case.

const chat = new ChatAnthropic({ model: "claude-3-5-sonnet-20240620", streaming: true });
const chatWithTools = chat.bindTools([weatherTool]);

const messages = [
  new HumanMessage("What's the weather in Paris?"),
];

for await (const response of chatWithTools.streamEvents(messages, { version: "v2" })) {
  if (response.event === "on_chat_model_stream") {
    console.log(response.data.chunk?.content);
  }
}

Output (truncated)

[ { index: 0, type: 'text_delta', text: ' to' } ]
[ { index: 0, type: 'text_delta', text: ' fetch' } ]
[ { index: 0, type: 'text_delta', text: ' the' } ]
[ { index: 0, type: 'text_delta', text: ' current' } ]
[ { index: 0, type: 'text_delta', text: ' weather data.' } ]
[ { index: 0, type: 'text_delta', text: ' Let' } ]
[ { index: 0, type: 'text_delta', text: ' me ' } ]
[ { index: 0, type: 'text_delta', text: 'do that for' } ]
[ { index: 0, type: 'text_delta', text: ' you right' } ]
[ { index: 0, type: 'text_delta', text: ' away' } ]
[ { index: 0, type: 'text_delta', text: '.' } ]
[ { index: 1, type: 'tool_use', id: 'toolu_01Ej6eX3dAskhZa3Nq2B6Ro4', name: 'get_current_weather', input: '' } ]
[ { index: 1, input: '', type: 'input_json_delta' } ]
[ { index: 1, input: '{"lo', type: 'input_json_delta' } ]
[ { index: 1, input: 'cation": "', type: 'input_json_delta' } ]
[ { index: 1, input: 'Paris"}', type: 'input_json_delta' } ]
[]

import { createReactAgent } from "@langchain/langgraph/prebuilt";

const chat = new ChatAnthropic({ model: "claude-3-5-sonnet-20240620", streaming: true });

const messages = [
  new HumanMessage("What's the weather in Paris?"),
];

const agent = createReactAgent({
  llm: chat,
  tools: [weatherTool],
});

for await (const response of agent.streamEvents({ messages }, { version: "v2" })) {
  if (response.event === "on_chat_model_stream") {
    console.log(response.data.chunk?.content);
  }
}

Output (truncated)

 Let
 me 
do that for you right
 away
.
{"loc
ation": 
"Paris"}

I am not sure about the streaming behavior, but at least the callback behavior could be fixed by excluding input tokens here:

langchainjs/libs/langchain-anthropic/src/chat_models.ts

Lines 167 to 174 in 660af3e

	} else if (
	Array.isArray(chunk.content) &&
	chunk.content.length >= 1 &&
	"input" in chunk.content[0]
	) {
	return typeof chunk.content[0].input === "string"
	? chunk.content[0].input
	: JSON.stringify(chunk.content[0].input);

Error Message and Stack Trace (if applicable)

No response

Description

"Argumnets" tokens should not appear in the callbacks and when streaming with LangGraph

System Info

npm info langchain

langchain@0.3.2 | MIT | deps: 12 | versions: 297
Typescript bindings for langchain
https://github.com/langchain-ai/langchainjs/tree/main/langchain/

keywords: llm, ai, gpt3, chain, prompt, prompt engineering, chatgpt, machine learning, ml, openai, embeddings, vectorstores

dist
.tarball: https://registry.npmjs.org/langchain/-/langchain-0.3.2.tgz
.shasum: aec3e679d3d6c36f469448380affa475c92fbd86
.integrity: sha512-kd2kz1cS/PIVrLEDFlrZsAasQfPLbY1UqCZbRKa3/QcpB33/n6xPDvXSMfBuKhvNj0bjW6MXDR9HZTduXjJBgg==
.unpackedSize: 2.9 MB

dependencies:
@langchain/openai: >=0.1.0 <0.4.0        jsonpointer: ^5.0.1                      uuid: ^10.0.0                            
@langchain/textsplitters: >=0.0.0 <0.2.0 langsmith: ^0.1.56-rc.1                  yaml: ^2.2.1                             
js-tiktoken: ^1.0.12                     openapi-types: ^12.1.3                   zod-to-json-schema: ^3.22.3              
js-yaml: ^4.1.0                          p-retry: 4                               zod: ^3.22.4                             

maintainers:
- nfcampos <nuno@boringbits.io>
- jacoblee93 <jacoblee93@gmail.com>
- andrewnguonly <andrewnguonly@gmail.com>
- davidduong <david@duong.cz>
- hwchase17 <hw.chase.17@gmail.com>
- basproul <braceasproul@gmail.com>

dist-tags:
latest: 0.3.2                              next: 0.3.2-rc.0                           tag-for-publishing-older-releases: 0.2.20  

published 2 weeks ago by jacoblee93 <jacoblee93@gmail.com>

• MacOS
• node v22.9.0
• npm 10.8.3

[dosubot]

Hey, @danielleontiev! I'm here to assist you with any bugs, questions, or contributions you have. Let's work together to solve this issue!

To resolve the issue of input_json_delta tokens being incorrectly passed to the handleLLMNewToken callback in the LangChain.js library when using the Anthropic chat model, you can modify the _formatContent function. Specifically, ensure that input_json_delta types are converted to tool_use types, as input_json_delta is not a valid type for Anthropic messages. This conversion is already present in the code:

if (contentPartCopy.type === "input_json_delta") {
  // `input_json_delta` type only represents yielding partial tool inputs
  // and is not a valid type for Anthropic messages.
  contentPartCopy.type = "tool_use";
}

This code snippet ensures that any input_json_delta type is converted to tool_use, which is a valid type for Anthropic messages, thus resolving the issue ^[1].

To continue talking to Dosu, mention @dosu.

[jacoblee93]

CC @bracesproul I think this is unintended

[dosubot]

Hi, @danielleontiev. I'm Dosu, and I'm helping the LangChain JS team manage their backlog. I'm marking this issue as stale.

Issue Summary

• The issue involves a bug where input_json_delta tokens are incorrectly passed to the handleLLMNewToken callback with the Anthropic chat model.
• A suggestion was made to modify the _formatContent function to convert input_json_delta types to tool_use types.
• This suggestion was met with negative feedback from you and other users.
• @jacoblee93 tagged @bracesproul, suggesting the issue might be unintended.

Next Steps

• Please let me know if this issue is still relevant to the latest version of the LangChain JS repository. If so, you can keep the discussion open by commenting on the issue.
• Otherwise, the issue will be automatically closed in 7 days.

Thank you for your understanding and contribution!

[jacoblee93]

Keeping this open