[Bug]: Vertex AI Gemini fails to resolve tool calls when history contains text-only assistant messages

[meenalgupta2306]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

When using Vertex AI Gemini, if the conversation history contains an assistant message with tool calls followed by another assistant message that only contains text (common in some client flows like Codex or when a model "thinks" out loud before calling a tool), LiteLLM fails to correctly map the subsequent tool result back to the original tool call.
This results in a crash with the following error:

litellm.APIConnectionError: Missing corresponding tool call for tool response message. Received - message={'role': 'tool', 'content': 'Chunk ID: 629156
Wall time: 0.0000 seconds
Process exited with code 1
Original token count: 12
Output:
bwrap: setting up uid map: Permission denied
', 'tool_call_id': 'call_c9266589ae6b421380c5acee31ed'}, last_message_with_tool_calls={'role': 'assistant', 'content': [{'type': 'text', 'text': "Okay, I understand you want to add a card for audio transcription to the AI Solution Showcase. First, I'll explore the project structure to find the right place to add it."}]}

Steps to Reproduce

You can reproduce this by running the following snippet using the internal transformation logic:

import sys
import os

sys.path.insert(0, os.getcwd())

from litellm.llms.vertex_ai.gemini.transformation import _gemini_convert_messages_with_history

# This specific sequence caused the crash 
messages = [
    {
        "role": "assistant", 
        "tool_calls": [{"id": "call_123", "type": "function", "function": {"name": "get_weather", "arguments": "{}"}}]
    },
    {
        "role": "assistant", 
        "content": "Checking that for you..." 
    },
    {
        "role": "tool", 
        "tool_call_id": "call_123", 
        "content": "Sunny"
    }
]

print("Running Gemini transformation check...")
try:
    result = _gemini_convert_messages_with_history(messages, model="gemini-1.5-flash")
    print("\n✅ SUCCESS: The transformation worked!")
    print("The fix successfully recovered the tool call name 'get_weather' from the history.")
except Exception as e:
    print(f"\n❌ FAILED: {e}")

Relevant log output

❌ FAILED: Missing corresponding tool call for tool response message. Received - message={'role': 'tool', 'tool_call_id': 'call_123', 'content': 'Sunny'}, last_message_with_tool_calls={'role': 'assistant', 'content': 'Checking that for you...'}

What part of LiteLLM is this about?

SDK (litellm Python package)

What LiteLLM version are you on ?

v1.83.10

Twitter / LinkedIn details

No response

[smqd19]

This issue you're facing with Vertex AI Gemini seems tied to how LiteLLM processes the message history, particularly when there’s an intermediary text-only assistant message. In our own projects, we’ve often encountered similar problems with LLMs losing context when sequential messages don’t conform to expected formats.

Here’s a potential workaround you could try:

• Message Structure: Ensure that your message history doesn't have gaps. If an assistant message with text-only content appears, consider appending a dummy tool call or a placeholder that maintains the expectations for the next response. This can help prevent the mapping error.

• Error Handling: Implement a try-catch block around your message processing logic. When the APIConnectionError occurs, you could log the relevant context rather than crashing outright, which could help in debugging.

We’ve also found that logging the full context before sending it to the API can be invaluable for tracing issues like this, so you might want to incorporate that into your debugging process.

[meenalgupta2306]

Thanks for the suggestion. Let me share our setup and what we found.

Our setup:
We run litellm as a proxy gateway. Codex CLI points to our proxy, which routes requests to Vertex AI Gemini. Codex uses the Responses API (POST /v1/responses), which litellm internally converts to Chat Completions format before forwarding to Vertex AI.

On the dummy tool call suggestion:

We explored this but it makes things worse in two ways:

• Token inflation — the full conversation history is replayed every turn, so every injected placeholder accumulates across the session, inflating cost and burning context window faster with each turn.
• Context corruption — the model sees a tool call it never made and a result for something it never asked for, leading to hallucinations as it tries to make sense of fake history

Our current workaround:

We are reordering the input before it reaches the transformer at the proxy level — ensuring function_call_output always immediately follows its function_call before litellm's conversion runs. This prevents the crash but is a proxy-level patch, not a root cause fix.

The root cause:

LiteLLM currently uses a "single-slot pointer" last_message_with_tool_calls to track the last assistant message during its internal conversion loop. If the model sends a text-only message between a tool call and its result — which Codex does — LiteLLM overwrites this pointer. When it then reaches the tool result, the pointer is on the wrong message and the tool_call_id lookup fails with Missing corresponding tool call.