Skip to content

Commit 1ba5066

Browse files
Split prompt cache token test by provider (#27)
The test was stubbing an OpenAI response with `cache_write_tokens`, but ruby_llm's OpenAI provider only extracts that field as of 1.15.0. On 1.8.0-1.14.1 it hardcodes `cache_creation_tokens` to 0 causing the test to fail. Split this test into two tests: - OpenAI: asserts only `cache_read.input_tokens` (the field every supported provider version exposes) - Anthropic: asserts both `cache_read` and `cache_creation` since Anthropic has surfaced both since the accessors were added Both tests skip on ruby_llm < 1.9.0 where the `cached_tokens` / `cache_creation_tokens` Message accessors don't exist yet.
1 parent 8963700 commit 1ba5066

1 file changed

Lines changed: 41 additions & 5 deletions

File tree

test/instrumentation_test.rb

Lines changed: 41 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ def setup
66

77
RubyLLM.configure do |c|
88
c.openai_api_key = "fake-key-for-testing"
9+
c.anthropic_api_key = "fake-key-for-testing"
910
end
1011
end
1112

@@ -92,10 +93,12 @@ def test_marks_streaming_chat_requests
9293
assert_equal true, span.attributes["gen_ai.request.stream"]
9394
end
9495

95-
def test_records_prompt_cache_tokens
96-
# RubyLLM's OpenAI provider maps `cached_tokens` ← `cache_read_tokens(usage)`
97-
# and `cache_creation_tokens` ← `cache_write_tokens(usage)`, both surfaced
98-
# on `Message#cached_tokens` / `Message#cache_creation_tokens`.
96+
def test_records_openai_prompt_cache_read_tokens
97+
# OpenAI exposes only `cached_tokens` via `prompt_tokens_details.cached_tokens`.
98+
# Its provider in ruby_llm does not surface a `cache_creation_tokens` value
99+
# until 1.15.0 (we only assert the cache-read attribute here).
100+
# The accessor itself was added in ruby_llm 1.9.0.
101+
skip "cached_tokens accessor not available before ruby_llm 1.9.0" unless RubyLLM::Message.instance_methods.include?(:cached_tokens)
99102
stub_request(:post, "https://api.openai.com/v1/chat/completions")
100103
.to_return(
101104
status: 200,
@@ -108,14 +111,47 @@ def test_records_prompt_cache_tokens
108111
prompt_tokens: 100,
109112
completion_tokens: 5,
110113
total_tokens: 105,
111-
prompt_tokens_details: { cached_tokens: 75, cache_write_tokens: 20 }
114+
prompt_tokens_details: { cached_tokens: 75 }
112115
}
113116
}.to_json
114117
)
115118

116119
chat = RubyLLM.chat(model: "gpt-4o-mini")
117120
chat.ask("Hi")
118121

122+
span = EXPORTER.finished_spans.first
123+
assert_equal 75, span.attributes["gen_ai.usage.cache_read.input_tokens"]
124+
assert_equal 0, span.attributes["gen_ai.usage.cache_creation.input_tokens"]
125+
end
126+
127+
def test_records_anthropic_prompt_cache_tokens
128+
# Anthropic's provider surfaces both `cached_tokens` (via
129+
# `cache_read_input_tokens`) and `cache_creation_tokens` (via
130+
# `cache_creation_input_tokens`). Accessors were added in ruby_llm 1.9.0.
131+
skip "cache token accessors not available before ruby_llm 1.9.0" unless RubyLLM::Message.instance_methods.include?(:cache_creation_tokens)
132+
stub_request(:post, "https://api.anthropic.com/v1/messages")
133+
.to_return(
134+
status: 200,
135+
headers: { "Content-Type" => "application/json" },
136+
body: {
137+
id: "msg_cache",
138+
type: "message",
139+
role: "assistant",
140+
model: "claude-3-5-sonnet-20241022",
141+
content: [{ type: "text", text: "Hello!" }],
142+
stop_reason: "end_turn",
143+
usage: {
144+
input_tokens: 100,
145+
output_tokens: 5,
146+
cache_read_input_tokens: 75,
147+
cache_creation_input_tokens: 20
148+
}
149+
}.to_json
150+
)
151+
152+
chat = RubyLLM.chat(model: "claude-3-5-sonnet-20241022")
153+
chat.ask("Hi")
154+
119155
span = EXPORTER.finished_spans.first
120156
assert_equal 75, span.attributes["gen_ai.usage.cache_read.input_tokens"]
121157
assert_equal 20, span.attributes["gen_ai.usage.cache_creation.input_tokens"]

0 commit comments

Comments
 (0)