Skip to content

fix(otlp): avoid panic in grpc retry without tokio runtime#3472

Open
Wassbdr wants to merge 4 commits intoopen-telemetry:mainfrom
Wassbdr:fix/3432-tonic-retry-noasync
Open

fix(otlp): avoid panic in grpc retry without tokio runtime#3472
Wassbdr wants to merge 4 commits intoopen-telemetry:mainfrom
Wassbdr:fix/3432-tonic-retry-noasync

Conversation

@Wassbdr
Copy link
Copy Markdown

@Wassbdr Wassbdr commented Apr 27, 2026

Summary

Fix panic when experimental-grpc-retry is enabled and exports run from a thread without an active Tokio runtime (e.g. thread-based BatchSpanProcessor).

Fixes #3432.

Root cause

The tonic retry wrapper used Tokio runtime delays unconditionally in retry-enabled builds. The thread-based processors run exports on plain threads without a Tokio reactor, so retry backoff triggered panic (there is no reactor running).

Changes

  • Make tonic retry runtime-aware:
    • use Tokio when a Tokio runtime is active
    • fallback to opentelemetry_sdk::runtime::NoAsync otherwise
  • Update tonic traces/logs/metrics exporters to use the updated wrapper signature
  • Add regression test for no-runtime path in tonic retry wrapper
  • Document fix in opentelemetry-otlp/CHANGELOG.md

Validation

Ran locally:

cargo fmt --all -- --check
cargo test -p opentelemetry-otlp --lib test_retry_wrapper_falls_back_without_tokio_runtime --features experimental-grpc-retry
cargo test -p opentelemetry-otlp --lib exporter::tonic::tests --features experimental-grpc-retry

All passed (1/1 + 25/25).

Use runtime-aware tonic retry fallback (Tokio -> NoAsync) and add regression test for no-runtime context.\n\nFixes open-telemetry#3432
@Wassbdr Wassbdr requested a review from a team as a code owner April 27, 2026 11:30
@linux-foundation-easycla
Copy link
Copy Markdown

linux-foundation-easycla Bot commented Apr 27, 2026

CLA Signed
The committers listed above are authorized under a signed CLA.

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 27, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.4%. Comparing base (ef8ff2f) to head (ac1ce14).

Additional details and impacted files
@@           Coverage Diff           @@
##            main   #3472     +/-   ##
=======================================
+ Coverage   83.3%   83.4%   +0.1%     
=======================================
  Files        128     128             
  Lines      25225   25280     +55     
=======================================
+ Hits       21026   21103     +77     
+ Misses      4199    4177     -22     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Wassbdr added 3 commits April 27, 2026 13:52
…case

Introduce new tests for the tonic retry wrapper to ensure correct behavior when the Tokio runtime is available and when the retry feature is disabled. These tests validate the retry logic and the number of attempts made during the operation.
…ion checks

Refactor metric tests to include a retry mechanism for flushing metrics and waiting for the gRPC collector to be reachable. This improves the reliability of the tests by ensuring that metrics are properly emitted and validated under varying conditions.
Refactor the panic message in the force_flush function to improve readability by removing unnecessary line breaks. This change enhances the clarity of error reporting during metric flushing attempts.
@scottgerring
Copy link
Copy Markdown
Member

Per comment on #3432 , let's hold off on this one until we know what we are doing with the Runtime trait, as it may help

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: BatchSpanProcessor panics when experimental-grpc-retry feature is enabled

2 participants