perf(objstore): paginate S3 scan and share a cached scanner client by ksaurabhAparavi · Pull Request #1209 · rocketride-org/rocketride-server

ksaurabhAparavi · 2026-06-10T13:30:11Z

Summary

Replace the StartAfter + page result-equality dedup loop with proper ListObjectsV2 continuation-token pagination.
Cache and share one S3 client across the scanner threads (mutex-guarded, reset on list errors so the next scan re-connects) and raise ClientConfiguration.maxConnections to 64 so the shared pool does not serialize threads.
Drop the per-object Content-Type HEAD request from processEntry; owner metadata is fetched inline via ListObjectsV2 FetchOwner, removing a round-trip per object.

The generic objstore connector inherits the same base scan, so it benefits too.

Performance

Measured against a live bucket:

	Objects	Time
Before	291,422	3 hours 14 mins
Now	379,076	5.18 min

Type

refactor / perf

Testing

Tests added or updated
Tested locally (full from-source build + live S3 scan of 379,076 objects)
./builder test passes (not run)

Checklist

Commit messages follow conventional commits
No secrets or credentials included
Wiki updated (if applicable)
Breaking changes documented (if applicable) — none

Linked Issue

Fixes #1208

Summary by CodeRabbit

Bug Fixes
- Enhanced concurrent request handling by increasing S3 connection pool capacity.
Refactor
- Optimized scan client management with internal caching and resource pooling.
- Streamlined object discovery by eliminating redundant metadata API calls.
- Improved pagination logic for more efficient and reliable object enumeration.

Replace the StartAfter + result-equality dedup loop with proper ListObjectsV2 continuation-token pagination, so large buckets are walked page by page instead of relying on fragile end-of-listing heuristics. Cache the S3 client and share it across the scanner threads (guarded by a mutex, reset on list errors so the next scan re-connects), and raise ClientConfiguration.maxConnections to 64 so the shared connection pool does not serialize the scanner threads. Drop the per-object Content-Type HEAD request from processEntry; owner metadata is now fetched inline via ListObjectsV2 FetchOwner, removing a HEAD round-trip per object.

coderabbitai · 2026-06-10T13:30:30Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 570d3abb-70db-4377-9c7f-572548e59440

📥 Commits

Reviewing files that changed from the base of the PR and between 724c5ad and 29b92cd.

📒 Files selected for processing (3)

packages/server/engine-lib/engLib/store/endpoints/objstore/base/base.cpp
packages/server/engine-lib/engLib/store/endpoints/objstore/base/base.hpp
packages/server/engine-lib/engLib/store/endpoints/objstore/base/scan.cpp

📝 Walkthrough

Walkthrough

The PR optimizes S3 object-store bucket scanning by caching a shared S3 client across scanner threads, increasing the connection pool size, eliminating per-object HEAD requests, and replacing result-comparison pagination heuristics with proper ListObjectsV2 continuation-token pagination.

Changes

S3 Scan Performance Optimization

Layer / File(s)	Summary
Connection pool configuration for concurrent scanner threads `packages/server/engine-lib/engLib/store/endpoints/objstore/base/base.cpp`	S3 `ClientConfiguration.maxConnections` is set to 64 to prevent thread serialization when scanner threads share the connection pool.
Scan client caching interface `packages/server/engine-lib/engLib/store/endpoints/objstore/base/base.hpp`	`IBaseEndpoint::processEntry` signature is simplified to remove explicit `client` and `bucket` parameters; new protected methods `getScanClient()` and `resetScanClient()` manage a cached, mutex-guarded S3 client internally.
Cached scan client implementation and entry processing simplification `packages/server/engine-lib/engLib/store/endpoints/objstore/base/scan.cpp`	`getScanClient()` performs lazy initialization with mutex guarding; `resetScanClient()` drops the cached client; `processEntry()` is updated to accept only S3 object, entry, and callback; per-object `getContentType()` HEAD requests are removed.
ListObjectsV2 pagination with continuation tokens `packages/server/engine-lib/engLib/store/endpoints/objstore/base/scan.cpp`	`scanObjects()` uses the cached client and drives pagination via `ListObjectsV2` with `continuationToken`, `FetchOwner(true)`, and delimiter; pagination ends when `GetIsTruncated()` is false; returned common prefixes are processed as folders and objects as files; scan timing and counts are local variables instead of static.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 A faster scan hops through the cloud,
With clients shared (one pool, not loud),
No more head-checks for every file,
Continuation tokens make listing worthwhile,
Three hours down to moments—quite the mile! 🚀

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main performance improvements: S3 pagination and shared cached scanner client, directly matching the core changes.
Linked Issues check	✅ Passed	All coding objectives from issue `#1208` are met: ListObjectsV2 pagination implemented, S3 client cached and shared with mutex protection, maxConnections increased to 64, and per-object HeadObject requests eliminated.
Out of Scope Changes check	✅ Passed	All changes are directly scoped to addressing the performance bottlenecks identified in issue `#1208`; no unrelated modifications detected.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 Infer (1.2.0)

packages/server/engine-lib/engLib/store/endpoints/objstore/base/scan.cpp

packages/server/engine-lib/engLib/store/endpoints/objstore/base/scan.cpp:31:10: fatal error: 'engLib/eng.h' file not found
31 | #include <engLib/eng.h>
| ^~~~~~~~~~~~~~
1 error generated.
Error: the following clang command did not run successfully:
/opt/infer-linux-x86_64-v1.2.0/lib/infer/facebook-clang-plugins/clang/install/bin/clang-18
@/tmp/coderabbit-infer/29b92cd2b7a54724aa9227e687db1302d1e25553-3af6bafa7a8d9807/tmp/clang_command_.tmp.12a105.txt
++Contents of '/tmp/coderabbit-infer/29b92cd2b7a54724aa9227e687db1302d1e25553-3af6bafa7a8d9807/tmp/clang_command_.tmp.12a105.txt':
"-cc1" "-load"
"/opt/infer-linux-x86_64-v1.2.0/lib/infer/infer/bin/../../facebook-clang-plugins/libtooling/build/FacebookClangPlugin.dylib"
"-add-plugin" "BiniouASTExporter" "-plugin-arg-BiniouASTExporter" "-"
"-plugin-arg-BiniouASTExporter" "PREPEND_CURRENT_DIR=1"
"-plugin-arg-BiniouASTExporter" "MAX_STRING_SIZE=65535" "-cc1" "-triple"
"x86_64-unknown-linux-g

... [truncated 1151 characters] ...

tem" "/usr/local/include" "-internal-isystem"
"/usr/lib/gcc/x86_64-linux-gnu/12/../../../../x86_64-linux-gnu/include"
"-internal-externc-isystem" "/usr/include/x86_64-linux-gnu"
"-internal-externc-isystem" "/include" "-internal-externc-isystem"
"/usr/include" "-Wno-ignored-optimization-argument" "-Wno-everything"
"-fdeprecated-macro" "-ferror-limit" "19" "-fgnuc-version=4.2.1"
"-fskip-odr-check-in-gmf" "-fcxx-exceptions" "-fexceptions"
"-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-o"
"/tmp/coderabbit-infer/3af6bafa7a8d9807/file.o" "-x" "c++"
"packages/server/engine-lib/engLib/store/endpoints/objstore/base/scan.cpp"
"-O0" "-fno-builtin" "-include"
"/opt/infer-linux-x86_64-v1.2.0/lib/infer/infer/bin/../lib/clang_wrappers/global_defines.h"
"-Wno-everything"

packages/server/engine-lib/engLib/store/endpoints/objstore/base/base.cpp

packages/server/engine-lib/engLib/store/endpoints/objstore/base/base.cpp:30:10: fatal error: 'engLib/eng.h' file not found
30 | #include <engLib/eng.h>
| ^~~~~~~~~~~~~~
1 error generated.
Error: the following clang command did not run successfully:
/opt/infer-linux-x86_64-v1.2.0/lib/infer/facebook-clang-plugins/clang/install/bin/clang-18
@/tmp/coderabbit-infer/29b92cd2b7a54724aa9227e687db1302d1e25553-f36ed950b06d4502/tmp/clang_command_.tmp.5f30ab.txt
++Contents of '/tmp/coderabbit-infer/29b92cd2b7a54724aa9227e687db1302d1e25553-f36ed950b06d4502/tmp/clang_command_.tmp.5f30ab.txt':
"-cc1" "-load"
"/opt/infer-linux-x86_64-v1.2.0/lib/infer/infer/bin/../../facebook-clang-plugins/libtooling/build/FacebookClangPlugin.dylib"
"-add-plugin" "BiniouASTExporter" "-plugin-arg-BiniouASTExporter" "-"
"-plugin-arg-BiniouASTExporter" "PREPEND_CURRENT_DIR=1"
"-plugin-arg-BiniouASTExporter" "MAX_STRING_SIZE=65535" "-cc1" "-triple"
"x86_64-unknown-linux-g

... [truncated 1151 characters] ...

tem" "/usr/local/include" "-internal-isystem"
"/usr/lib/gcc/x86_64-linux-gnu/12/../../../../x86_64-linux-gnu/include"
"-internal-externc-isystem" "/usr/include/x86_64-linux-gnu"
"-internal-externc-isystem" "/include" "-internal-externc-isystem"
"/usr/include" "-Wno-ignored-optimization-argument" "-Wno-everything"
"-fdeprecated-macro" "-ferror-limit" "19" "-fgnuc-version=4.2.1"
"-fskip-odr-check-in-gmf" "-fcxx-exceptions" "-fexceptions"
"-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-o"
"/tmp/coderabbit-infer/f36ed950b06d4502/file.o" "-x" "c++"
"packages/server/engine-lib/engLib/store/endpoints/objstore/base/base.cpp"
"-O0" "-fno-builtin" "-include"
"/opt/infer-linux-x86_64-v1.2.0/lib/infer/infer/bin/../lib/clang_wrappers/global_defines.h"
"-Wno-everything"

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions · 2026-06-10T13:30:46Z

🤖 Internal: Discord sync marker

Auto-managed by the Discord notification workflow. Stores the linked Discord message ID. Do not edit or delete.

ksaurabhAparavi requested review from Rod-Christensen, jmaionchi and stepmikhaylov as code owners June 10, 2026 13:30

github-actions Bot added the module:server C++ engine and server components label Jun 10, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf(objstore): paginate S3 scan and share a cached scanner client#1209

perf(objstore): paginate S3 scan and share a cached scanner client#1209
ksaurabhAparavi wants to merge 1 commit into
rocketride-org:developfrom
ksaurabhAparavi:refactor/objstore-s3-scan-pagination

ksaurabhAparavi commented Jun 10, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jun 10, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

github-actions Bot commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ksaurabhAparavi commented Jun 10, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Performance

Type

Testing

Checklist

Linked Issue

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Poem

Uh oh!

github-actions Bot commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ksaurabhAparavi commented Jun 10, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 10, 2026 •

edited

Loading