feat(iceberg): single JVM per process instead of per stream-chunk by vikaxsh · Pull Request #962 · datazip-inc/olake

vikaxsh · 2026-05-26T10:49:08Z

Description

This PR refactors the Iceberg writer architecture to use a single shared JVM per OLake CLI invocation instead of creating one JVM per writer thread/chunk.

Previously, every Iceberg writer initialization spawned a dedicated JVM-backed Iceberg client and gRPC server, resulting in significant memory overhead under concurrent backfill and CDC workloads. Large syncs could create many JVM processes, leading to excessive memory consumption and potential OOM issues.

Changes

Introduced a shared JVM architecture where all streams and chunks communicate with a single JVM instance.
Added ThreadSession-based isolation to maintain per-stream/per-chunk state within the shared JVM.
Moved stream-specific configuration from JVM startup arguments to gRPC request metadata:
- Namespace
- Upsert mode
- Identifier field creation
- Iceberg partition transforms
Added StreamMetaCtx on the Go side to propagate stream-specific metadata with every request.
Added ThreadSession management in Java to maintain isolated:
- Iceberg table handles
- Table operators
- Writers
- Commit state
Added CLOSE_SESSION RPC operation for explicit session cleanup and resource release.
Retained catalog initialization and other truly global resources at the JVM level.

Benefits

JVM heap is allocated only once per OLake run.
Significantly reduces memory consumption during concurrent syncs.
Eliminates excessive JVM process creation.
Preserves stream-level isolation through session-based state management.
Improves scalability for parallel backfill and CDC workloads.

Fixes # (issue)

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

Validated concurrent backfill execution using multiple chunks sharing a single JVM instance.
Validated CDC streams operating concurrently through the shared JVM.
Verified session creation and cleanup via CLOSE_SESSION.
Verified successful table creation, schema evolution, writes, and commits across multiple sessions.
Verified memory utilization remains stable compared to the previous multi-JVM architecture.

Screenshots or Recordings

N/A

Documentation

Documentation Link: [link to README, olake.io/docs, or olake-docs]
N/A (bug fix, refactor, or test changes only)

Related PR's (If Any):

N/A

… into feat/unified-jvm

…se using a cancelled flag

…eat/unified-jvm

… into feat/unified-jvm

…eat/unified-jvm

hash-data · 2026-06-23T04:35:15Z

+// (namespace, upsert, partition spec, identifier-field) was already captured by
+// the JVM on the GET_OR_CREATE_TABLE payload during Setup, so RECORDS / COMMIT
+// payloads carry only the thread_id the JVM routes on (callers add schema/payload).
+func (w *LegacyWriter) newMetadata() *proto.IcebergPayload_Metadata {


we can remove this

hash-data · 2026-06-23T04:37:11Z

+		case <-done:
+		case <-ctx.Done():
+			logger.Warnf("Context cancelled, killing Iceberg JVM")
+			_ = s.cmd.Process.Kill()


check once again this logic

this comment is for myself

hash-data · 2026-06-24T08:44:57Z

-		err := i.server.closeIcebergClient()
-		if err != nil {
-			logger.Errorf("Thread[%s]: error closing Iceberg client: %s", i.options.ThreadID, err)
+		if cleanupErr := i.writer.Cleanup(); cleanupErr != nil {


do we still need separate this?

hash-data · 2026-06-24T09:08:29Z

+    private static final String FILE_TYPE_EQUALITY_DELETE = "equalityDelete";
+    private static final String FILE_TYPE_POSITIONAL_DELETE = "positionalDelete";
+
+    private final Catalog icebergCatalog;


can we remove unenecessary variables

hash-data · 2026-06-24T09:41:44Z

 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.List;
+import java.util.function.BooleanSupplier;


not required

hash-data · 2026-06-24T09:43:18Z

+                    session.icebergTable.refresh();
+                    String commitState = session.op.getCommitState(session.icebergTable);
+                    sendResponse(responseObserver, session.icebergTable.schema().toString(),
+                            commitState != null ? commitState : "");


dont we send some text if there is not state?

hash-data · 2026-06-24T09:46:47Z

+            // DROP_TABLE carries "db.table" in destTableName and must NOT create
+            // a per-thread session (computeIfAbsent would load/create the very
+            // table we're about to drop). Handle it before session setup.
+            if (request.getType() == IcebergPayload.PayloadType.DROP_TABLE) {


why we need it a separate?

hash-data · 2026-06-24T09:47:14Z

    }
-}
+
+    private static List<Map<String, String>> toPartitionList(List<IcebergPayload.PartitionField> protos) {


this we not had prev?

any reason for addition?

hash-data · 2026-06-24T09:53:56Z

+            // CLOSE_SESSION: just drop the session. No closeQuietly() — it would
+            // clear op.filesToCommit while an in-flight REGISTER_AND_COMMIT may be
+            // using that list. Nothing to close here; the session is GC'd.
+            if (request.getType() == ArrowPayload.PayloadType.CLOSE_SESSION) {


we dont need at multiple place right

vikaxsh added 2 commits May 26, 2026 16:18

feat(iceberg): single JVM per process instead of per stream-chunk

77fc64e

Merge branch 'staging' into feat/unified-jvm

f5dce7c

vikaxsh marked this pull request as ready for review May 26, 2026 10:49

vikaxsh added 2 commits May 26, 2026 16:20

chore: dummy commit

96967db

Merge branch 'feat/unified-jvm' of https://github.com/datazip-inc/olake…

2c856ca

… into feat/unified-jvm

vikaxsh marked this pull request as draft May 26, 2026 10:52

vikaxsh added 2 commits May 26, 2026 16:25

chore: undo dummy commit

582f1ff

feat: prevent concurrent writes to the same writer during session clo…

e7b0387

…se using a cancelled flag

vikaxsh requested a deployment to integration_tests June 2, 2026 05:49 — with GitHub Actions Waiting

Merge branch 'staging' of https://github.com/datazip-inc/olake into f…

9fc2f60

…eat/unified-jvm

vikaxsh marked this pull request as ready for review June 2, 2026 05:50

vikaxsh requested a deployment to integration_tests June 2, 2026 05:50 — with GitHub Actions Waiting

fix: lint issue

3c19007

vikaxsh temporarily deployed to integration_tests June 2, 2026 05:57 — with GitHub Actions Inactive

fix: keep thread id consistant in check conenction in dest

d6c5fcc

vikaxsh requested a deployment to integration_tests June 2, 2026 08:01 — with GitHub Actions Waiting

fix: 2pc state refresh issue

98c8259

vikaxsh requested a deployment to integration_tests June 3, 2026 05:46 — with GitHub Actions Waiting

chore: refactor code

71b04e3

vikaxsh requested a deployment to integration_tests June 3, 2026 06:58 — with GitHub Actions Waiting

chore: added a todo

65febe1

vikaxsh requested a deployment to integration_tests June 3, 2026 07:40 — with GitHub Actions Waiting

chore: need to remove this

fdba7b1

vikaxsh requested a deployment to integration_tests June 17, 2026 09:54 — with GitHub Actions Waiting

vikaxsh added 2 commits June 17, 2026 15:33

fix: lint issue

b686551

Merge branch 'feat/unified-jvm' of https://github.com/datazip-inc/olake…

e0e78f1

… into feat/unified-jvm

vikaxsh requested a deployment to integration_tests June 17, 2026 10:04 — with GitHub Actions Waiting

fix: remove duplicated Shutdownable

1174788

vikaxsh requested a deployment to integration_tests June 17, 2026 10:36 — with GitHub Actions Waiting

hash-data and others added 3 commits June 22, 2026 19:36

refactor: go side refactoring (#997)

46ba339

fix: remove backoff retry in iceberg namespace creation

f665d49

Merge branch 'staging' of https://github.com/datazip-inc/olake into f…

fc515ce

…eat/unified-jvm

vikaxsh requested a deployment to integration_tests June 23, 2026 06:36 — with GitHub Actions Waiting

fix: lint issue

5fbc965

vikaxsh requested a deployment to integration_tests June 23, 2026 06:54 — with GitHub Actions Waiting

feat: pass shared session in OlakeRowsIngester and OlakeArrowIngester

23f1ae7

vikaxsh requested a deployment to integration_tests June 23, 2026 10:00 — with GitHub Actions Waiting

vikaxsh added 2 commits June 23, 2026 16:11

Merge branch 'staging' of https://github.com/datazip-inc/olake into f…

26b25d7

…eat/unified-jvm

chore: remove whitspace

bfd90b0

vikaxsh requested a deployment to integration_tests June 23, 2026 10:43 — with GitHub Actions Waiting

fix: interface nil panic

bfb3d17

vikaxsh requested a deployment to integration_tests June 23, 2026 10:53 — with GitHub Actions Waiting

fix: add sleep time before remvoing session

a820f0f

vikaxsh requested a deployment to integration_tests June 23, 2026 11:45 — with GitHub Actions Waiting

hash-data reviewed Jun 24, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(iceberg): single JVM per process instead of per stream-chunk#962

feat(iceberg): single JVM per process instead of per stream-chunk#962
vikaxsh wants to merge 33 commits into
stagingfrom
feat/unified-jvm

vikaxsh commented May 26, 2026 •

edited

Loading

Uh oh!

hash-data Jun 23, 2026

Uh oh!

hash-data Jun 23, 2026

Uh oh!

hash-data Jun 24, 2026

Uh oh!

hash-data Jun 24, 2026

Uh oh!

hash-data Jun 24, 2026

Uh oh!

Uh oh!

hash-data Jun 24, 2026

Uh oh!

hash-data Jun 24, 2026

Uh oh!

hash-data Jun 24, 2026

Uh oh!

hash-data Jun 24, 2026

Uh oh!

hash-data Jun 24, 2026

Uh oh!

hash-data Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

vikaxsh commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Benefits

Type of change

How Has This Been Tested?

Screenshots or Recordings

Documentation

Related PR's (If Any):

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vikaxsh commented May 26, 2026 •

edited

Loading