CBG-5326: Function to handle document channel history compaction by RIT3shSapata · Pull Request #8218 · couchbase/sync_gateway

RIT3shSapata · 2026-04-30T13:44:17Z

CBG-5326

Describe your PR here...

Added a function to compact the channel history of a given document based on sequence number
Added a high level test coverage for the same

Pre-review checklist

Removed debug logging (fmt.Print, log.Print, ...)
Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Dependencies (if applicable)

Link upstream PRs
Update Go module dependencies when merged

Integration Tests

Copilot

Pull request overview

Adds a new DatabaseCollection helper to compact per-document channel assignment history up to a given sequence, with a REST-level test validating the basic behavior.

Changes:

Introduces CompactDocChannelHistory(ctx, docid, seq) to prune ChannelSetHistory and persist the updated _sync (and _mou) xattrs.
Adds an API test that creates/removes/re-adds channels to generate history, then compacts and validates the result.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 8 comments.

File	Description
`db/crud.go`	Adds the `CompactDocChannelHistory` implementation and persists updated sync metadata via xattr update.
`rest/api_test.go`	Adds a high-level test covering history generation and compaction behavior.

adamcfraser

Left one comment - have a look at that as well as the copilot comments and assign back once you've updated the PR.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

+		// Compact at seq 3
+		err = collection.CompactDocChannelHistory(ctx, "doc4", 2)
+		require.NoError(t, err)
+
+		syncDataAfter, err := collection.GetDocSyncData(ctx, "doc4")
+		require.NoError(t, err)
+
+		// Verify: only entries with End > 3 or End == 0 (still active) should remain
+		for _, entry := range syncDataAfter.ChannelSet {
+			assert.True(t, entry.End == 0 || entry.End > uint64(3))
+		}
+
+		// Verify only channels with seq > 3 should remain
+		for _, entry := range syncDataAfter.Channels {
+			assert.True(t, entry.Seq > uint64(3))


+		syncDataBefore, err := collection.GetDocSyncData(ctx, "doc7")
+		require.NoError(t, err)
+		err = collection.CompactDocChannelHistory(ctx, "doc7", 2)
+		require.NoError(t, err)
+
+		syncData, err := collection.GetDocSyncData(ctx, "doc7")
+		require.NoError(t, err)
+		// Should still have active channels
+		assert.Less(t, len(syncData.ChannelSet), len(syncDataBefore.ChannelSet))
+		assert.Less(t, len(syncData.Channels), len(syncDataBefore.Channels))


+// CompactDocChannelHistory removes channel history entries that ended at or before the given sequence number.
+// This is used to truncate stale channel assignment history to reduce storage overhead.
+func (c *DatabaseCollection) CompactDocChannelHistory(ctx context.Context, docid string, seq uint64) (err error) {
+	key := realDocID(docid)
+	if key == "" {
+		return base.HTTPErrorf(400, "Invalid doc ID")
+	}
+
+	_, xattrs, cas, err := c.dataStore.GetWithXattrs(ctx, key, c.syncGlobalSyncMouRevSeqNoAndUserXattrKeys())
+	if err != nil {
+		return
+	}


Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

+	metadataOnlyUpdate := computeMetadataOnlyUpdate(doc.Cas, revSeqNo, doc.MetadataOnlyUpdate)
+
+	rawMouXattr, err := base.JSONMarshal(metadataOnlyUpdate)
+	if err != nil {
+		return base.RedactErrorf("failed to marshal _mou when attempting to compact channel history for doc: %s. Error: %v", base.UD(docid), err)
+	}
+
+	// build macro expansion for sync data. This will avoid the update to xattrs causing an extra import event (i.e. sync cas will be == to doc cas)
+	opts := &sgbucket.MutateInOptions{}
+	spec := []sgbucket.MacroExpansionSpec{
+		sgbucket.NewMacroExpansionSpec(xattrCasPath(base.SyncXattrName), sgbucket.MacroCas),
+		sgbucket.NewMacroExpansionSpec(XattrMouCasPath(), sgbucket.MacroCas),
+	}
+	opts.MacroExpansion = spec
+	opts.PreserveExpiry = true // if doc has expiry, we should preserve this
+
+	updatedXattr := map[string][]byte{
+		base.SyncXattrName: rawSyncXattr,
+		base.MouXattrName:  rawMouXattr,
+	}


adamcfraser

I like the new approach for computing changes. Some comments about handling for on-demand import and minor style suggestions.

adamcfraser · 2026-05-08T18:29:38Z

+	// build macro expansion for sync data. This will avoid the update to xattrs causing an extra import event (i.e. sync cas will be == to doc cas)
+	opts := &sgbucket.MutateInOptions{}
+	spec := []sgbucket.MacroExpansionSpec{
+		sgbucket.NewMacroExpansionSpec(xattrCasPath(base.SyncXattrName), sgbucket.MacroCas),


It's correct to update sync.cas here, because we don't want this compaction to trigger a new document import. However, this means that we need to make sure this processing isn't being run on documents that haven't yet been imported - otherwise this cas update will make it appear as if this document has already been imported (and so the sync metadata would be incorrect).

I think the best way to avoid this is to trigger on-demand import as needed when initially fetching the document. I think you can do that by using GetDocument instead of GetWithXattrs when originally fetching the document on line 229 - this has the on-demand import work built-in.

Will want to add a test case for that (running compact on a non-imported document).

Added check for non imported document and triggering on demand import in the same function. I went through the GetDocument method and the xattr keys which it fetches is base.SyncXattrName, base.VvXattrName, base.GlobalXattrName, the xattr keys which are required for compaction are base.SyncXattrName, base.VirtualXattrRevSeqNo, base.MouXattrName.

Let me know your thoughts on this approach, or would you prefer I use GetDocument itself.

Also while writing the test for this scenario, ran into a few challenges, will discuss this further offline.

gregns1 · 2026-05-11T15:53:47Z

+	if !isSgWrite {
+		var importErr error
+
+		doc, importErr = c.OnDemandImportForGet(ctx, docid, doc, rawDoc, xattrs, cas)


Make sure you have a nil check on doc here too to avoid the same panic found in https://jira.issues.couchbase.com/browse/CBG-5355

Code else where for reference:

sync_gateway/db/crud.go

Line 198 in d09f0fd

if doc == nil {

We have a ticket in backlog to make this code better but not sure if we'll get to it in the 4.1 timeframe so be good to protect against it here.

refactor the slices and maps delete oprations, get and umarshall only the required xattrs, refactor the spec for mou update, use helper function to compute mou, refactor naked returns, early exit in a case where there is no compaction

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

+		return base.HTTPErrorf(400, "Invalid doc ID")
+	}
+
+	xattrKeys := []string{base.SyncXattrName, base.VirtualXattrRevSeqNo, base.MouXattrName}


+			return importErr
+		}
+		if doc == nil {
+			return fmt.Errorf("skipping compaction of document %s, %v ", base.UD(docid), base.ErrNotFound)


+// TestCompactNonImportedDocWithAutoImport verifies that when CompactDocChannelHistory is called
+// on a non-imported document, it automatically imports the document first before performing compaction.
+func TestCompactNonImportedDocWithAutoImport(t *testing.T) {
+	if !base.TestUseXattrs() {
+		t.Skip("CompactDocChannelHistory requires XATTR-based metadata")
+	}
+
+	// Create RestTester with AutoImport disabled to allow non-imported documents
+	rtConfig := RestTesterConfig{
+		SyncFn: channels.DocChannelsSyncFunction,
+		DatabaseConfig: &DatabaseConfig{DbConfig: DbConfig{
+			AutoImport: false,


RIT3shSapata self-assigned this Apr 30, 2026

Copilot AI review requested due to automatic review settings April 30, 2026 13:44

Copilot started reviewing on behalf of RIT3shSapata April 30, 2026 13:44 View session

Copilot AI reviewed Apr 30, 2026

View reviewed changes

Comment thread db/crud.go

Comment thread rest/api_test.go

Comment thread rest/api_test.go Outdated

Comment thread rest/api_test.go Outdated

Comment thread rest/api_test.go Outdated

Comment thread db/crud.go Outdated

Comment thread db/crud.go Outdated

Comment thread db/crud.go

RIT3shSapata requested a review from adamcfraser April 30, 2026 16:22

RIT3shSapata assigned adamcfraser and unassigned RIT3shSapata Apr 30, 2026

adamcfraser requested changes May 1, 2026

View reviewed changes

Comment thread db/crud.go Outdated

adamcfraser assigned RIT3shSapata and unassigned adamcfraser May 1, 2026

RIT3shSapata requested review from adamcfraser and Copilot May 5, 2026 13:56

RIT3shSapata assigned adamcfraser and unassigned RIT3shSapata May 5, 2026

Copilot started reviewing on behalf of RIT3shSapata May 5, 2026 13:57 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

adamcfraser requested changes May 7, 2026

View reviewed changes

Comment thread db/crud.go

Comment thread db/crud.go Outdated

Comment thread db/crud.go Outdated

Comment thread db/crud.go Outdated

Comment thread db/crud.go Outdated

Comment thread db/crud.go Outdated

RIT3shSapata force-pushed the CBG-5326 branch from ed372a1 to 40b882b Compare May 8, 2026 11:44

RIT3shSapata requested review from adamcfraser and Copilot May 8, 2026 13:10

Copilot started reviewing on behalf of RIT3shSapata May 8, 2026 13:11 View session

Copilot AI reviewed May 8, 2026

View reviewed changes

adamcfraser requested changes May 8, 2026

View reviewed changes

adamcfraser assigned RIT3shSapata and unassigned adamcfraser May 8, 2026

gregns1 reviewed May 11, 2026

View reviewed changes

RIT3shSapata added 4 commits May 13, 2026 12:32

add function to delete the channel history of the document

194e0fd

lint fix

4e7f5b1

fixes based on pr comments

b4c398e

fix seq of subtests

9d45b92

RIT3shSapata added 3 commits May 13, 2026 12:32

fixes based on pr comments

c5b9ae5

refactor the slices and maps delete oprations, get and umarshall only the required xattrs, refactor the spec for mou update, use helper function to compute mou, refactor naked returns, early exit in a case where there is no compaction

fixes based on pr comments

c518a09

add test for autoimport during channel history compaction

738ae0a

RIT3shSapata force-pushed the CBG-5326 branch from 8e56474 to 738ae0a Compare May 13, 2026 10:01

lint fix

592b361

RIT3shSapata requested review from adamcfraser and Copilot May 13, 2026 10:11

RIT3shSapata assigned adamcfraser and unassigned RIT3shSapata May 13, 2026

Copilot started reviewing on behalf of RIT3shSapata May 13, 2026 10:11 View session

Copilot AI reviewed May 13, 2026

View reviewed changes

lint fix

faad903

Conversation

RIT3shSapata commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-review checklist

Dependencies (if applicable)

Integration Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

adamcfraser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

adamcfraser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

adamcfraser May 8, 2026

Choose a reason for hiding this comment

Uh oh!

RIT3shSapata May 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gregns1 May 11, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

RIT3shSapata commented Apr 30, 2026 •

edited

Loading