quickwit-oss
diff --git a/‎docs/internals/adr/002-sort-schema-parquet-splits.md‎
Lines changed: 17 additions & 12 deletions b/‎docs/internals/adr/002-sort-schema-parquet-splits.md‎
Lines changed: 17 additions & 12 deletions
diff --git a/‎docs/internals/adr/gaps/002-fixed-sort-schema.md‎
Lines changed: 10 additions & 9 deletions b/‎docs/internals/adr/gaps/002-fixed-sort-schema.md‎
Lines changed: 10 additions & 9 deletions
diff --git a/‎docs/internals/adr/gaps/004-incomplete-split-metadata.md‎
Lines changed: 11 additions & 10 deletions b/‎docs/internals/adr/gaps/004-incomplete-split-metadata.md‎
Lines changed: 11 additions & 10 deletions
diff --git a/‎quickwit/quickwit-parquet-engine/Cargo.toml‎
Lines changed: 1 addition & 0 deletions b/‎quickwit/quickwit-parquet-engine/Cargo.toml‎
Lines changed: 1 addition & 0 deletions
diff --git a/‎quickwit/quickwit-parquet-engine/src/lib.rs‎
Lines changed: 1 addition & 0 deletions b/‎quickwit/quickwit-parquet-engine/src/lib.rs‎
Lines changed: 1 addition & 0 deletions
diff --git a/‎quickwit/quickwit-parquet-engine/src/row_keys/tests.rs‎
Lines changed: 5 additions & 16 deletions b/‎quickwit/quickwit-parquet-engine/src/row_keys/tests.rs‎
Lines changed: 5 additions & 16 deletions
diff --git a/‎quickwit/quickwit-parquet-engine/src/storage/mod.rs‎
Lines changed: 2 additions & 0 deletions b/‎quickwit/quickwit-parquet-engine/src/storage/mod.rs‎
Lines changed: 2 additions & 0 deletions
diff --git a/‎quickwit/quickwit-parquet-engine/src/storage/split_writer.rs‎
Lines changed: 6 additions & 5 deletions b/‎quickwit/quickwit-parquet-engine/src/storage/split_writer.rs‎
Lines changed: 6 additions & 5 deletions
@@ -52,9 +52,9 @@ Each column has:
 | **Name** | Column name as it appears in the Parquet schema |
 | **Direction** | Ascending (`+`, default) or descending (`-`). `timestamp` defaults to descending |
 | **Type** | Inferred from Parquet schema: string/binary (lexicographic), integer types (numeric), float types (numeric, NaN sorts after all values per IEEE 754 total order) |
-| **Null handling** | Nulls sort **after** non-null values for ascending columns, **before** non-null values for descending columns |
+| **Null handling** | Nulls always sort **after** non-null values (`nulls_first: false`), regardless of column direction |
 
-**Note on null handling:** The current implementation uses `nulls_first: true` for all columns. This must be changed to match the design: ascending columns should use `nulls_first: false` (nulls last), descending columns should use `nulls_first: true` (nulls first). This ensures nulls cluster at the end of each column's value range in both directions.
+**Note on null handling:** Nulls sort last for all columns. This simplifies compaction: when a sort column is absent from a split, all rows are treated as null for that column. With nulls-last, these rows cluster at the end and don't interfere with key-range comparisons between splits that do have the column. Implemented in PR #6295.
 
 ### 2. Schema Requirements
 
@@ -169,22 +169,27 @@ Phase 4 of the locality compaction roadmap extends sorting to the Tantivy pipeli
 
 | Component | Location | Status |
 |-----------|----------|--------|
-| Fixed sort at ingestion | `quickwit-parquet-engine/src/storage/writer.rs:84-109` | Done (Matthew Kim). Hardcoded sort on MetricName, TagService, TagEnv, TagDatacenter, TagRegion, TagHost, TimestampSecs |
-| Sort column definition | `quickwit-parquet-engine/src/schema/fields.rs:146-158` | Done. `ParquetField::sort_order()` returns fixed column list |
-| lexsort_to_indices usage | `quickwit-parquet-engine/src/storage/writer.rs` | Done. Arrow sort + take kernel applied in `sort_batch()` |
+| Fixed sort at ingestion | `quickwit-parquet-engine/src/storage/writer.rs` | Done (Matthew Kim). Replaced by configurable sort in PR #6287 |
+| Configurable sort schema | `quickwit-parquet-engine/src/table_config.rs` | Done (PR #6287). `TableConfig` with `effective_sort_fields()` override; `ParquetWriter` resolves sort fields dynamically |
+| Sort schema parser | `quickwit-parquet-engine/src/sort_fields/parser.rs` | Done (PR #6290). Parses `column\|...\|&metadata\|timestamp/V2` with directions, LSM cutoff, version |
+| Per-column sort direction | `sort_fields/parser.rs` + `storage/writer.rs` | Done (PR #6290 + #6287). Parser extracts `+`/`-` suffix; writer respects `descending` flag |
+| lexsort_to_indices usage | `quickwit-parquet-engine/src/storage/writer.rs` | Done. Arrow sort + take kernel with stable-sort tiebreaker |
+| Physical column ordering | `quickwit-parquet-engine/src/storage/writer.rs` | Done (PR #6287). Sort columns first, then sorted_series, then alphabetical |
+| timeseries_id computation | `quickwit-indexing/src/ingest/arrow_metrics.rs` | Done (PR #6286). SipHash-2-4 over canonicalized tag key/value pairs, computed at OTLP ingest |
+| sorted_series column | `quickwit-parquet-engine/src/sorted_series/mod.rs` | Done (PR #6290). Order-preserving binary encoding of sort schema tag values + timeseries_id |
+| RowKeys (min/max boundaries) | `quickwit-parquet-engine/src/row_keys/mod.rs` | Done (PR #6292). First/last row sort column values as proto, stored in KV metadata + MetricsSplitMetadata |
+| Zonemap regexes | `quickwit-parquet-engine/src/zonemap/mod.rs` | Done (PR #6295). Prefix-preserving superset regex per string sort column, stored in KV metadata + MetricsSplitMetadata |
+| Sort metadata in Parquet key_value_metadata | `quickwit-parquet-engine/src/storage/writer.rs` | Done (PR #6292 + #6295). `qh.sort_fields`, `qh.row_keys`, `qh.row_keys_json`, `qh.window_start`, `qh.window_duration_secs`, `qh.zonemap_regexes` |
+| Parquet native sorting_columns field | `quickwit-parquet-engine/src/storage/writer.rs` | Done (PR #6287). `sorting_columns()` sets column indices and directions |
+| Nulls-last ordering | `quickwit-parquet-engine/src/storage/writer.rs` | Done (PR #6295). `nulls_first: false` for all sort columns — nulls always sort after non-null values regardless of direction. Tested ascending + descending |
 
 ### Not Yet Implemented
 
 | Component | Notes | Gap |
 |-----------|-------|-----|
-| Sort schema parser | Parse `column\|...\|timestamp&metadata/V2` format | [GAP-002](./gaps/002-fixed-sort-schema.md) |
-| Sort schema in metastore | Schema stored per-index in metastore, mutable at runtime, propagated to pipelines on change | [GAP-002](./gaps/002-fixed-sort-schema.md) |
-| Configurable sort directions | Currently all ascending. Need per-column `+`/`-` | [GAP-002](./gaps/002-fixed-sort-schema.md) |
-| Correct null ordering | Currently `nulls_first: true` for all. Need nulls-last for ascending | [GAP-002](./gaps/002-fixed-sort-schema.md) |
+| Sort schema in metastore | Schema stored per-index in metastore, mutable at runtime, propagated to pipelines on change. Currently `TableConfig::default()` is hardcoded in `indexing_pipeline.rs` | [GAP-002](./gaps/002-fixed-sort-schema.md) (Phase 32) |
 | Parquet column index + offset index emission | Enable page-level min/max stats at write time | [GAP-004](./gaps/004-incomplete-split-metadata.md) |
-| Sort metadata in PostgreSQL | sort_schema, per-column min/max/regex in MetricsSplitMetadata | [GAP-004](./gaps/004-incomplete-split-metadata.md) |
-| Sort metadata in Parquet key_value_metadata | sort_schema, min/max/regex embedded in file | [GAP-004](./gaps/004-incomplete-split-metadata.md) |
-| Parquet native sorting_columns field | Declare sort order in Parquet file metadata | [GAP-004](./gaps/004-incomplete-split-metadata.md) |
+| Sort metadata in PostgreSQL | Full migration for row_keys + zonemap columns in `metrics_splits` table | [GAP-004](./gaps/004-incomplete-split-metadata.md) |
 
 ## References
 
 
@@ -1,8 +1,9 @@
 # GAP-002: Fixed Hardcoded Sort Schema
 
-**Status**: Open
+**Status**: Partially resolved
 **Discovered**: 2026-02-19
 **Context**: Codebase analysis during Phase 1 locality compaction design. Sort implementation by Matthew Kim provides the foundation but is not configurable.
+**Resolution**: PRs #6287–#6292 replaced the hardcoded sort with a configurable `TableConfig` + sort schema parser. Remaining: per-index metastore storage, pipeline propagation, null ordering fix.
 
 ## Problem
 
@@ -95,14 +96,14 @@ All of these systems treat sort order as a configurable, per-table property.
 
 ## Next Steps
 
-- [ ] Define sort schema parser for `column|...|timestamp&metadata/V2` format
-- [ ] Store sort schema as per-index property in the metastore (mutable at runtime)
-- [ ] Propagate sort schema changes from metastore to indexing pipelines on appropriate nodes
-- [ ] Replace `ParquetField::sort_order()` with schema-driven column selection
-- [ ] Fix null ordering: ascending columns use `nulls_first: false`, descending columns use `nulls_first: true`
-- [ ] Support per-column sort direction (`+`/`-` suffix)
-- [ ] Implement optional timeseries_id computation (xxHash64 or SipHash over canonicalized tags)
-- [ ] Store sort schema in MetricsSplitMetadata and Parquet file key_value_metadata
+- [x] Define sort schema parser for `column|...|timestamp&metadata/V2` format — PR #6290 (`sort_fields/parser.rs`), supports column names, `+`/`-` direction, `&` LSM cutoff, `/V2` version
+- [ ] Store sort schema as per-index property in the metastore (mutable at runtime) — future Phase 32
+- [ ] Propagate sort schema changes from metastore to indexing pipelines on appropriate nodes — future Phase 32
+- [x] Replace `ParquetField::sort_order()` with schema-driven column selection — PR #6287 (`writer.rs` uses `resolved_sort_fields` from `TableConfig.effective_sort_fields()`)
+- [x] Fix null ordering: nulls always sort last (`nulls_first: false`) regardless of column direction — PR #6295 (`sort_batch()`, `sorting_columns()`, SS-1 verification). Tested with ascending and descending columns in `test_nulls_sort_last_ascending_and_descending`
+- [x] Support per-column sort direction (`+`/`-` suffix) — PR #6290 (parser) + PR #6287 (writer respects `descending` flag)
+- [x] Implement optional timeseries_id computation — PR #6286 (SipHash-2-4 over canonicalized tags, computed at OTLP ingest in `arrow_metrics.rs`)
+- [x] Store sort schema in MetricsSplitMetadata and Parquet file key_value_metadata — PR #6292 (`qh.sort_fields` KV entry + `metadata.sort_fields`)
 
 ## References
 
 
@@ -1,8 +1,9 @@
 # GAP-004: Incomplete Split Metadata for Compaction and Query Pruning
 
-**Status**: Open
+**Status**: Partially resolved
 **Discovered**: 2026-02-19
 **Context**: Codebase analysis during Phase 1 locality compaction design
+**Resolution**: PRs #6287–#6295 added sort_schema, window, min/max (RowKeys), regex (zonemap), and sorting_columns to MetricsSplitMetadata and Parquet KV metadata. Remaining: Parquet column index + offset index enabling, PostgreSQL migration for new columns.
 
 ## Problem
 
@@ -111,16 +112,16 @@ Rich per-split (per-file) metadata for query pruning is standard in modern colum
 
 ## Next Steps
 
-- [ ] Add `sort_schema`, `window_start`, `window_duration_secs` to `MetricsSplitMetadata`
-- [ ] Add `SortColumnValue` tagged union type (string, i64, u64, f64, null)
-- [ ] Add `schema_column_min_values`, `schema_column_max_values`, `schema_column_regexes` to `MetricsSplitMetadata`
-- [ ] Create PostgreSQL migration adding new columns to `metrics_splits`
-- [ ] Update `PgMetricsSplit` and `InsertableMetricsSplit` in `postgres.rs`
+- [x] Add `sort_schema`, `window_start`, `window_duration_secs` to `MetricsSplitMetadata` — PR #6292 (`sort_fields`, `window` fields in `metadata.rs`)
+- [x] Add `SortColumnValue` tagged union type (string, i64, u64, f64, null) — PR #6292 (proto `ColumnValue` with `oneof` in `event_store_sortschema.proto`)
+- [x] Add `schema_column_min_values`, `schema_column_max_values`, `schema_column_regexes` to `MetricsSplitMetadata` — PR #6292 (`row_keys_proto: Option<Vec<u8>>` for min/max RowKeys), PR #6295 (`zonemap_regexes: HashMap<String, String>`)
+- [ ] Create PostgreSQL migration adding new columns to `metrics_splits` — partially done in PR #6292 (`InsertableMetricsSplit` has `row_keys`); full migration pending
+- [ ] Update `PgMetricsSplit` and `InsertableMetricsSplit` in `postgres.rs` — partially done in PR #6292; zonemap columns pending
 - [ ] Enable Parquet column index and offset index in writer properties
-- [ ] Set `sorting_columns` in Parquet file metadata based on sort schema
-- [ ] Write `sort_schema`, `schema_column_min_values`, `schema_column_max_values`, `schema_column_regexes`, `window_start`, `window_duration_secs` to Parquet `key_value_metadata`
-- [ ] Compute per-column min/max during split writing (scan sort + metadata-only columns)
-- [ ] Compute per-column regex during split writing (follow Husky implementation)
+- [x] Set `sorting_columns` in Parquet file metadata based on sort schema — PR #6287 (`sorting_columns()` method in `writer.rs`)
+- [x] Write `sort_schema`, `schema_column_min_values`, `schema_column_max_values`, `schema_column_regexes`, `window_start`, `window_duration_secs` to Parquet `key_value_metadata` — PR #6292 (`qh.sort_fields`, `qh.row_keys`, `qh.row_keys_json`, `qh.window_start`, `qh.window_duration_secs`), PR #6295 (`qh.zonemap_regexes`)
+- [x] Compute per-column min/max during split writing (scan sort + metadata-only columns) — PR #6292 (`extract_row_keys()` reads first/last row of sorted batch)
+- [x] Compute per-column regex during split writing (follow Husky implementation) — PR #6295 (`extract_zonemap_regexes()` with ported Go DFA automaton)
 
 ## References
 
 
@@ -30,6 +30,7 @@ ulid = { workspace = true }
 
 [dev-dependencies]
 proptest = { workspace = true }
+regex = { workspace = true }
 tempfile = { workspace = true }
 
 [features]
 
@@ -31,6 +31,7 @@ pub mod split;
 pub mod storage;
 pub mod table_config;
 pub mod timeseries_id;
+pub mod zonemap;
 
 #[cfg(any(test, feature = "testsuite"))]
 pub mod test_helpers;
@@ -414,7 +414,7 @@ fn test_row_keys_in_parquet_kv_metadata() {
 
     let temp_dir = std::env::temp_dir();
     let path = temp_dir.join("test_row_keys_kv.parquet");
-    let (_, row_keys_proto) = writer
+    let (_, (row_keys_proto, _zonemap_regexes)) = writer
         .write_to_file_with_metadata(&batch, &path, None)
         .unwrap();
 
@@ -687,21 +687,10 @@ fn test_row_keys_consistent_across_all_storage_paths() {
     // -- Min row (first after sort): cpu.usage / api / ... --
     assert_eq!(col_string(&min_cols[0]), "cpu.usage", "min metric_name");
     // service: "api" is the smallest non-null service value.
-    // Whether null sorts before "api" depends on nulls_first — the writer
-    // uses nulls_first=true, so the null-service row (cpu.usage/null)
-    // sorts before (cpu.usage/api). Let's check.
-    // Actually with nulls_first=true, null < "api", so the first row
-    // after sort is cpu.usage with service=null.
-    //
-    // But ColumnValue for null is { value: None }, so let's handle both.
-    // The key question: does the row with service=null sort before service="api"?
-    // With nulls_first=true: yes.
-    if min_cols[1].value.is_none() {
-        // First sorted row has null service (cpu.usage with no service).
-        // This is correct with nulls_first=true.
-    } else {
-        assert_eq!(col_string(&min_cols[1]), "api", "min service");
-    }
+    // The writer uses nulls_first=false (nulls sort last), so null-service
+    // rows sort after all non-null values. The first sorted row for
+    // cpu.usage has service="api".
+    assert_eq!(col_string(&min_cols[1]), "api", "min service");
 
     // timeseries_id at index 6 — should be an integer.
     assert!(
 
@@ -20,4 +20,6 @@ mod writer;
 
 pub use config::{Compression, ParquetWriterConfig};
 pub use split_writer::ParquetSplitWriter;
+#[cfg(test)]
+pub(crate) use writer::PARQUET_META_ZONEMAP_REGEXES;
 pub use writer::{ParquetWriteError, ParquetWriter};
@@ -147,14 +147,15 @@ impl ParquetSplitWriter {
         let mut metadata = builder.build();
 
         // Write with compaction metadata embedded in Parquet KV metadata.
-        // The writer sorts the batch and extracts RowKeys from the sorted
-        // first/last rows, returning them alongside the byte count.
-        let (size_bytes, row_keys_proto) =
-            self.writer
-                .write_to_file_with_metadata(batch, &file_path, Some(&metadata))?;
+        // The writer sorts the batch and extracts RowKeys + zonemap regexes
+        // from the sorted data, returning them alongside the byte count.
+        let (size_bytes, (row_keys_proto, zonemap_regexes)) = self
+            .writer
+            .write_to_file_with_metadata(batch, &file_path, Some(&metadata))?;
 
         metadata.size_bytes = size_bytes;
         metadata.row_keys_proto = row_keys_proto;
+        metadata.zonemap_regexes = zonemap_regexes;
 
         info!(
             split_id = %split_id,