feat: add production realism pack for schema-lens v0.1.1

Author: kamlendrachauhan

README.md

@@ -2,7 +2,7 @@
 
 Schema Lens is a Solr schema evolution impact simulator. It compares baseline vs shadow relevance before you ship schema/query changes.
 
-Target first release: `v0.1.0`.
+Target release: `v0.1.1`.
 
 ## Why
 
@@ -11,13 +11,20 @@ Schema and query parameter changes can silently degrade ranking quality. `schema
 ## Features (v0.1)
 
 - CLI commands: `validate`, `inspect`, `shadow create`, `shadow index`, `replay`, `compare`, `report`, `run`
+- Additional commands: `queries extract`, `docs sample`, `gate`
 - SolrCloud-first shadow collection lifecycle via Collections API
 - Schema change operations:
   - `schema.field.update`
   - `schema.fieldType.replace`
   - `schema.analyzer.remove_filter`
   - `queryparams.set`
 - Query replay and metrics: overlap@K, jaccard@K, kendall tau@K
+- Schema preflight dependency/risk report: `schema_risk.json`
+- Optional production realism sources:
+  - Queries from logs (`queries.source.type=log`)
+  - Docs sampled directly from Solr (`data.docs_source.type=solr`)
+- Optional structured explain capture (`evaluation.explain.structured=true`)
+- Quality gate policy command for CI (`schema-lens gate`)
 - Reproducible outputs: `run_manifest.json`, `replay.json`, `compare.json`, `report.json`, `report.html`
 
 ## Quickstart
@@ -50,6 +57,50 @@ schema-lens run examples/changesets/fieldtype-change.yaml --out out/demo
 cat out/demo/report.json
 ```
 
+## Run Against Production Safely
+
+Use the production realism example:
+
+```bash
+schema-lens run examples/changesets/prod_realism_example.yaml --out out/prod_like_run
+```
+
+Extract canonical replay queries from logs:
+
+```bash
+schema-lens queries extract \
+  --from examples/logs/solr_requests.log \
+  --out out/queries_extracted.jsonl \
+  --max 500 \
+  --sample reservoir
+```
+
+Sample docs from Solr:
+
+```bash
+schema-lens docs sample \
+  --solr-url http://localhost:8983/solr \
+  --collection products \
+  --mode cursormark \
+  --query "*:*" \
+  --fl "id,title,text,category,price" \
+  --sample-n 5000 \
+  --out out/docs_sample.jsonl
+```
+
+Apply CI relevance gate:
+
+```bash
+schema-lens gate \
+  --compare out/prod_like_run/compare.json \
+  --policy examples/policy/gate_default.yaml
+```
+
+Warnings:
+- Keep sampled docs lean (`fl`) and bounded (`sample_n`).
+- Keep sanitization enabled when replaying from real logs.
+- Review `schema_risk.json` before approving schema rollout.
+
 ## Example command
 
 ```bash

docs/changeset-spec.md

@@ -26,18 +26,43 @@ shadow:
 
 data:
   docs_source:
-    type: "file"
-    path: "examples/docs.jsonl"
+    type: "file" # "file" | "solr"
+    path: "examples/docs.jsonl" # required for type=file
     format: "jsonl"
     id_field: "id"
+    # solr source options (type=solr):
+    # solr_url: "http://localhost:8983/solr"
+    # collection: "products"
+    # mode: "export" # "export" | "cursormark"
+    # query: "*:*"
+    # sort: "id asc"
+    # fl: "id,title,text,category"
+    # sample_n: 50000
+    # batch_size: 500
+    # out_sample_path: "out/docs_sample.jsonl"
   sample_n: 50000
 
 queries:
   source:
-    type: "file"
+    type: "file" # "file" | "log"
     path: "examples/queries.txt"
-    format: "simple"
+    format: "simple" # file: "simple" | "jsonl", log: "solr_params" | "jsonl"
   max_queries: 2000
+  sampling:
+    mode: "reservoir" # "top" | "reservoir"
+    seed: 42
+  sanitize:
+    enabled: true
+    rules:
+      - type: "mask_email"
+      - type: "mask_uuid"
+      - type: "drop_param"
+        name: "token"
+      - type: "drop_param"
+        name: "auth"
+
+preflight:
+  fail_on_risk: false
 
 changes:
   - op: "schema.field.update"
@@ -64,6 +89,7 @@ evaluation:
     - kendall_tau
   explain:
     enabled: true
+    structured: false
     max_queries: 25
     max_docs_per_query: 3
 ```
@@ -72,7 +98,8 @@ evaluation:
 
 - `baseline.solr_url`
 - `baseline.collection`
-- `data.docs_source.path`
+- `data.docs_source.path` when `data.docs_source.type=file`
+- `data.docs_source.solr_url` and `data.docs_source.collection` when `data.docs_source.type=solr`
 - `queries.source.path`
 
 ## Supported operations
@@ -85,6 +112,9 @@ evaluation:
 ## Notes
 
 - `queryparams.set` affects replay parameters only.
+- `queries.source.type=log` enables log extraction + canonical JSONL replay generation.
+- `data.docs_source.type=solr` samples docs from Solr and writes reproducible JSONL output.
+- Preflight always emits `schema_risk.json`; set `preflight.fail_on_risk=true` to block execution on HIGH risks.
 - `schema.analyzer.remove_filter.filter_class` can be a Java class (for example `solr.LowerCaseFilterFactory`) or the short filter name (`lowercase`).
 - `shadow.allow_shared_configset_fallback=true` allows a non-isolated fallback when Solr blocks configset clone operations (401 on trusted base configsets). This is explicit and can affect baseline behavior.
 - Empty `changes` is allowed with a warning.

examples/changesets/prod_realism_example.yaml

@@ -0,0 +1,80 @@
+schema_lens_version: 1
+
+baseline:
+  solr_url: "http://localhost:8983/solr"
+  collection: "products"
+  request_defaults:
+    rows: 10
+    fl: "id,score"
+    defType: "edismax"
+    extra_params:
+      fq: []
+      qf: "title^3 text"
+      pf: "title^10"
+
+shadow:
+  mode: "solrcloud"
+  solr_url: "http://localhost:8983/solr"
+  collection_name_template: "{collection}__shadow__{ts}"
+  num_shards: 1
+  replication_factor: 1
+  cleanup: true
+  allow_shared_configset_fallback: true
+
+preflight:
+  fail_on_risk: false
+
+data:
+  docs_source:
+    type: "solr"
+    solr_url: "http://localhost:8983/solr"
+    collection: "products"
+    mode: "cursormark"
+    query: "*:*"
+    sort: "id asc"
+    fl: "id,title,text,category,price"
+    sample_n: 200
+    batch_size: 50
+    out_sample_path: "out/docs_sample.jsonl"
+
+queries:
+  source:
+    type: "log"
+    path: "examples/logs/solr_requests.log"
+    format: "solr_params"
+  max_queries: 100
+  sampling:
+    mode: "reservoir"
+    seed: 42
+  sanitize:
+    enabled: true
+    rules:
+      - type: "mask_email"
+      - type: "mask_uuid"
+      - type: "drop_param"
+        name: "auth"
+      - type: "drop_param"
+        name: "token"
+
+changes:
+  - op: "schema.field.update"
+    field: "title"
+    set:
+      stored: true
+  - op: "queryparams.set"
+    set:
+      qf: "title^5 text"
+      pf: "title^20"
+
+evaluation:
+  k: 10
+  metrics:
+    - overlap
+    - jaccard
+    - kendall_tau
+  explain:
+    enabled: true
+    structured: true
+    max_queries: 5
+    max_docs_per_query: 2
+

examples/logs/solr_requests.log

@@ -0,0 +1,4 @@
+q=bearings&fq=category:tools&defType=edismax&qf=title%5E3%20text&rows=20
+q=pipes&fq=price:%5B10%20TO%2050%5D&sort=price%20asc
+/browse/select?q=bolts&fq=tenant:acme&mm=2<75%25&pf=title%5E10&debug=false
+

examples/policy/gate_default.yaml

@@ -0,0 +1,28 @@
+version: 1
+k: 10
+
+fail:
+  - metric: "avg_overlap"
+    op: "<"
+    value: 0.80
+  - metric: "pct_high_risk_queries"
+    op: ">"
+    value: 5.0
+  - metric: "pct_queries_overlap_lt"
+    args:
+      threshold: 0.60
+    op: ">"
+    value: 10.0
+
+warn:
+  - metric: "pct_med_risk_queries"
+    op: ">"
+    value: 20.0
+
+golden_queries:
+  enabled: true
+  file: "../queries/golden.jsonl"
+  requirements:
+    must_contain_topk: 10
+    max_missing_pct: 0.0
+

examples/queries/golden.jsonl

@@ -0,0 +1,3 @@
+{"name":"supplier exact","params":{"q":"SupplierId:1001247","defType":"lucene"},"expected_ids":["1001247"]}
+{"name":"category head","params":{"q":"bearings","defType":"edismax","qf":"title^3 text"},"expected_ids":["B-001","B-019"]}
+

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "schema-lens"
-version = "0.1.0"
+version = "0.1.1"
 description = "Schema Evolution & Impact Simulator for Solr"
 readme = "README.md"
 requires-python = ">=3.11"

schema_lens/__init__.py

@@ -1,3 +1,3 @@
 """schema-lens package."""
 
-__version__ = "0.1.0"
+__version__ = "0.1.1"

schema_lens/changesets/validator.py

@@ -52,16 +52,52 @@ def validate_changeset(changeset: Changeset, check_paths: bool = True) -> Valida
     if version not in (None, 1):
         report.errors.append(f"Unsupported schema_lens_version: {version}")
 
-    required = [
-        "baseline.solr_url",
-        "baseline.collection",
-        "data.docs_source.path",
-        "queries.source.path",
-    ]
+    required = ["baseline.solr_url", "baseline.collection"]
     for key in required:
         if _get_in(raw, key) in (None, ""):
             report.errors.append(f"Missing required field: {key}")
 
+    docs_source = _get_in(raw, "data.docs_source") or {}
+    if not isinstance(docs_source, dict):
+        report.errors.append("data.docs_source must be an object")
+        docs_source = {}
+    docs_source_type = str(docs_source.get("type", "file"))
+    if docs_source_type not in {"file", "solr"}:
+        report.errors.append("data.docs_source.type must be 'file' or 'solr'")
+    if docs_source_type == "file":
+        if not docs_source.get("path"):
+            report.errors.append("Missing required field: data.docs_source.path")
+    else:
+        for key in ("solr_url", "collection"):
+            if not docs_source.get(key):
+                report.errors.append(f"Missing required field: data.docs_source.{key}")
+        mode = docs_source.get("mode")
+        if mode and mode not in {"export", "cursormark"}:
+            report.errors.append("data.docs_source.mode must be 'export' or 'cursormark'")
+
+    query_source = _get_in(raw, "queries.source") or {}
+    if not isinstance(query_source, dict):
+        report.errors.append("queries.source must be an object")
+        query_source = {}
+    query_source_type = str(query_source.get("type", "file"))
+    if query_source_type not in {"file", "log"}:
+        report.errors.append("queries.source.type must be 'file' or 'log'")
+    if not query_source.get("path"):
+        report.errors.append("Missing required field: queries.source.path")
+
+    if query_source_type == "log":
+        fmt = str(query_source.get("format", "solr_params"))
+        if fmt not in {"solr_params", "jsonl"}:
+            report.errors.append("queries.source.format must be 'solr_params' or 'jsonl'")
+
+    sampling_mode = _get_in(raw, "queries.sampling.mode")
+    if sampling_mode is not None and sampling_mode not in {"top", "reservoir"}:
+        report.errors.append("queries.sampling.mode must be 'top' or 'reservoir'")
+
+    preflight_fail = _get_in(raw, "preflight.fail_on_risk")
+    if preflight_fail is not None and not isinstance(preflight_fail, bool):
+        report.errors.append("preflight.fail_on_risk must be boolean")
+
     changes = raw.get("changes", [])
     if not isinstance(changes, list):
         report.errors.append("changes must be a list")
@@ -103,12 +139,11 @@ def validate_changeset(changeset: Changeset, check_paths: bool = True) -> Valida
             report.errors.append(f"{loc}.set must be an object")
 
     if check_paths:
-        docs_path = _get_in(raw, "data.docs_source.path")
+        docs_path = _get_in(raw, "data.docs_source.path") if docs_source_type == "file" else None
         queries_path = _get_in(raw, "queries.source.path")
-        path_entries = (
-            ("data.docs_source.path", docs_path),
-            ("queries.source.path", queries_path),
-        )
+        path_entries = [("queries.source.path", queries_path)]
+        if docs_path is not None:
+            path_entries.append(("data.docs_source.path", docs_path))
         for label, p in path_entries:
             if isinstance(p, str):
                 fp = _resolve_input_path(changeset.path, p)