Runtime rule hot-update for MAL and LAL

Add a runtime-rule receiver that exposes a REST surface on port 17128 (off
by default) so operators can hot-update OTEL MAL, log MAL, telegraf MAL,
and LAL rule files without restarting OAP. State persists in management
storage; every node in an OAP cluster converges on a new content within
~30 s.

REST endpoints (full reference in
docs/en/setup/backend/backend-runtime-rule-api.md):
- POST  /runtime/rule/addOrUpdate   create or replace
- POST  /runtime/rule/inactivate    soft-pause (preserves measure + history)
- POST  /runtime/rule/delete        destructive removal of an INACTIVE rule
- GET   /runtime/rule               fetch one rule (raw or JSON, ETag)
- GET   /runtime/rule/bundled       static-vs-runtime overlay per catalog
- GET   /runtime/rule/list          NDJSON rule state across the cluster
- GET   /runtime/rule/dump          tar.gz of all stored rules

Engine, cluster + storage:
- New management entity RuntimeRule (catalog,name,content,status,updateTime)
  with per-backend RuntimeRuleManagementDAO save/find/list/delete.
- DSLManager + RuleEngine SPI orchestrate compile / verify / commit /
  rollback per file; MAL and LAL each ship an engine, with a shared
  apply pipeline (StructuralCommitCoordinator, SuspendResumeCoordinator,
  ApplierResolver, PostApplyVerifier, alarm-window reset).
- Cluster Suspend/Resume RPC and Forward RPC over the cluster bus so
  any peer can drive a structural cutover that pauses dispatch on every
  node, persists the rule, then resumes — with a 60 s self-heal backstop
  for missed Resumes.
- LOCAL_CACHE_VERIFY mode: non-main nodes verify backend shape on boot
  and refuse to start when their declared model diverges from what the
  main installed, instead of silently registering against an
  incompatible schema.
- Storage-model remove lifecycle: StorageModels.remove + per-backend
  dropTable so a runtime-rule delete can drop the backing measure on
  BanyanDB and the model on every backend without restarting.
- BanyanDB schema-watch fence: schema mutations wait (best-effort,
  bounded 2 s) for every data node to apply the new revision before
  unparking dispatch, so the typical case gets a clean cutover where
  samples after 200 OK use the new shape.
- BanyanDB also gains shape-mismatch detection: at boot, resources whose
  on-disk shape diverges from the declared model are skipped with an
  ERROR diff, instead of silently dropping samples.

Bundled vs runtime overlay:
- StaticRuleRegistry holds bundled rules; the merge resolver lets a
  runtime rule override a bundled (catalog,name) without rewriting
  static files. RuntimeRuleOverrideResolver SPI threads operator-side
  overrides into the bundle resolution.

E2E coverage under test/e2e-v2/cases/runtime-rule/:
- mal-storage/{banyandb,elasticsearch,postgresql} — full 10-phase
  lifecycle (CREATE → FILTER_ONLY → STRUCTURAL → DUMP → 4× ILLEGAL
  → SHAPE-BREAK → INACTIVATE → ACTIVATE → DELETE → DUMP) with a
  per-phase `step` label so verification queries attribute data
  back to the phase that wrote it.
- lal — log-mal aggregation rule + LAL hot-swap, swctl asserts that
  the extracted metric carries the swap-flipped step label.
- cluster — 2-OAP convergence over ZooKeeper.

Dependency bumps (driven by BanyanDB schema-consistency RPCs whose
generated validation code requires protobuf-java 4.x):
gRPC 1.70 → 1.80, protobuf-java 3.25.5 → 4.33.1, pgv 1.2.1 → 1.3.0,
Netty 4.2.10 → 4.2.12, Netty-tcnative 2.0.75 → 2.0.77.

Security: the admin port has no built-in authentication; the module is
disabled by default. docs/en/security/README.md spells out the operator
duty to gateway-protect with IP allow-lists, audit every request, and
keep the port off the cluster-external interface.

Author: wu-sheng

.claude/skills/gh-pull-request/SKILL.md

@@ -32,6 +32,46 @@ license-eye header check
 
 If invalid files are found, fix with `license-eye header fix` and re-check.
 
+### 3. Unnecessary fully-qualified class names
+
+The project checkstyle forbids inline FQCNs — every type reference in code should resolve
+through an `import`, not a fully-qualified name. Checkstyle does not always catch this (it
+misses cases like inline `java.util.HashMap`, `java.util.concurrent.TimeUnit`, or
+`org.apache.skywalking.oap.server.telemetry.api.HistogramMetrics.Timer` used as a local
+variable type, generic parameter, or `new` target). Audit the files the branch touched
+before pushing:
+
+Use the `Grep` tool (ripgrep) rather than BSD `grep` on macOS — the scan below relies on a
+negative lookahead that BSD `grep` doesn't support and GNU `grep -P` does:
+
+```
+pattern: ^(?!\s*(import |package |\s*\*)).*\b(java\.util\.|java\.io\.|java\.nio\.|java\.util\.concurrent\.|javassist\.|org\.apache\.skywalking\.)[A-Z][A-Za-z0-9_]*
+glob:    *.java
+output_mode: content
+-n: true
+```
+
+Scope the scan to files the branch touched, not the whole tree — pre-existing FQDNs on
+unrelated files generate noise. Use `git diff --name-only master...HEAD -- '*.java'` to get
+the changed list, then run the ripgrep pattern against each.
+
+Acceptable exceptions (same as the `CLAUDE.md` rule):
+ - Two classes with the same simple name would collide if both imported.
+ - A Javadoc `{@link}` where the short name would be ambiguous to the reader.
+ - Inside a string literal (e.g., a class name passed to `Class.forName`).
+
+Fix every other hit — add an `import` and switch to the short name. This includes
+`new java.util.HashMap<>()`, `java.util.Set<String>` parameter types, and
+`org.apache.skywalking.oap.server.telemetry.api.HistogramMetrics.Timer` as a local
+variable type. Field declarations, method signatures, local variables, and generic
+type arguments should all use the imported short name.
+
+Re-run checkstyle after the fix — a sloppy `sed`/`replace_all` can corrupt the `import`
+line itself (e.g., turning `import java.util.concurrent.locks.ReentrantLock;` into
+`import ReentrantLock;`), which causes a cryptic checkstyle `Range [0, -1) out of
+bounds for length N` error, not a normal violation line. If you see that error, inspect
+the imports block first.
+
 ## Commit and push
 
 After checks pass, commit and push:

.github/workflows/skywalking.yaml

@@ -294,7 +294,9 @@ jobs:
           distribution: temurin
       - name: Integration test
         run: |
-          # Exclude slow integration tests and run those tests separately below.
+          # Exclude slow integration tests (run in slow-integration-test). Runtime-rule
+          # and BanyanDB storage CRUD are verified end-to-end in the dedicated e2e cases
+          # (see test/e2e-v2/cases/runtime-rule/ and test/e2e-v2/cases/banyandb).
           ./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true -DexcludedGroups=slow || \
           ./mvnw -B clean integration-test -Dcheckstyle.skip -DskipUTs=true -DexcludedGroups=slow
 
@@ -394,6 +396,18 @@ jobs:
             config: test/e2e-v2/cases/storage/es/es-sharding/e2e.yaml
             env: ES_VERSION=8.18.8
 
+          - name: Runtime Rule MAL Storage BanyanDB
+            config: test/e2e-v2/cases/runtime-rule/mal-storage/banyandb/e2e.yaml
+          - name: Runtime Rule MAL Storage PostgreSQL
+            config: test/e2e-v2/cases/runtime-rule/mal-storage/postgresql/e2e.yaml
+          - name: Runtime Rule MAL Storage Elasticsearch 8.18.8
+            config: test/e2e-v2/cases/runtime-rule/mal-storage/elasticsearch/e2e.yaml
+            env: ES_VERSION=8.18.8
+          - name: Runtime Rule LAL Hot-Update
+            config: test/e2e-v2/cases/runtime-rule/lal/e2e.yaml
+          - name: Runtime Rule Cluster Convergence
+            config: test/e2e-v2/cases/runtime-rule/cluster/e2e.yaml
+
           - name: Alarm ES
             config: test/e2e-v2/cases/alarm/es/e2e.yaml
           - name: Alarm ES Sharding

.gitignore

@@ -43,4 +43,5 @@ test/script-cases/scripts/**/*.generated-classes/
 
 # Claude Code local settings
 .claude/settings.local.json
+.claude/scheduled_tasks.lock
 *.generated-classes/

CLAUDE.md

@@ -90,6 +90,17 @@ public class XxxModuleProvider extends ModuleProvider {
 - No star imports (`import xxx.*`)
 - No unused or redundant imports
 - No empty statements (standalone `;`)
+- No fully-qualified class names inline in code — always add an `import` statement and
+  use the short name. Acceptable exceptions: (a) two classes with the same simple name
+  would collide if both imported, (b) the class appears exactly once in a Javadoc
+  `{@link}` where the short name would be ambiguous to the reader. Field declarations,
+  method signatures, local variables, and generic type arguments should always use the
+  imported short name — `private RemoteClientManager rcm;`, not `private
+  org.apache.skywalking.oap.server.core.remote.client.RemoteClientManager rcm;`.
+- No one-line delegate methods. A wrapper whose only body is a single forwarding call
+  to another class (`return Other.foo(a, b);`) adds a hop without value. Inline the
+  call at the use site, or call the underlying object directly (including via method
+  reference: `obj::foo` instead of `this::wrapperOfFoo`).
 
 **Required patterns:**
 - `@Override` annotation required for overridden methods
@@ -105,6 +116,13 @@ public class XxxModuleProvider extends ModuleProvider {
 - Package names: `org.apache.skywalking.*` or `test.apache.skywalking.*`
 - Type names: `PascalCase` or `UPPER_CASE_WITH_UNDERSCORES`
 - Local variables/parameters/members: `camelCase`
+- **Function-oriented naming, not abstract metaphor**: classes and methods are named for
+  what they do, not for an abstract concept. Prefer concrete verbs (`load`, `apply`,
+  `unregister`, `compile`, `verify`, `commit`, `rollback`) over metaphorical ones
+  (`seed`, `hydrate`, `bootstrap`, `prime`). Class names follow the same rule —
+  `StaticRuleLoader` (loads static rules), not `StaticBundleSeeder`; `DSLSyncTimer` (syncs
+  DB → state on a timer), not `TickRunner`. If you can't name a method without reaching
+  for a metaphor, the method is probably doing too much; split it.
 
 **File limits:**
 - Max file length: 3000 lines
@@ -257,6 +275,10 @@ Actions owned by `actions/*` (GitHub), `github/*`, and `apache/*` are always all
 10. **Relative paths in docs are valid**: Relative file paths (e.g., `../../../oap-server/...`) in documentation work both in the repo and on the documentation website, supported by website build tooling
 11. **Module service registration**: When adding a service to `CoreModule.services()`, update ALL `CoreModuleProvider` implementations — not just the main one. Search with `grep -rn "extends CoreModuleProvider" oap-server/ --include="*.java"`. The `MockCoreModuleProvider` in `server-tools/profile-exporter/` also needs it, or the profile exporter e2e test will fail at startup.
 12. **Multiple OAP packagings**: The OAP server is not only the main `server-starter`. The `server-tools/` directory contains standalone tools (e.g., profile exporter) that boot with mock module providers and a subset of modules. Changes to core module contracts (services, required modules) must be reflected in these tools too.
+13. **`moduleManager.find(X.NAME)` requires `X.NAME` in `requiredModules()`**: every call to `moduleManager.find(SomeModule.NAME)` (direct or through a helper) must have `SomeModule.NAME` in the provider's `requiredModules()` array. Missing declarations cause runtime exceptions the first time the code path fires — not at module boot. Wrapping the call in `try { ... } catch (Throwable)` is NOT a substitute; declare the module and keep the try/catch only for defensive handling of transient provider outages. When auditing a branch, grep for `moduleManager.find(` across the touched module and verify each target name appears in `requiredModules()`. Example modules that frequently catch teams out: `AlarmModule` (used by alarm-kernel reset), `LogAnalyzerModule` (used by LAL factory lookup).
+14. **Don't look up `ClusterModule` services directly**: the `ClusterModule` (ZooKeeper / K8s / Nacos coordination) exposes `ClusterRegister` / `ClusterNodesQuery` / `ClusterCoordinator`. Most receiver / analyzer modules don't declare `ClusterModule` in `requiredModules()`, so calling `moduleManager.find(ClusterModule.NAME)` will throw at runtime. Instead, go through `CoreModule`'s `RemoteClientManager` service — it's already populated by the cluster module and exposes the peer list every OAP needs. If a module genuinely needs cluster-coordinator primitives, declare `ClusterModule.NAME` in `requiredModules()` explicitly.
+15. **No `ThreadLocal` side-channels to hijack downstream behaviour**: routing a caller's intent through a `ThreadLocal` that downstream code reads (e.g., `if (PeerMode.isActive()) skipSomething()`) is almost always the wrong answer — it creates invisible coupling between far-apart code paths, leaks across async hand-offs (executors, gRPC threads, Armeria event loops), and makes the behaviour impossible to understand from a method signature. The correct fix is almost always to **extend the interface** — add a parameter, a new method, a new mode enum that appears in the signature. Rare exceptions: propagating OpenTelemetry context where the whole industry has standardised on `ThreadLocal`, or security principals enforced by a framework. In all other cases, prefer an explicit API extension, even if it costs more lines.
+16. **BanyanDB schema-visibility: fence on `mod_revision`, do NOT poll metadata**: every BanyanDB Create / Update / Delete returns an etcd `mod_revision` (0 on a delete that didn't record a tombstone). After firing DDL, fence on `BanyanDBClient.getSchemaWatcher().awaitRevisionApplied(maxRev, timeout)` before unparking dispatch / firing data writes — this blocks until every data node has caught up, which the registry's read-after-write does not guarantee. For deletes that returned `mod_revision == 0`, fall back to `awaitSchemaDeleted(SchemaKey, timeout)`. The previous "poll `findMeasure` until you can read your own write" idiom existed before the `SchemaBarrierService` proto landed and has been replaced — do not reintroduce it. JDBC and ES are synchronous-DDL on the coordinator so they don't need a fence.
 
 ## Analysis and Design Principles
 

apm-protocol/apm-network/pom.xml

@@ -92,11 +92,11 @@
                       protobuf-java version that grpc depends on.
                     -->
                     <protocArtifact>
-                        com.google.protobuf:protoc:${com.google.protobuf.protoc.version}:exe:${os.detected.classifier}
+                        com.google.protobuf:protoc:${protobuf-java.version}:exe:${os.detected.classifier}
                     </protocArtifact>
                     <pluginId>grpc-java</pluginId>
                     <pluginArtifact>
-                        io.grpc:protoc-gen-grpc-java:${protoc-gen-grpc-java.plugin.version}:exe:${os.detected.classifier}
+                        io.grpc:protoc-gen-grpc-java:${grpc.version}:exe:${os.detected.classifier}
                     </pluginArtifact>
                 </configuration>
                 <executions>

docker/.env

@@ -6,6 +6,6 @@
 # docker compose up
 
 ELASTICSEARCH_IMAGE=docker.elastic.co/elasticsearch/elasticsearch-oss:7.4.2
-BANYANDB_IMAGE=ghcr.io/apache/skywalking-banyandb:7568a326bb7b10b6aa804bf0f4239904c347d9d5
+BANYANDB_IMAGE=ghcr.io/apache/skywalking-banyandb:69c8f4d20ebb6532ea4c16a7ed7114dd6ec9770b
 OAP_IMAGE=ghcr.io/apache/skywalking/oap:latest
 UI_IMAGE=ghcr.io/apache/skywalking/ui:latest

docker/oap/docker-entrypoint.sh

@@ -1,3 +1,4 @@
+
 #!/bin/sh
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements.  See the NOTICE file

docs/en/changes/changes.md

@@ -1,8 +1,46 @@
 ## 10.5.0
 
 #### Project
+
+* **Runtime rule hot-update for MAL and LAL.** Operators can now ship metric (MAL) and log
+  (LAL) rule changes without restarting OAP. A push to a new admin endpoint persists the rule
+  to the configured storage backend, and every node in the cluster converges to the new
+  content within ~30 seconds. Common workflows:
+  * `addOrUpdate` — create or replace a rule. Body is the raw YAML you would normally ship
+    with OAP's static rule files. Returns 200 once the rule is applied locally and
+    persisted; peers pick it up on their next periodic scan (≤ 30 s).
+  * `inactivate` — soft-pause a rule. The OAP stops emitting metrics for that rule but the
+    backend measure (and its history) is preserved, so a later `addOrUpdate` to the same
+    `(catalog, name)` is lossless. This is the safe way to take a rule offline.
+  * `delete` — destructive removal. Requires the rule to already be `INACTIVE` (otherwise
+    returns 409); drops the backend measure and removes the row.
+  * `get` / `bundled` / `list` / `dump` — read-side endpoints for fetching a single rule's
+    YAML (with `ETag` support), listing the static-vs-runtime overlay per catalog,
+    inspecting cluster-wide rule state in NDJSON, and exporting all rules as a tar.gz for
+    backup / DR.
+  Hot-updates survive OAP restart: at boot OAP merges bundled rule files with persisted
+  runtime rules, so the cluster never silently regresses to the bundled defaults.
+  **The endpoint is disabled by default and listens on port `17128` when enabled. It has
+  no built-in authentication — operators must gateway-protect it with IP allow-lists and
+  never expose it to the public internet.**
+* **BanyanDB schema mismatches are now visible at boot, not silent.** If BanyanDB already
+  holds a resource whose shape doesn't match what the current rule declares (e.g., a rule
+  was edited on disk while OAP was offline), OAP now skips that resource, logs an ERROR
+  with the declared-vs-backend diff, and continues booting — previously the mismatch was
+  silently accepted and samples for the affected resource were quietly dropped. To
+  re-shape a mismatched metric, push the desired YAML through
+  `POST /runtime/rule/addOrUpdate`.
 * Bump infra-e2e to testcontainers-go v0.42.0 (apache/skywalking-infra-e2e#146), which uses Docker Compose v2 plugin natively and removes docker-compose v1 dependency.
 * Remove deprecated `version` field from all docker-compose files for Compose v2 compatibility.
+* **Best-effort schema-cutover fence for BanyanDB.** After firing a schema install or drop
+  OAP now waits up to a bounded window (default 2s) for every BanyanDB data node to apply
+  the change before resuming dispatch — the typical case gets a clean cutover where
+  samples after `200 OK` use the new shape. On laggard timeout, OAP logs a warning and
+  proceeds anyway so a single slow node doesn't wedge the apply.
+* Bump dependencies: gRPC `1.70.0` → `1.80.0`, protobuf-java `3.25.5` → `4.33.1`, Netty
+  `4.2.10.Final` → `4.2.12.Final`, Netty-tcnative `2.0.75` → `2.0.77`, pgv (protoc-gen-validate)
+  `1.2.1` → `1.3.0`. Driven by the new BanyanDB schema-consistency RPCs whose generated
+  validation code requires the `protobuf-java 4.x` runtime.
 
 #### OAP Server
 * Add Zipkin Virtual GenAI e2e test. Use `zipkin_json` exporter to avoid protobuf dependency conflict
@@ -54,4 +92,3 @@
 * Add WeChat / Alipay Mini Program monitoring setup documentation, plus a client-side-monitoring section in the security guide covering public-internet ingress (OTLP + `/v3/segments`) for mobile / browser / mini-program SDKs.
 
 All issues and pull requests are [here](https://github.com/apache/skywalking/issues?q=milestone:10.5.0)
-