Layer extension: register custom layers without modifying OAP source

`Layer` becomes a registry-backed value type with the same public API as
the prior enum (`Layer.SERVICE`, `value()`, `name()`, `valueOf(int)`,
`valueOf(String)`, `nameOf(String)`, `values()`). Built-in layers stay as
`public static final Layer` constants with frozen ordinals 0–49.

External layers can be registered through any of three new paths, all
funneling through `Layer.register(name, ordinal, normal)`:

  - `layer-extensions.yml` on the OAP classpath / config dir — operator
    config; one line per layer, no code change required.
  - `LayerExtension` Java SPI under `META-INF/services/` — for plugin jars.
  - Inline `layerDefinitions:` block at the top of any MAL or LAL rule
    file — the cleanest path when a layer ships together with the rules
    that produce its telemetry.

Storage encoding is unchanged: layers persist by ordinal int in BanyanDB,
Elasticsearch, and JDBC. Built-in ordinals 0–49 stay frozen; 50–999 are
reserved by convention for future built-ins; external layers are
recommended to start at >= 1000. The recommendation is informational —
collisions are reported loudly at boot via the ordinal-uniqueness check.

The registry is sealed at the start of `Core.notifyAfterCompleted()` after
every module's prepare/start has run. Subsequent `Layer.register` calls
throw, so runtime-rule MAL/LAL `/addOrUpdate` requests now reject inline
`layerDefinitions:` with a clear error pointing operators at the boot-time
registration paths.

`UITemplateInitializer` no longer iterates `Layer.values()` for folder
discovery; it walks `ui-initialized-templates/**/*.json` recursively and
trusts each template's own `configuration.layer` field, so dashboards for
external layers are auto-discovered without code changes.

`Layer.values()` now throws before seal — pre-seal the registry may still
grow, and a partial snapshot would silently mislead callers. The cached
sorted snapshot is computed once at seal so post-seal calls are O(1) plus
an array clone.

BanyanDB `MetadataRegistry.parseTagSpec` previously relied on
`Class.isEnum()` to map Layer columns to TAG_TYPE_INT; with `Layer` no
longer an enum, that path now treats `Layer.class` explicitly, matching
the existing ES / JDBC handling.

Author: wu-sheng

.claude/skills/new-monitoring-feature/SKILL.md

@@ -21,11 +21,22 @@ There is one skill per narrow concern. This one is the wiring map.
 
 ## 0. Register the `Layer` — the feature's entry point
 
-A `Layer` is how OAP slices services / instances / endpoints by data source. **Every new feature needs a new `Layer` enum value.** The UI, storage partitioning, menu navigation, and OAL aggregation all key off it.
+A `Layer` is how OAP slices services / instances / endpoints by data source. The UI, storage partitioning, menu navigation, and OAL aggregation all key off it.
 
-**Only one place to edit** — `oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java`. Add a new enum constant with a unique id and `normal` flag. Ids are never reused; pick the next integer. Examples: `IOS(47, true)`, `APISIX(27, true)`, `VIRTUAL_DATABASE(11, false)` for inferred/non-real services.
+`Layer` is a registry-backed value type (no longer a closed enum). Built-in layers are declared as `public static final Layer` constants in `oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java`; external layers are registered through the `Layer.register(name, ordinal, normal)` API at boot. **Pick the registration path that matches the scope of your feature:**
 
-UI template folders are auto-discovered: `UITemplateInitializer.UI_TEMPLATE_FOLDER` is computed from `Layer.values()` + `"custom"` at class-init time. Drop a `ui-initialized-templates/<layer-name-lowercased>/` folder on disk and the initializer picks it up on the next boot. Missing folders are silently skipped. There is no allowlist to append to.
+| Your feature ships as | Registration path |
+|---|---|
+| Part of the OAP distribution (in-tree, the common case for new SkyWalking-supported targets) | Add a `public static final Layer` constant to `Layer.java` with the next sequential ordinal in `0–49`. Examples: `IOS = register("IOS", 47, true)`, `APISIX = register("APISIX", 21, true)`, `VIRTUAL_DATABASE = register("VIRTUAL_DATABASE", 14, false)` for inferred/non-real services. |
+| An out-of-tree MAL or LAL rule file | Add a top-level `layerDefinitions:` block to the rule file. The DSL loader funnels each entry through `Layer.register` before compiling the rule. One file ships the layer + the rules that produce its telemetry. |
+| An out-of-tree plugin module (jar) | Implement `org.apache.skywalking.oap.server.core.analysis.LayerExtension` and register via `META-INF/services/`. Discovered by `LayerExtensionLoader` during `CoreModuleProvider.prepare()`. |
+| Operator-deployed config (no code, no DSL) | Add an entry to `oap-server/server-starter/src/main/resources/layer-extensions.yml` (or override on the OAP node's classpath). |
+
+**Ordinal conventions:** `0–49` is in active use by built-ins. `50–999` is reserved by convention for future built-in layers. External layers are recommended (not required) to start at `>= 1000` to avoid colliding with future built-ins on OAP upgrade. Collisions in either direction are detected at boot via the ordinal-uniqueness check, which fails OAP startup loudly.
+
+**Storage encoding is the ordinal int**, persisted in BanyanDB / Elasticsearch / JDBC. Every OAP node that reads or writes a given layer must agree on its `(name, ordinal)` mapping — deploy `layer-extensions.yml` and any `layerDefinitions:` rule files identically across all nodes. The registry is sealed at the start of `Core.notifyAfterCompleted()`; later registration attempts throw.
+
+**UI template folders are auto-discovered by file scan, not by `Layer.values()`.** `UITemplateInitializer` walks `ui-initialized-templates/**/*.json` recursively (depth 2) and trusts each template's own `configuration.layer` field. Drop a folder of dashboard JSONs on disk and the initializer picks them up on the next boot — folder name is purely organizational.
 
 **Component ID lookup in Java code**: IDs declared in `component-libraries.yml` are loaded at runtime into `ComponentLibraryCatalogService`'s `componentName2Id` map — they are **not** exposed as Java enum constants. To look up by name in listener code, inject the catalog service and resolve once at construction:
 ```java
@@ -35,11 +46,14 @@ int myComponentId = catalog.getComponentId("My-Component-Name");
 ```
 Cache as an `int` field; runtime comparisons are then plain `componentId == myComponentId`. **Trap:** there is a `ComponentsDefine` class under `skywalking-trace-receiver-plugin/src/test/java/.../mock/ComponentsDefine.java` — it is a test-only mock holding five hand-picked constants (Tomcat, Dubbo, RocketMQ, MongoDB). Do not import or extend it from production code.
 
-Emit the layer from every source object your feature produces:
+Emit the layer from every source object your feature produces. Built-in layers have a static-field accessor; external layers are looked up by name through the registry:
+
 ```java
-service.setLayer(Layer.<YOUR_LAYER>);
-serviceInstance.setServiceLayer(Layer.<YOUR_LAYER>);
-endpoint.setServiceLayer(Layer.<YOUR_LAYER>);
+// Built-in layer (constant)
+service.setLayer(Layer.IOS);
+
+// External layer (registered via yaml / SPI / layerDefinitions:)
+service.setLayer(Layer.nameOf("IOT_FLEET"));
 ```
 
 Downstream (the core OAL, `service ly <LAYER>` swctl query, topology filters, UI root dashboard's layer selector) all work off this single enum value.

docs/en/changes/changes.md

@@ -97,6 +97,7 @@
 * MAL: add `safeDiv(divisor)` on `SampleFamily` that yields `0` when the divisor is `0` instead of `Infinity`/`NaN`. Replace `/` with `safeDiv(...)` in Envoy AI Gateway latency-average rules so `sum / count * 1000` no longer produces dropped or out-of-range samples when a counter is zero in a window.
 * Fix: `envoy-ai-gateway` metrics rules, make the metrics value return `0` when the divisor is `0`.
 * Fix: LAL compiler treated `(tag("x") as Integer) + (tag("y") as Integer)` as string concatenation instead of numeric addition. Expressions like `input_tokens + output_tokens < 10000` produced the concatenated string `"2589115"` rather than the integer sum `2704`, so token-threshold conditions never triggered `abort {}`. The compiler now detects all-numeric operands (cast to `Integer` or `Long`) and emits proper `long` arithmetic.
+* Custom `Layer`s can be declared without modifying the OAP source — via an operator-managed `layer-extensions.yml`, inline `layerDefinitions:` block in a MAL or LAL rule file, or a plugin extension. UI dashboard templates for new layers are auto-discovered from the `ui-initialized-templates/` directory. Recommended ordinal range for external layers is `>= 1000`; conflicting names or ordinals are reported at boot.
 
 #### UI
 * Add mobile menu icon and i18n labels for the iOS layer.

docs/en/concepts-and-designs/lal.md

@@ -30,6 +30,51 @@ Use `tag 'key': sourceAttribute("attr")` in the extractor to selectively persist
 Layer should be declared in the LAL script to represent the analysis scope of the logs.
 LAL rules are routed by layer — only rules matching the incoming log's layer are evaluated.
 
+### Inline layer declarations (`layerDefinitions:`)
+
+A LAL file may declare its own custom layers with a top-level `layerDefinitions:` block.
+Each entry is funneled through `Layer.register(name, ordinal, normal)` **before**
+the rules in the same file compile, so a LAL file is fully self-describing — a new
+monitoring target can land as a single LAL file without an enum edit elsewhere in the OAP
+source.
+
+```yaml
+layerDefinitions:
+  - name: IOT_FLEET    # upper-snake-case, must match [A-Z][A-Z0-9_]*
+    ordinal: 1000      # unique across all layers; >= 1000 recommended
+    normal: true       # true = agent-installed (default), false = conjectured/virtual
+
+rules:
+  - name: iot-fleet-access
+    layer: IOT_FLEET
+    dsl: |
+      filter {
+        text { regexp $/(?<status>\d+)\s+(?<path>\S+)/$ }
+        sink { sampler { rateLimit { rpm 1800 } } }
+      }
+```
+
+Notes:
+- **Storage encoding is the ordinal int**, persisted in BanyanDB / Elasticsearch / JDBC.
+  Every OAP node that reads or writes a given layer must agree on its `(name, ordinal)`
+  mapping — deploy a LAL file with `layerDefinitions:` identically across all nodes.
+- **Identical re-registration is a no-op**, so the same `IOT_FLEET` entry can appear in
+  multiple LAL files (and additionally in a MAL file, in `layer-extensions.yml`, or via the
+  `LayerExtension` SPI). Conflicting registrations cause OAP boot to fail loudly with the
+  offending file in the stack trace.
+- **Ordinals 0–49** are in active use by the OAP distribution's built-in layers; **50–999**
+  are reserved by convention for future built-ins. External layers should start at `>= 1000`
+  — enforcement is not strict, but staying above the reserved band avoids upgrade-time
+  collisions.
+- `layer: auto` works with extension layers too — the extractor body can call
+  `layer "IOT_FLEET"` and the runtime resolves it through the registry.
+
+Three other registration paths exist for layers that are **not** specific to a LAL file: an
+operator-managed `layer-extensions.yml`, a `LayerExtension` Java SPI for plugin jars, and
+the built-in static fields in `Layer.java` for distribution layers. See
+[`Layer.java`](../../../oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java)
+javadoc for the full picture.
+
 When `layer: auto` is declared, the rule matches logs where `service.layer` is absent (common for OTLP
 sources that don't set this attribute). The script is expected to set the layer in the extractor:
 

docs/en/concepts-and-designs/mal.md

@@ -377,6 +377,44 @@ name: <string>
 exp: <string>
 ```
 
+### <layer_definitions>
+
+Optional top-level block for declaring custom layers inline alongside the rules that produce
+their telemetry. Each entry is funneled through `Layer.register(name, ordinal, normal)`
+**before** the rules in the same file compile, so a MAL file is fully self-describing — a new
+monitoring target can land as a single MAL file without an enum edit elsewhere in the OAP source.
+
+```yaml
+layerDefinitions:
+  - name: IOT_FLEET    # upper-snake-case, must match [A-Z][A-Z0-9_]*
+    ordinal: 1000      # unique across all layers; >= 1000 recommended
+    normal: true       # true = agent-installed (default), false = conjectured/virtual
+
+metricsRules:
+  - name: device_battery_percentage
+    exp: iot_device_battery_level.tagAverage(['service'], ['host'])
+expSuffix: instance(['host'], ['service'], Layer.nameOf('IOT_FLEET'))
+```
+
+Notes:
+- **Storage encoding is the ordinal int**, persisted in BanyanDB / Elasticsearch / JDBC. Every
+  OAP node that reads or writes a given layer must agree on its `(name, ordinal)` mapping —
+  deploy a MAL file with `layerDefinitions:` identically across all nodes.
+- **Identical re-registration is a no-op**, so the same `IOT_FLEET` entry can appear in multiple
+  MAL files (and additionally in a LAL file, in `layer-extensions.yml`, or via the
+  `LayerExtension` SPI). Conflicting registrations (same name with different ordinal, or same
+  ordinal with different name) cause OAP boot to fail loudly with the offending file in the
+  stack trace.
+- **Ordinals 0–49** are in active use by the OAP distribution's built-in layers; **50–999** are
+  reserved by convention for future built-ins. External layers should start at `>= 1000` —
+  enforcement is not strict, but staying above the reserved band avoids upgrade-time collisions.
+
+Three other registration paths exist for layers that are **not** specific to a MAL file: an
+operator-managed `layer-extensions.yml`, a `LayerExtension` Java SPI for plugin jars, and the
+built-in static fields in `Layer.java` for distribution layers. See
+[`Layer.java`](../../../oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java)
+javadoc for the full picture.
+
 ## More Examples
 
 Please refer to [OAP Self-Observability](../../../oap-server/server-starter/src/main/resources/otel-rules/oap.yaml).

oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/provider/LALConfigs.java

@@ -33,6 +33,8 @@
 import java.util.Map;
 import lombok.Data;
 import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oap.server.core.UnexpectedException;
+import org.apache.skywalking.oap.server.core.analysis.LayerDefinition;
 import org.apache.skywalking.oap.server.core.rule.ext.RuleSetMerger;
 import org.apache.skywalking.oap.server.library.module.ModuleManager;
 import org.apache.skywalking.oap.server.library.module.ModuleStartException;
@@ -48,6 +50,12 @@
 @Slf4j
 public class LALConfigs {
     private List<LALConfig> rules;
+    /**
+     * Optional inline layer registrations. When present, each entry is registered through
+     * {@code Layer.register(...)} before the rules in this file are compiled, so a
+     * LAL file is self-describing for any custom layers it references.
+     */
+    private List<LayerDefinition> layerDefinitions;
 
     public static List<LALConfigs> load(final String path, final List<String> files) throws Exception {
         return loadInternal(path, files, null, /* useInstalledManager= */ true);
@@ -127,6 +135,7 @@ private static List<LALConfigs> loadInternal(final String path, final List<Strin
                     if (configs == null || configs.getRules() == null) {
                         continue;
                     }
+                    registerInlineLayers(ruleName, configs);
                     // sourceFileName is only present for entries that came from disk; resolver-
                     // only rules synthesise a name so diagnostics still print something.
                     final String src = sourceFileName.getOrDefault(ruleName, ruleName + ".yaml");
@@ -141,4 +150,25 @@ private static List<LALConfigs> loadInternal(final String path, final List<Strin
             throw new ModuleStartException("Failed to load LAL config rules", e);
         }
     }
+
+    /**
+     * Funnel any inline {@code layerDefinitions:} entries through {@code Layer.register}.
+     * Conflict checks (reserved-range, name uniqueness, ordinal uniqueness, sealed-state) live
+     * in {@code Layer.register}; failures here surface with the offending rule name in
+     * the stack trace, which is enough for an operator to find the bad file.
+     */
+    private static void registerInlineLayers(final String ruleName, final LALConfigs configs) {
+        final List<LayerDefinition> defs = configs.getLayerDefinitions();
+        if (defs == null || defs.isEmpty()) {
+            return;
+        }
+        for (final LayerDefinition def : defs) {
+            try {
+                def.register();
+            } catch (RuntimeException e) {
+                throw new UnexpectedException(
+                    "LAL rule " + ruleName + " layerDefinitions entry rejected: " + def, e);
+            }
+        }
+    }
 }

oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/Rule.java

@@ -21,6 +21,7 @@
 import lombok.Data;
 import lombok.NoArgsConstructor;
 import org.apache.skywalking.oap.meter.analyzer.v2.MetricRuleConfig;
+import org.apache.skywalking.oap.server.core.analysis.LayerDefinition;
 
 import java.util.List;
 
@@ -36,6 +37,12 @@ public class Rule implements MetricRuleConfig {
     private String expPrefix;
     private String filter;
     private List<MetricsRule> metricsRules;
+    /**
+     * Optional inline layer registrations. When present, each entry is registered through
+     * {@code Layer.register(...)} before the rule's expressions compile, so the
+     * rule file is self-describing for any custom layers it references.
+     */
+    private List<LayerDefinition> layerDefinitions;
 
     @Override
     public String getSourceName() {

oap-server/analyzer/meter-analyzer/src/main/java/org/apache/skywalking/oap/meter/analyzer/v2/prometheus/rule/Rules.java

@@ -37,6 +37,7 @@
 import java.util.stream.Stream;
 
 import org.apache.skywalking.oap.server.core.UnexpectedException;
+import org.apache.skywalking.oap.server.core.analysis.LayerDefinition;
 import org.apache.skywalking.oap.server.core.rule.ext.RuleSetMerger;
 import org.apache.skywalking.oap.server.library.module.ModuleManager;
 import org.apache.skywalking.oap.server.library.util.ResourceUtils;
@@ -149,9 +150,31 @@ private static Rule parseRule(final String ruleName, final byte[] bytes) {
                 return null;
             }
             rule.setName(ruleName);
+            registerInlineLayers(ruleName, rule);
             return rule;
         } catch (IOException e) {
             throw new UnexpectedException("Load rule " + ruleName + " failed", e);
         }
     }
+
+    /**
+     * Funnel any inline {@code layerDefinitions:} entries through {@code Layer.register}.
+     * Conflict checks (reserved-range, name uniqueness, ordinal uniqueness, sealed-state) live
+     * in {@code Layer.register}; failures here surface with the offending rule name in
+     * the stack trace, which is enough for an operator to find the bad file.
+     */
+    private static void registerInlineLayers(final String ruleName, final Rule rule) {
+        final List<LayerDefinition> defs = rule.getLayerDefinitions();
+        if (defs == null || defs.isEmpty()) {
+            return;
+        }
+        for (final LayerDefinition def : defs) {
+            try {
+                def.register();
+            } catch (RuntimeException e) {
+                throw new UnexpectedException(
+                    "MAL rule " + ruleName + " layerDefinitions entry rejected: " + def, e);
+            }
+        }
+    }
 }

oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/CoreModuleProvider.java

@@ -25,6 +25,8 @@
 import org.apache.skywalking.oap.server.core.analysis.ApdexThresholdConfig;
 import org.apache.skywalking.oap.server.core.rule.ext.RuleSetMerger;
 import org.apache.skywalking.oap.server.core.analysis.DisableRegister;
+import org.apache.skywalking.oap.server.core.analysis.Layer;
+import org.apache.skywalking.oap.server.core.analysis.LayerExtensionLoader;
 import org.apache.skywalking.oap.server.core.analysis.StreamAnnotationListener;
 import org.apache.skywalking.oap.server.core.analysis.meter.MeterEntity;
 import org.apache.skywalking.oap.server.core.analysis.meter.MeterSystem;
@@ -184,6 +186,12 @@ public void onInitialized(final CoreModuleConfig initialized) {
 
     @Override
     public void prepare() throws ServiceNotProvidedException, ModuleStartException {
+        // Load external Layer registrations (yaml + SPI) before anything else in Core
+        // wires up. Downstream modules' prepare()/start() — including MAL/LAL DSL loaders
+        // that may declare inline `layerDefinitions:` blocks — register on top of this
+        // baseline. The registry is sealed at the start of notifyAfterCompleted().
+        LayerExtensionLoader.load();
+
         if (moduleConfig.isActiveExtraModelColumns()) {
             DefaultScopeDefine.activeExtraModelColumns();
         }
@@ -473,6 +481,12 @@ public void start() throws ModuleStartException {
 
     @Override
     public void notifyAfterCompleted() throws ModuleStartException {
+        // Seal the Layer registry: every module's prepare() and start() has now run, so
+        // MAL/LAL/SPI/yaml all had their full window to register external layers. After
+        // this point, Layer.register() throws — UI templates and downstream
+        // queries see a stable layer set.
+        Layer.seal();
+
         try {
             if (!RunningMode.isInitMode()) {
                 grpcServer.start();