fix(agent): tolerate empty system_info in heartbeat JSON (v1.8.3)

A self-hosted agent could fail every heartbeat with HTTP 400
`Invalid JSON: Expecting property name enclosed in double quotes` when the
system-info block it collects came back empty on an unusual host. The agent
builds the heartbeat JSON as text, so an empty `$system_info` collapsed the
`$system_info,` line to a bare comma and broke the payload.

- Agent script (linux + macos, kept in sync): guard the fragment-form
  register_agent and send_heartbeat builders so an empty system_info falls back
  to a valid key and can never emit a bare comma. Uses the most portable bash
  glob test (no POSIX class / pattern substitution; verified on bash 3.2-5.2 and
  on Ubuntu/Debian/Rocky/Alpine/Amazon Linux). True no-op for healthy agents.
- Backend heartbeat endpoint: parse the body as-is first and only run the
  malformed-JSON repair when parsing fails, so a valid heartbeat from any agent
  version is byte-for-byte untouched. The repair (now a testable helper) recovers
  a leading or doubled comma (the empty-system_info artifact) in addition to the
  existing empty-value / trailing-comma fixes.

No agent version bump; self-upgrade and daemon mode are unaffected. Healthy
agents of every version behave identically. Full backend suite green.

Addresses #31.

Author: taylanbakircioglu

README.md

@@ -2415,6 +2415,7 @@ Developed with ❤️ for the HAProxy community
 
 ## Release Notes
 
+- **v1.8.3** (2026-06-25) — **Agent heartbeat JSON fix** (Issue #31): a self-hosted agent could fail every heartbeat with `HTTP 400 Invalid JSON: Expecting property name enclosed in double quotes` when the system-info block it collects came back empty on an unusual host, leaving a stray comma in the hand-built heartbeat JSON. The agent script now substitutes a valid placeholder when that block is empty so it can no longer emit a stray comma, and the backend heartbeat endpoint now parses valid payloads as-is and, only when a body fails to parse, tolerates that specific malformed pattern (a leading or doubled comma) so an already-deployed agent recovers on its next heartbeat after this build is deployed. Backend + agent-script only; healthy agents of every version are byte-for-byte unaffected.
 - **v1.8.2** (2026-06-25) — **ACME nonce fix** (Issue #35 follow-up): the ACME client now scopes the anti-replay nonce **per certificate authority** so a nonce issued by one CA is never sent to another. This fixes ZeroSSL/Google account registration failing with `malformed: The Replay Nonce could not be base64url-decoded` (the client previously shared one nonce across CAs and only auto-retried on `badNonce`). Account registration now always uses a fresh nonce from the target CA, and the retry covers this case too. Backend-only; HTTP-01 and Let's Encrypt are unaffected.
 - **v1.8.1** (2026-06-24) — **ACME DNS-01 fixes** (Issue #35 follow-up): Cloudflare API tokens are now sanitized so a pasted token with quotes/spaces no longer fails with "Invalid request headers"; ZeroSSL/Google **External Account Binding (EAB)** can be entered per-account in the register dialog and EAB-required failures show a clear message; and **Apply Management** now categorizes cluster ACME enable/disable changes under their own "ACME Challenge Routing" section and **Apply/Reject All** correctly process them (previously "Rejected 0 HA/VIP change(s)"), consistent with every other entity. Fully backward compatible.
 - **v1.8.0** (2026-06-23) — **ACME DNS-01 challenge support** (Issue #35): Auto SSL can now validate via a **DNS TXT record** (`_acme-challenge.<domain>`) instead of HTTP-01 on port 80, enabling certificates for **internal/isolated clusters with no public ingress** and **wildcard** certificates (`*.example.com`). Pluggable **per-account DNS provider** (Manual + Cloudflare to start; credentials verified on save and **encrypted at rest**, never returned by the API or logged), the same **PENDING → APPLIED** pipeline, a **bounded automatic retry** on propagation lag, and a **DNS-01 event timeline** in the order detail. **Opt-in** via Settings → ACME (global switch, default off); **HTTP-01 is byte-for-byte unchanged**, with **zero agent or rendered-config changes**. Manual DNS-01 certificates cannot auto-renew unattended; the UI states this and disables auto-renew for them.

backend/main.py

@@ -9,7 +9,7 @@
 from datetime import datetime, timedelta
 
 # Build/deploy marker for the v1.8.x (Issue #35, DNS-01) rollout — ensures the pipeline ships this commit's image.
-_version_info = {"version": "1.8.2", "releaseName": "ACME nonce fix (ZeroSSL registration)", "releaseDate": "2026-06-25"}
+_version_info = {"version": "1.8.3", "releaseName": "Agent heartbeat JSON fix", "releaseDate": "2026-06-25"}
 for _vpath in ["/app/version.json", os.path.join(os.path.dirname(__file__), "..", "version.json")]:
     try:
         with open(_vpath) as _vf:

backend/routers/agent.py

@@ -29,6 +29,40 @@
     "linux": "2.0.0"
 }
 
+
+def _sanitize_agent_json(body_str: str):
+    """Repair the common malformed-JSON patterns a hand-built agent heartbeat can emit.
+
+    Agents assemble their heartbeat JSON as text in bash, so an empty interpolated value can leave
+    a structurally-invalid comma (issue #31). Returns (possibly_repaired_str, was_changed). The
+    repairs are conservative and target only structural artifacts an agent produces; they never
+    alter this endpoint's legitimate string values (the agent emits no string containing ',,' —
+    haproxy_stats_csv is base64/comma-free and the rest are constrained os/kernel/ip/version text).
+    """
+    import re
+    sanitized = False
+    # Fix 1: empty value before a comma ("server_statuses": ,)
+    if re.search(r':\s*,', body_str):
+        body_str = re.sub(r':\s*,', ': null,', body_str); sanitized = True
+    # Fix 2: empty value before a closing brace ("field":})
+    if re.search(r':\s*}', body_str):
+        body_str = re.sub(r':\s*}', ': null}', body_str); sanitized = True
+    # Fix 3: trailing comma before } or ]
+    if re.search(r',(\s*[}\]])', body_str):
+        body_str = re.sub(r',(\s*[}\]])', r'\1', body_str); sanitized = True
+    # Fix 4: leading comma run right after an opening brace/bracket (issue #31): an empty
+    # $system_info as the first member collapses to '{ , "name": ...'. The ': ,' fix above cannot
+    # catch this because there is no key/colon before the comma.
+    if re.search(r'([{\[])(\s*,)+', body_str):
+        body_str = re.sub(r'([{\[])(\s*,)+', r'\1', body_str); sanitized = True
+    # Fix 5: a run of commas between members (issue #31): an empty $system_info between two fields
+    # produces '"version": "x",\n    ,\n    "haproxy_status": ...'. Runs after Fix 1/3 so only
+    # structural commas remain; collapse any comma run to a single comma.
+    if re.search(r',(\s*,)+', body_str):
+        body_str = re.sub(r',(\s*,)+', ',', body_str); sanitized = True
+    return body_str, sanitized
+
+
 def get_platform_key(agent_platform: str) -> str:
     """Convert agent platform to standardized platform key - fixed empty platform fallback"""
     platform = agent_platform.lower() if agent_platform else 'unknown'
@@ -1455,47 +1489,33 @@ async def agent_heartbeat_by_name(
     import json
     from pydantic import ValidationError
 
-    # Read raw body and sanitize common JSON errors from agents
+    # Read raw body. Parse VALID JSON as-is (the normal case for every agent version) and only
+    # fall back to the malformed-JSON repair when the body does not parse. This guarantees a healthy
+    # heartbeat from any agent version is byte-for-byte untouched — the repair regexes can never run
+    # against a well-formed payload (issue #31; strictly safer than repairing unconditionally).
     try:
         raw_body = await request.body()
         body_str = raw_body.decode('utf-8')
-        
-        # Sanitize common malformed JSON patterns from agents
-        original_body = body_str
-        sanitized = False
-        
-        # Fix 1: Empty values before comma (most common: "server_statuses": ,)
-        if re.search(r':\s*,', body_str):
-            body_str = re.sub(r':\s*,', ': null,', body_str)
-            sanitized = True
-        
-        # Fix 2: Empty values before closing brace
-        if re.search(r':\s*}', body_str):
-            body_str = re.sub(r':\s*}', ': null}', body_str)
-            sanitized = True
-        
-        # Fix 3: Trailing commas
-        if re.search(r',(\s*[}\]])', body_str):
-            body_str = re.sub(r',(\s*[}\]])', r'\1', body_str)
-            sanitized = True
-        
-        if sanitized:
-            # Extract agent name for logging
-            agent_name = "unknown"
-            try:
-                name_match = re.search(r'"name"\s*:\s*"([^"]+)"', body_str)
-                if name_match:
-                    agent_name = name_match.group(1)
-            except:
-                pass
-            
-            logger.info(f"Sanitized malformed JSON from agent '{agent_name}' - fixed empty values and trailing commas")
-            logger.debug(f"Original JSON (preview): {original_body[:300]}")
-            logger.debug(f"Sanitized JSON (preview): {body_str[:300]}")
-        
-        # Parse sanitized JSON into Pydantic model
-        heartbeat_dict = json.loads(body_str)
-        
+
+        try:
+            heartbeat_dict = json.loads(body_str)
+        except json.JSONDecodeError:
+            # Malformed body (would otherwise be a hard 400). Attempt a conservative repair of the
+            # comma artifacts a hand-built agent heartbeat can emit, then re-parse.
+            repaired, changed = _sanitize_agent_json(body_str)
+            if changed:
+                agent_name = "unknown"
+                try:
+                    name_match = re.search(r'"name"\s*:\s*"([^"]+)"', repaired)
+                    if name_match:
+                        agent_name = name_match.group(1)
+                except Exception:
+                    pass
+                logger.info(f"Repaired malformed JSON from agent '{agent_name}' before parsing")
+                logger.debug(f"Original JSON (preview): {body_str[:300]}")
+                logger.debug(f"Repaired JSON (preview): {repaired[:300]}")
+            heartbeat_dict = json.loads(repaired)  # may still raise -> handled as 400 below
+
         # DEBUG: Log cluster_id for auto-register troubleshooting
         if heartbeat_dict.get('name'):
             logger.info(f"HEARTBEAT DEBUG: agent={heartbeat_dict.get('name')}, cluster_id={heartbeat_dict.get('cluster_id')}, has_cluster_id={bool(heartbeat_dict.get('cluster_id'))}")

backend/tests/test_agent_heartbeat_sanitizer.py

@@ -0,0 +1,97 @@
+"""Issue #31 — agent heartbeat JSON sanitizer.
+
+A self-hosted agent builds its heartbeat JSON as text in bash. When a collected value is empty,
+the payload can contain a structurally-invalid comma that broke the heartbeat with
+`HTTP 400 Invalid JSON: Expecting property name enclosed in double quotes`. The backend now
+repairs that pattern in `_sanitize_agent_json` so an already-deployed agent recovers without a
+re-install. These tests pin that behaviour and prove the repair never corrupts a healthy payload.
+"""
+import json
+
+from routers.agent import _sanitize_agent_json
+
+
+def _assert_parses(raw: str) -> dict:
+    out, _ = _sanitize_agent_json(raw)
+    return json.loads(out)  # raises if the repair did not produce valid JSON
+
+
+def test_reporter_empty_system_info_bare_comma():
+    # The exact shape the reporter hit: an empty $system_info collapses '    $system_info,' to a
+    # bare comma between two members -> '"version": "x",\n    ,\n    "haproxy_status": ...'.
+    raw = (
+        '{\n'
+        '    "name": "test",\n'
+        '    "hostname": "h",\n'
+        '    "status": "online",\n'
+        '    "version": "2.0.0",\n'
+        '    ,\n'
+        '    "haproxy_status": "running",\n'
+        '    "cluster_id": 1\n'
+        '}'
+    )
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "test"
+    assert parsed["status"] == "online"
+    assert parsed["haproxy_status"] == "running"
+
+
+def test_empty_numeric_subfield_before_comma():
+    # An empty unquoted numeric ("memory_total": ,) — covered by the pre-existing Fix 1.
+    raw = '{ "name": "t", "cpu_count": , "memory_total": , "status": "online" }'
+    parsed = _assert_parses(raw)
+    assert parsed["cpu_count"] is None and parsed["memory_total"] is None
+    assert parsed["status"] == "online"
+
+
+def test_empty_value_before_closing_brace():
+    raw = '{ "name": "t", "status": "online", "applied_config_version": }'
+    parsed = _assert_parses(raw)
+    assert parsed["applied_config_version"] is None
+
+
+def test_leading_comma_first_member():
+    # Empty $system_info as the FIRST member -> '{ , "name": ... }'.
+    raw = '{\n    ,\n    "name": "t",\n    "status": "online"\n}'
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "t"
+
+
+def test_comma_run_two_empty_fields():
+    # Two empties in a row (odd-length comma run) must still collapse to valid JSON.
+    raw = '{ "a": 1,\n    ,\n    ,\n    "b": 2 }'
+    parsed = _assert_parses(raw)
+    assert parsed["a"] == 1 and parsed["b"] == 2
+
+
+def test_trailing_comma_regression():
+    # Pre-existing Fix 3 must still hold after the new fixes were added.
+    raw = '{ "name": "t", "status": "online", }'
+    parsed = _assert_parses(raw)
+    assert parsed["name"] == "t"
+
+
+def test_healthy_payload_is_untouched():
+    # A well-formed agent payload must pass through unchanged (sanitized=False) and its values —
+    # including the base64 stats CSV and the nested server_statuses — must be byte-identical.
+    payload = {
+        "name": "agent-1",
+        "status": "online",
+        "cluster_id": 1,
+        "server_statuses": {"be_app": {"s1": "UP", "s2": "DOWN"}},
+        "network_interfaces": ["eth0", "eth1"],
+        "haproxy_stats_csv": "IyBwdmJjLGJhY2tlbmQsZnJvbnRlbmQs",  # base64: contains commas only inside a quoted string is impossible (base64 has none)
+        "applied_config_version": "cluster-1-v42",
+    }
+    raw = json.dumps(payload)
+    out, changed = _sanitize_agent_json(raw)
+    assert changed is False
+    assert out == raw  # byte-identical
+    assert json.loads(out) == payload
+
+
+def test_idempotent_on_already_clean_minimal():
+    raw = '{"name": "t", "status": "online"}'
+    out, changed = _sanitize_agent_json(raw)
+    assert changed is False
+    assert out == raw

backend/tests/test_agent_script_heartbeat_json.py

@@ -0,0 +1,46 @@
+"""Issue #31 — agent-script hardening guard (static).
+
+The agent install scripts hand-build the heartbeat JSON, so if `collect_system_info` ever yields
+nothing the `$system_info,` line collapses to a bare comma and the whole heartbeat is invalid JSON
+(HTTP 400). The fix adds a guard at every fragment-form call site that substitutes a single valid
+key when system_info is empty. This static check enforces that the guard is present AND kept in
+sync across BOTH platform scripts — the project requires the two agent-script copies to stay in
+lockstep. (Empty numeric subfields like "memory_total": , are a separate, milder case already
+repaired by the backend sanitizer, so they are intentionally NOT guarded in the script — guarding
+them with a strict integer test would wrongly reject the scientific-notation that mawk emits for
+multi-GB sizes on Debian/Ubuntu.)
+"""
+import os
+
+_SCRIPT_DIR = os.path.join(
+    os.path.dirname(os.path.dirname(os.path.abspath(__file__))),  # backend/
+    "utils", "agent_scripts",
+)
+
+
+def _read(name: str) -> str:
+    with open(os.path.join(_SCRIPT_DIR, name), "r") as f:
+        return f.read()
+
+
+LINUX = _read("linux_install.sh")
+MACOS = _read("macos_install.sh")
+
+# The empty-system_info guard — present at BOTH fragment call sites (register_agent + send_heartbeat).
+_B2_GUARD = '[[ "$system_info" != *\'"\'* ]] && system_info=\'"operating_system": "unknown"\''
+
+
+def test_b2_guard_present_and_in_sync():
+    # Two fragment-form call sites per script (register_agent + send_heartbeat), identical wording.
+    assert LINUX.count(_B2_GUARD) == 2, "linux_install.sh missing/duplicated empty-system_info guard"
+    assert MACOS.count(_B2_GUARD) == 2, "macos_install.sh missing/duplicated empty-system_info guard"
+
+
+def test_b2_guard_precedes_every_fragment_system_info_use():
+    # Every '    $system_info,' fragment line (the one that breaks on an empty value) must be in a
+    # function whose system_info was guarded. We assert the count of guards matches the count of
+    # fragment-form interpolations' call sites: each script has exactly one register + one
+    # send_heartbeat fragment builder feeding those lines, both guarded above.
+    for name, script in (("linux", LINUX), ("macos", MACOS)):
+        assert script.count("    $system_info,") >= 1, f"{name}: fragment heartbeat form unexpectedly gone"
+        assert script.count(_B2_GUARD) == 2, f"{name}: each fragment call site must carry the guard"

backend/utils/agent_scripts/linux_install.sh

@@ -1055,7 +1055,12 @@ register_agent() {
     local arch=$(uname -m)
     platform=$(uname -s | tr '[:upper:]' '[:lower:]')  # Remove local to make it global
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     local json_payload=$(cat <<SIMPLE_EOF
 {
     "name": "$AGENT_NAME",
@@ -1357,7 +1362,12 @@ send_heartbeat() {
     local server_statuses=$(get_server_statuses)
     local haproxy_stats_csv=$(get_haproxy_stats_csv)
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     # Get HAProxy version for heartbeat (safe extraction, fallback to "unknown")
     local haproxy_version="unknown"
     if command -v haproxy &> /dev/null; then

backend/utils/agent_scripts/macos_install.sh

@@ -920,7 +920,12 @@ register_agent() {
     local arch=$(uname -m)
     platform=$(uname -s | tr '[:upper:]' '[:lower:]')  # Remove local to make it global
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     local json_payload=$(cat <<SIMPLE_EOF
 {
     "name": "$AGENT_NAME",
@@ -1191,7 +1196,12 @@ send_heartbeat() {
     local server_statuses=$(get_server_statuses)
     local haproxy_stats_csv=$(get_haproxy_stats_csv)
     local system_info=$(collect_system_info)
-    
+    # issue #31: if collect_system_info produced no JSON content (empty on an unusual host), the
+    # '$system_info,' line below would collapse to a bare comma and break the heartbeat JSON. A
+    # valid fragment always contains a quoted key; if none is present, fall back to one. The glob
+    # '*"*' is the most portable bash test (no POSIX class / pattern-substitution), safe on bash 3.x+.
+    [[ "$system_info" != *'"'* ]] && system_info='"operating_system": "unknown"'
+
     # Get HAProxy version for heartbeat (safe extraction, fallback to "unknown")
     local haproxy_version="unknown"
     if command -v haproxy &> /dev/null; then

frontend/package.json

@@ -1,6 +1,6 @@
 {
   "name": "haproxy-openmanager-frontend",
-  "version": "1.8.2",
+  "version": "1.8.3",
   "description": "HAProxy Load Balancer Management UI",
   "license": "AGPL-3.0-or-later",
   "dependencies": {

version.json

@@ -1,5 +1,5 @@
 {
-  "version": "1.8.2",
-  "releaseName": "ACME nonce fix (ZeroSSL registration)",
+  "version": "1.8.3",
+  "releaseName": "Agent heartbeat JSON fix",
   "releaseDate": "2026-06-25"
 }