When using dbt and datalake (clickhouse) ingestion sources, tables/models aren't siblings?

[gfaugere-chut]

I've tested datahub and ingested both dbt as a source and the related datalake (clickhouse).
The ingestion seemed OK, and lineage is properly generated between entities

However, I was expecting the dbt model (urn:li:dataset:(urn:li:dataPlatform:dbt,my_schema.my_table,PROD)) and clickhouse table (urn:li:dataset:(urn:li:dataPlatform:clickhouse,my_schema.my_table,PROD)) to be siblings, and my dbt documentation and columns descriptions to be "pushed" to the table

Am I mistaken, is this expected behaviour? Or is this something that's configurable and that I've missed in the documentation?

[github-actions]

Linear: ING-2361

[sgomezvillamor]

Hey @gfaugere-chut, great question — and you're right that the docs don't make this flow easy to figure out on your own, so let me clarify!

How siblings work

When you ingest both dbt and a warehouse source (like ClickHouse), DataHub creates a sibling relationship between the two entities. Rather than copying or "pushing" metadata, DataHub merges them in the UI — both URNs appear as a single entity, and the primary sibling's metadata (dbt, by default) is displayed on top. So your dbt descriptions and column docs will surface on the ClickHouse entity once siblings are properly linked.

Most likely root cause: missing target_platform

For siblings to be linked, your dbt recipe needs to explicitly declare the warehouse platform:

source:
  type: dbt
  config:
    target_platform: clickhouse  # 👈 this is required
    # ... rest of your config

Without this, dbt generates sibling URNs that don't match your ClickHouse entities, so the association never happens — even if lineage looks correct in the UI.

ClickHouse-specific gotcha: URN casing

ClickHouse is case-sensitive for table/column names. If your dbt models emit lowercase identifiers but the ClickHouse ingestion source produces mixed-case or uppercase ones, the URNs won't match and siblings still won't link. You can work around this by setting:

convert_column_urns_to_lowercase: true

in your dbt config (this is applied automatically for Snowflake but not for ClickHouse yet — something we should improve).

---

Closing this for now since it looks like expected behaviour + a config gap. Please reopen if adding target_platform: clickhouse doesn't fix it — happy to dig deeper! 🙌

---

Generated by Claude Code

[gfaugere-chut]

I do have target_platform: clickhouse in my dbt recipe config, and as you can see in my original message the urns casing are matching properly :(

[sgomezvillamor]

Thanks for confirming — that rules out the most obvious config issues, so let's dig deeper.

Where siblings are actually created

The sibling relationship is not created by GMS — it's created by the SiblingAssociationHook, which runs inside the datahub-mae-consumer container. GMS logs won't show anything useful here.

Here's the flow for a dbt model + ClickHouse table pair:

1. The dbt ingestion source emits an upstreamLineage aspect on the ClickHouse entity, pointing to the dbt model as its upstream.
2. The mae-consumer picks up that Metadata Change Log (MCL) event and the SiblingAssociationHook fires.
3. It finds exactly one dbt upstream → creates the bilateral sibling relationship.

There's one known silent failure mode worth flagging: if either the dbt entity or the ClickHouse entity already carries a siblings aspect from a previous (possibly partial or stale) ingestion run, the hook will skip silently:

if (existingDbtSiblingAspect != null || existingSourceSiblingAspect != null) {
    log.debug("Skipping sibling creation - existing siblings found");
    return;
}

This is only logged at DEBUG level, so it won't appear in default logs — making it very easy to miss.

---

Suggested next steps

1. Confirm the sibling feature works at all — try ingesting dbt + Snowflake or dbt + BigQuery and check if siblings appear. This tells you if the issue is ClickHouse-specific or a broader environment problem.
2. Verify ENABLE_SIBLING_HOOK=true in your mae-consumer environment (it defaults to true, but worth double-checking).
3. Check mae-consumer logs (not GMS!) for the string "Associating" — if it's absent, the hook either didn't fire or was skipped silently.
4. Enable DEBUG logging for com.linkedin.metadata.kafka.hook.siblings on the mae-consumer and re-run ingestion — this will surface the skip reason if the hook is being bypassed.
5. If you suspect stale siblings: delete the siblings aspect from both the dbt and ClickHouse entities via the DataHub API, then re-run both ingestions from scratch.

Happy to help interpret whatever you find in the logs! 🙏

---

Generated by Claude Code

[sgomezvillamor]

One more thing to check first — before all the steps above:

Step 0: Verify the lineage exists on the ClickHouse entity.

Open the ClickHouse entity in the DataHub UI and check its Lineage tab — the dbt model should appear as an upstream. If it doesn't, the dbt ingestion source never emitted the upstreamLineage aspect for that entity, and the sibling hook would never have had a chance to fire regardless of any other setting.

This can happen if the dbt node was treated as ephemeral, or was filtered out by entities_enabled config. It's the most fundamental thing to rule out first.

---

Generated by Claude Code

[gfaugere-chut]

As you can see, the lineage seems to be working:

I don't have other data sources I can ingest to check if it's related to clickhouse :/

I'm using the helm chart with close to defaults values, I don't see a dedicated mae-consumer, is it bundled with an other pod?

[sgomezvillamor]

it should be in gms pod then

https://github.com/acryldata/datahub-helm/blob/df1366aefd372301027d86d14f12548f64f1ae8d/charts/datahub/subcharts/datahub-gms/templates/deployment.yaml#L176-L184