Open Source Data Catalog Options for AI Infrastructure

What a Catalog Must Actually Deliver for AI Infrastructure to Work

Human analysts have always operated with implicit knowledge that never makes it into the catalog. A finance analyst knows that "revenue" means recognized ARR when the sales team uses it and something else entirely when the CFO does. That analyst resolves the ambiguity through context, institutional memory, and the occasional Slack message. An agent receives a definition and treats it as ground truth. Whatever is in the catalog becomes the operative reality for every downstream query.

This is why discoverability, the traditional catalog design criterion, is necessary but not sufficient. Finding the dataset is step one. Knowing what it means, where its values came from, whether those values are current, and who is permitted to access them under what conditions: that is the actual requirement for AI consumers.

For agent workloads specifically, four things have to work. Semantic context: table and metric definitions that agents can consume programmatically, not just read in a UI. Column-level lineage: a traceable path from the value an agent reads back to the system that produced it. Query-time governance: permissions enforced at the moment of access, not inferred from a service account carrying more privilege than the originating user should have. Freshness signals: explicit metadata indicating whether the data reflects current state, so an agent does not confidently answer using a three-week-old snapshot.

There is a fifth requirement that most catalog evaluations underweight: audit trails. At agent-level query volume, logs that capture only the query itself are forensically inadequate. The log needs the originating user's identity, the agent's intent, and the lineage of the data it accessed, together, in one place. Without those three fields, a compliance audit after something goes wrong is nearly impossible to conduct meaningfully.

One distinction to make explicit before looking at specific tools: a catalog is not a semantic layer. A catalog tells agents what data exists. A semantic layer tells agents how to use it correctly, by intercepting query execution and applying metric definitions mathematically rather than descriptively. Enterprise AI infrastructure needs both. The open-source catalogs differ sharply in how much semantic function they carry natively versus how much they delegate to an external tool. That delegation is not inherently a weakness, but teams need to know it is happening so they can plan for it.

How the Open Source Catalog Landscape Is Currently Structured

Open source in this space covers a meaningful range. Apache Atlas and Amundsen are fully community-governed projects with limited commercial backing. Marquez sits within the OpenLineage ecosystem with a narrower, more clearly defined scope. DataHub and OpenMetadata each have commercial managed tiers, Acryl Data and a hosted cloud offering respectively, and both lead the open-source tier in community activity and release cadence based on GitHub contributor activity through 2024.

What open source reliably delivers: deployment inside your own infrastructure, adaptable metadata models, no vendor lock-in on the catalog schema, and significant integration flexibility. These are real advantages, particularly for organizations with sensitive data residency requirements or complex existing infrastructure.

What open source does not deliver out of the box: polished governance workflows, built-in data quality enforcement at scale, and the semantic layer functions that AI agents need at query time. These require either extension or a separate tool. Teams that underestimate that extension work frequently discover it after the catalog is already in production. That is the harder moment to discover it.

Databricks Unity Catalog occupies an adjacent position here. It is not fully open source, but it is widely deployed and its capabilities serve as a useful reference point for what the open-source tools are missing, particularly around query-time governance and fine-grained access control.

The actual decision is not "open source versus proprietary" in the abstract. It is which open-source foundation pairs most cleanly with the semantic and governance layer that a specific AI infrastructure requires. That pairing question is where the evaluation gets consequential.

DataHub's Strengths and Where It Stops Short for AI Workloads

DataHub originated at LinkedIn and is now the most starred open-source catalog project on GitHub, with the commercial managed tier operated by Acryl Data. The scale of community adoption translates directly into connector breadth and release velocity, two practical advantages when integrating with a heterogeneous data stack.

The genuine strengths: DataHub has strong column-level lineage and the broadest connector ecosystem in the open-source category. It ingests active metadata, including query history and BI tool usage, rather than relying solely on periodic batch snapshots. Its metadata graph models relationships between assets in a way that is queryable via GraphQL, which means agents and pipelines can interrogate the catalog programmatically rather than relying on a human-facing UI. That GraphQL interface is a material advantage for AI infrastructure. It makes the catalog a first-class participant in automated workflows instead of a passive reference document.

Here is where it stops. DataHub documents lineage and ownership well, but it does not enforce semantic definitions at query time. An agent can find a metric definition in the catalog and then ignore it when constructing a query, and nothing in DataHub will intercept that. The catalog entry is documentation; it is not a constraint.

Access control in DataHub operates at the catalog level: it governs who can see metadata. It does not govern who can query the underlying asset with row-level filtering or column masking. Query-time governance requires a separate enforcement layer, something like a policy-aware query engine or a platform-native access control mechanism layered below the catalog.

DataHub is the strongest open-source foundation for discovery and lineage. Teams using it for AI infrastructure will need to pair it with a semantic layer and a query-time policy engine. Make that architecture decision before the catalog goes into production.

OpenMetadata's Approach to End-to-End Coverage and Where the Gaps Remain

OpenMetadata was designed from the start to unify discovery, lineage, quality, and governance in a single platform. That vertical integration is its defining architectural choice and the primary way it differentiates from DataHub's metadata-graph-first approach.

The integration shows up in practical features. OpenMetadata includes native data quality workflows: test suites, profiling, and freshness checks that run inside the platform rather than requiring a separate tool. Team-based ownership models are native, not bolted on. Collaboration features, including conversations on datasets, task assignment, and announcements, make cross-functional data ownership operationally tractable rather than aspirational. Cross-functional ownership matters for AI infrastructure specifically because the semantic definitions that agents depend on need to be owned and maintained by people who understand the business logic, not just the people who built the pipeline.

The gaps are structurally similar to DataHub's, which says something about the limits of what a catalog can do without a semantic execution layer. OpenMetadata can store descriptions and tags, but metric definitions are not mathematically enforced at query execution. An agent reading from OpenMetadata can still misapply business logic. The catalog describes the correct behavior rather than enforcing it.

Fine-grained query-time access control is also not native. Row filters and column masking at the point of query execution require an external engine. Governed access to the underlying data must be enforced downstream of the catalog.

For teams without a dedicated data engineering function, OpenMetadata's integrated approach reduces operational overhead relative to DataHub. Some depth in individual capabilities is the tradeoff. It still requires a semantic execution layer and a policy-enforcing query engine for full AI infrastructure use, but the baseline governance posture it provides out of the box is more complete than what DataHub delivers without extension.

Apache Atlas, Amundsen, and Marquez: What Each One Covers and What It Was Built For

These three projects deserve honest, specific treatment rather than the vague "also consider" framing they usually receive.

Apache Atlas was originally built for Hadoop and Hive governance. Its strongest capabilities are classification, tagging, and policy propagation within that ecosystem. Outside the Hadoop stack, integration is heavy lifting. Active metadata capability is limited. In the current landscape, Atlas is not a realistic general-purpose AI catalog. It solves a real problem for organizations still running significant Hive workloads, but it is not a foundation to build forward-looking AI infrastructure on. That is just the honest read.

Amundsen came out of Lyft and was designed for search and discovery. It is genuinely good at surfacing popular and trusted datasets through usage signals, which matters when a data lake has accumulated enough assets that discoverability is the primary friction. Lineage support is partial. There is no native quality enforcement or semantic layer. For teams where search is the primary need and semantic enforcement will live entirely elsewhere, Amundsen is a lightweight and operationally straightforward choice. It is not a complete AI infrastructure catalog and does not attempt to be.

Marquez came out of WeWork and implements the OpenLineage standard. It is narrowly focused on operational lineage for data pipelines: tracking job runs, dataset inputs and outputs, and run-level metadata. It does not attempt to be a full catalog. What it does, it does well. The OpenLineage standard it implements is worth separating from the tool itself. OpenLineage is a lineage interchange format that makes lineage data portable across platforms. Teams can collect lineage in Marquez and surface it in DataHub or OpenMetadata. That is a legitimate architectural pattern, particularly for teams that want to decouple lineage collection from catalog presentation and preserve optionality as their stack evolves.

Atlas, Amundsen, and Marquez each solve one well-defined problem. None functions as a complete AI infrastructure catalog on its own, and none has the active development cadence to close that gap soon.

The Semantic Context Gap That All Open Source Catalogs Share

Every open-source catalog evaluated here documents meaning. None of them enforce it at query execution. That distinction is the central limitation for AI infrastructure.

A catalog entry stating that revenue equals recognized ARR excluding professional services is useful documentation. A semantic layer that intercepts the query and applies that definition mathematically is enforcement. Agents benefit from the latter. Documentation helps a human analyst remember to apply the definition correctly; it does not prevent an agent from ignoring it.

Research on production AI data stacks consistently reflects this structural reality: pairing a semantic context layer with a catalog layer produces materially higher query accuracy in agent workloads than relying on catalog documentation alone. The catalog is a necessary condition for AI infrastructure, not a sufficient one.

The architectural pattern that emerges is consistent across mature AI data stacks. An open-source catalog handles discovery, lineage, and ownership. A semantic or metrics layer handles definitional enforcement at query time. Tools in this space include Cube, AtScale, and the dbt Semantic Layer, among platform-native equivalents depending on the compute environment. A query-time policy engine handles access control. These three components address distinct requirements and cannot cleanly substitute for one another.

Teams that skip the semantic layer and assume catalog descriptions are sufficient are replicating exactly the failure pattern that prevents AI pilots from scaling. The demo works on clean, well-labeled data where definitions happen to be applied correctly. Production fails on the messy reality, where the agent reads a definition and then queries a field that does not implement it consistently across sources. That is not a model problem. It is an architecture problem, and it is visible before the model is ever selected if the infrastructure is evaluated with the right criteria.

Governance Requirements That Open Source Catalogs Enforce Natively vs. Require You to Build

Understanding what open-source catalogs actually enforce versus what they document with the expectation that enforcement lives elsewhere is the most practically important question in this evaluation. The answer is narrower than most teams assume going in.

What open-source catalogs handle natively: asset-level ownership assignment, metadata-level access control over who can view catalog entries, and coarse-grained classification and tagging. These are real governance capabilities. They are also the governance capabilities that matter most for human analysts, which is the user profile these tools were originally designed for.

What they do not handle natively: row-level filtering at query time, column masking based on data sensitivity and user identity, identity-scoped permissions that follow data consistently across compute engines, and audit logs that capture agent identity and query lineage together. These are the governance capabilities that matter most for AI agents.

The identity scoping issue is particularly consequential. An agent fans out a single user request into dozens of sub-queries across multiple systems. Each of those queries must be governed under the originating user's identity, not under a shared service account carrying the agent's own elevated permissions. If the governance model does not preserve that identity chain through the agent's query execution, the access controls on the underlying data become effectively unenforced. The agent's service account bypasses them even when the originating user would lack that access.

There is a related sensitivity aggregation problem that operates at query time rather than at catalog registration time. Two individually non-sensitive datasets can produce a sensitive result when joined. A catalog can classify each dataset at registration. It cannot evaluate the sensitivity of the result set that an agent produces by combining them at query time. That evaluation must happen in the query engine, not in the catalog.

The practical implication: audit which governance capabilities your AI workloads require before selecting a catalog, then identify explicitly where each requirement will be met. The catalog, the semantic layer, and the query engine each cover different parts of the governance surface. Teams that map those requirements after deploying the catalog typically find they have committed to a foundation before understanding what they still need to build.

How to Choose a Starting Point Given Your Team's Actual AI Infrastructure Needs

Two axes drive the selection decision more than any feature comparison. First, how much semantic enforcement your AI workloads require: agents making autonomous decisions need stronger enforcement than agents supporting human analysts who provide a final review. Second, how much governance infrastructure your team is actually prepared to operate alongside the catalog, because the semantic layer and query-time policy engine are not optional components for production AI workloads. The question is only who builds and operates them.

If lineage and discovery are the primary requirement, and semantic enforcement will live in a clearly defined separate layer, DataHub is the most capable open-source foundation. It has the most active metadata depth, the broadest connector coverage, and the GraphQL interface that makes programmatic catalog access straightforward for agent workloads.

If the priority is a single platform covering discovery, quality, and team-based ownership without requiring a dedicated data engineering function to integrate and maintain separate components, OpenMetadata's integrated approach reduces operational overhead. Some depth in individual capabilities is the tradeoff, but for teams where operational simplicity is a genuine constraint, that tradeoff is often the right one.

If operational pipeline lineage and OpenLineage interoperability are the specific requirement, Marquez solves that problem cleanly. It pairs with either DataHub or OpenMetadata as a lineage collection layer, and the portability the OpenLineage standard provides is worth preserving if there is any likelihood of changing the primary catalog later.

For any of these choices: plan the semantic layer and query-time governance layer before the catalog goes into production. Retrofitting both after agents are running is significantly harder than designing for them upfront, because you are working against live workloads that have already encoded assumptions about what the catalog provides. That is a lesson the field has learned the hard way, repeatedly, and there is no particular reason your team needs to learn it again.