Open Source Data Catalog Options for AI Infrastructure

Open source data catalogs are going through a structural repositioning: the primary consumer of catalog metadata is no longer a human analyst browsing a UI, but an AI agent querying at runtime. For years, catalog design assumed a human would browse, search, and interpret what the catalog surfaced, and that assumption shaped every feature priority on the roadmap: searchability, visual lineage diagrams, UI polish, tagging workflows. None of that logic holds when the thing reading the catalog isn't a person with judgment and context, but a system that queries, ingests, and acts in milliseconds.

Agents don't browse. They retrieve governed context at inference time, they expect a machine-readable interface rather than a dashboard, and they produce confident output whether or not the context handed to them was accurate. That last part is the dangerous one. A human analyst who gets confused by a vague column name pauses and asks someone. An agent doesn't pause; it answers.

The clearest signal that this shift is already reshaping the market: OpenMetadata, one of the leading open source catalog projects, has rebranded itself with the tagline "The Open Context Layer for Data and AI". When the category leader changes its own name for itself, that renaming is an admission that the old framing no longer describes what the product needs to do.

The risk of getting this wrong is already being quantified. Gartner analyst Andres Garcia-Rodeja projected that a majority of agentic analytics projects relying solely on the Model Context Protocol will fail by 2028, because of the absence of a consistent semantic layer. A catalog built for human discovery can absolutely expose its metadata to an agent through a technical connection, but connectivity isn't context. Handing an agent raw metadata without governance or semantic grounding just lets it be wrong faster, and with more confidence, than a human ever could be.

Enterprise data estates built for human analysts, not agent consumers

The urgency behind this redesign comes from a mismatch between the data and the catalog: the data itself was built for human analysts, and "clean enough for reporting" is a different standard from "fit for agent use". A dashboard tolerates ambiguity because a person is looking at it and can apply judgment. An agent making a decision off the same data has no such judgment to apply.

What counts as AI-ready data is a narrower and stricter bar than what most governance programs check for. A dataset can pass every existing audit a company runs and still be unfit for an agent to reason over.

Agents compound this problem because they carry no institutional memory. A human analyst who hits a vague column name can walk over and ask someone what it means. An agent cannot ask that question, and a stale or partial answer looks exactly as confident as a correct one. That's the failure mode: not a crash, not an error message, but a wrong answer delivered with full certainty and no visible seam.

Workday's data leader, Joe DosSantos, described this failure in direct terms: "We built a revenue analysis agent and it couldn't answer one question… we were missing this translation layer". What was missing was the translation layer between the two, converting raw tables into something an agent can actually reason about.

One-time data cleaning sprints don't close this gap. They treat a symptom, not the underlying system, and within months the same quality problems return as new records flow through the same unchanged processes. The cost of the problem also grows the later it's caught. Organizations that skip upstream governance end up paying downstream rework costs that can dwarf what the original implementation budget set aside.

The research points to the same structural causes recurring across failed AI projects: errors introduced at the point of ingestion spread across every platform the data later touches, and governance permission layers are often simply absent rather than misconfigured. The IBM Institute for Business Value found that only 29% of technology leaders strongly agree that their enterprise data meets the quality, accessibility, and security standards needed to scale generative AI. That describes most of the market, not a fringe concern.

The forward-looking number is just as stark. Gartner projects that a majority of AI projects will miss their value targets by 2027, tracing the cause back to governance structures that are fragmented and reactive rather than built ahead of the need. This is not a catalog feature gap but a mismatch between how enterprise data was built and what an agent consumer actually requires from it, and no amount of better search or prettier lineage diagrams fixes a foundation that was never poured for this purpose.

The four criteria for an AI-ready catalog

Four criteria now separate a catalog that functions as real AI infrastructure from one that's merely a browsable directory an agent happens to be able to reach: semantic context, governed query access, real-time freshness, and MCP-native exposure. These aren't feature checkboxes borrowed from the old human-browsing era; they're the evaluation lens for everything a team building agent infrastructure needs to check before it trusts a catalog with production traffic.

Semantic context means something more specific than a glossary. A data catalog tells a user what exists and who owns it. A semantic layer defines how a given metric is actually calculated. A context layer packages both of those together with access policy, and serves the combined result at the moment an agent makes an inference. Active metadata drives behavior: it routes a query, triggers a quality check, or surfaces governed context at the exact moment data is being interpreted. Skipping this tier makes the agent silently, confidently wrong rather than causing an obvious break, which is a harder failure to catch than a crash.

Governed query access has to work at the point of query itself, scoped to the actual end user's identity rather than assumed from a broad, shared service account. A financial institution couldn't explain why its AI model had denied a loan, and the investigation found that an outdated third-party dataset had been silently introduced weeks earlier, skewing the credit scoring behind the decision. The audit failed because there was no lineage trail connecting the bad dataset to the bad outcome. Treating the agent as exempt from the access rules that govern the data it reads is how that loan case happened.

Real-time freshness matters because a snapshot that's perfectly fine for a dashboard is already stale by the time an agent uses it to make a decision. A related test is whether the platform validates its own assertions against what's actually happening operationally, or whether it lets stale entries sit indefinitely without ever being checked against reality.

MCP-native exposure is the criterion most likely to be misread as a simple yes-or-no feature. Native support means there's nothing extra for a team to build or maintain on top of it. By September 2026, having an MCP server at all stopped being the differentiator, because nearly every serious catalog can talk to an agent in some form. What actually separates these products now is whether the agent on the other end of that connection gets a usable, governed answer, or just a raw metadata dump with no semantic grounding behind it. The real question to ask a vendor isn't "does it have an MCP server," but whether the context that server exposes is governed, lineage-aware, and scoped to the agent's actual permissions. Those four criteria, not star counts on a GitHub repo and not the length of a connector list, are what the tool-by-tool comparison that follows is measured against.

How the leading open source catalogs perform against these criteria

Every leading open source catalog today supports some version of the AI-consumer model. The differences that matter are in how much infrastructure each one demands to deliver that support, and how much of the semantic and governance layer comes built in versus left for the team to construct by hand.

The largest open source catalog by community size has moved fastest and heaviest on the agent-infrastructure front. Its latest confirmed release, v1.7.0.14, shipped September 29, 2026, following a v1.5.x line that added agent-context integration to the search CLI and policy targeting alongside it, with v1.7.0 hardening the system further through optimistic locking and assertion notes. On MCP, it ships a first-party MCP Server along with an AI Context Kit (v1.4.0) that lets agents query catalog metadata directly, with no glue code required to bridge the gap. Its lineage is column-level with interactive impact analysis at a production grade, and its governance model includes domain-level RBAC, data contracts, and glossary management. A production deployment calls for Kafka, Elasticsearch or OpenSearch, a schema registry, and Kubernetes, and a realistic estimate runs to weeks of setup time with at least two dedicated engineers before the first piece of metadata is even ingested. OpenAI's internal data agent, Kepler, runs on this catalog across roughly 70,000 datasets for thousands of employees, which stands as a concrete production case for the agentic model this whole category is now built around.

OpenMetadata takes a different approach built around breadth rather than modularity. It comes from the founders of Apache Hadoop, Apache Atlas, and Uber Databook, built on a schema-first design, and its parent company, Collate, joined the Linux Foundation as a Silver Member in March 2026. Rather than assembling separate modules, it ships discovery, governance, lineage, quality profiling, and collaboration as one unified platform, and it carries the broadest connector coverage in this comparison at over 120 pre-built connectors. It ships its own MCP server and serves cataloged metadata directly to agents. Its AI Governance standards in OpenMetadata 2.0 reach across the entire AI lifecycle, covering LLMs, agents, prompts, vector databases, governance frameworks, compliance evidence, policy evaluation, approval workflows, and audit reports, treating AI assets under the same governance umbrella as the data assets beneath them. These production cases illustrate what that looks like in practice: Gorgias centralized tens of thousands of data assets and dramatically reduced time-to-discovery, and Thndr automated PII classification across millions of user accounts with a data team of just six people. The open limitation is enterprise RBAC, which is still maturing, with open feature requests around granular per-property permissions and federated role management, and some reported concerns about stability under high concurrency. This makes it a strong fit for mid-size teams on modern cloud-native stacks who want full feature coverage without the infrastructure weight of the previous option.

Databricks Unity Catalog, in its open source form, was announced as the industry's first open source catalog for data and AI governance across clouds, data formats, and data platforms. It runs under an Apache 2.0 license, built on an OpenAPI spec, and it's compatible with both the Apache Hive metastore API and the Apache Iceberg REST catalog API. It's hosted at LF AI & Data under the Linux Foundation. Its standout feature for agent governance is the Unity AI Gateway, which adds a single entry point for model and agent access, with AI spend budgets tracked by user and team, and enforcement of compliance, safety, audit, and identity controls at the point of query rather than as documentation sitting in a policy binder. This is a strong fit for teams already running on Databricks infrastructure who want open governance paired with cross-cloud, cross-format reach.

Apache Atlas, by contrast, shows exactly where the older generation of catalogs runs into the new criteria. Its stable release, 2.5.0, shipped April 30, 2026, and the project traces back to the Hadoop ecosystem it was originally built for. It holds the strongest classification and tag-propagation capabilities in this comparison, with hierarchical classifications that propagate automatically, a feature unmatched among the others for Hadoop-heavy environments. But on MCP, the gap is real and specific: there's no official first-party MCP server, only unofficial, community-built third-party wrappers carrying no Apache Software Foundation backing. For anyone building agent-native infrastructure, it's the most significant shortfall in this review.

Marquez occupies a narrower role. It's a lightweight, API-first lineage service built around the OpenLineage standard, and it holds Graduated project status under the LF AI & Data Foundation. On MCP, there's no first-party server, and a community proposal has sat open since December 2025 with no committed timeline attached to it. It's the right choice for teams that need a best-in-class lineage layer and plan to build a full catalog on top of it, not a standalone pick for AI agent infrastructure on its own.

Marmot represents the newest and leanest entry in this comparison. It's a single Go binary running on Postgres that becomes an AI context layer the moment it starts, because the MCP server ships as part of that same binary rather than as a separate service to deploy and maintain. It carries the smallest infrastructure footprint of any MCP-native open source option covered here. That makes it a good fit for small teams or greenfield AI-first projects that need MCP-native context exposure without taking on real operational overhead.

Amundsen, built originally at Lyft, earned its reputation as a lightweight catalog with PageRank-based search, historically the simplest option available for small teams. Its maintainers formally archived the project on September 10, 2026. It's included here only for teams already running it in production, not as a new selection for anyone starting fresh, and no credible open source entrant has stepped in to take its place in the simple, small-team category it used to serve.