Every figure a business reports eventually draws the same question: why is this different from last month's? Sometimes a CFO asks it, sometimes an auditor, sometimes a board member who read the last deck more closely than anyone expected. The question is not hostile, and it never goes away. For thirty years the answer was a person who could reconstruct the derivation on the spot: the analyst who knew the figure came from the reconciled mart, that the definition of "active" changed in March, that one region joined the Nordics this year.

Now picture the figure produced by an agent that wrote its own , joined three tables, applied a filter it judged reasonable, and returned a clean number to no decimal places. The question arrives as usual. Can you reconstruct what happened? If the answer is no, your analytics is a confident guess that survived a meeting.

The unglamorous part nobody wants to own

Lineage is the record of where data came from and what was done to it on the way to the number in front of you. Table-to-table lineage, this mart is built from those three sources, is the version most tools draw, because it is the easy one: it falls out of parsing the transformation code. Column-level lineage, and increasingly value-level, is the harder and more useful kind: this measure is this column, aggregated this way, filtered by that definition, as of this load.

It is dull to maintain and almost nobody wants to own it, which is why it is the first thing skipped and the most painfully missed. It has always been treated as an engineering convenience, the thing you reach for when a pipeline breaks and you need to find what fed the broken table. That framing undersells it. Without lineage, a number is just a claim. With it, you can show where the number came from and why it holds.

And it only defends the number if it is unbroken. A trail that covers the transformation step but loses the ingestion before it, or the dashboard after it, tells you the middle of a story whose beginning and end are missing. Lineage is the one artifact that has to run the whole length of the path, from the source system to the answer on the slide, or it does not do its job. That completeness is what makes it an audit trail at all, and it is the first thing an assembled toolchain gives up.

A lone retro data-processing console emitting one continuous unbroken ribbon of printout that runs the full width of the scene, from a stack of tape reels to a single finished report card.
The audit trail as one unbroken ribbon: from the source reels at the back to the finished number in front, kept whole by a single machine. Author's illustration.

Lineage used to live in someone's head

Lineage was easy to neglect for two decades because the people who needed it already had it.

The consumer of lineage used to be an engineer, usually the one who built the pipeline in the first place. Ask them what feeds the revenue mart and they would tell you without looking, because they wrote the job. Change a source column and they could do the in the time it took to say it out loud, because the dependency graph was already in their head. On most teams the analyst who built the dashboard sat a few desks away, or was the same person. The derivation of any number was never more than a short conversation from whoever produced it.

So for a long time few teams built anything to capture it, and they were not wrong not to. Why maintain an expensive graph of something a colleague could recite from memory? The humans were the lineage system: undocumented, unversioned, and entirely dependent on those specific people staying in their jobs. Most organisations were happy to make that trade, usually without noticing they had made it.

It cracked once before AI. As went self-service, data landed in front of people who had never met the pipeline, and "ask whoever built it" stopped scaling. Five and more years ago, in the years and the early days of , developers on sprawling estates commonly built their own lineage apps: scrape the , read the file headers, and hand users the resulting flow as its own app or right inside the one they were using, half self-service enablement and half living documentation. That was real, standing, hand-built work, the tedious kind, and for a while plenty of teams did it because the platform would not.

That has mostly fallen out of practice since. Qlik folded lineage into the analytics layer after buying the lineage vendor in 2021 , so the home-grown apps stopped being necessary there. grew a lineage view and a catalog that do a version of the same thing , though usually scoped to the BI layer rather than the whole path.

Every version of that, though, is still design-time lineage drawn for a human reader. AI is the second wave, and it breaks the arrangement in a way self-service never did. That arrangement had always rested on one quiet assumption: a person in the loop, holding the thread. Self-service only moved the person. AI removes them, and when you pull the person out, the informal system goes too.

What the agent changes

Taking the human out of the loop is the promise of , so the narration now has to come from somewhere else. Nobody holds the derivation in their head anymore, because nobody built the query. The dashboard was assembled, the question answered, the analysis written by an agent, and an agent does not remember the way a colleague does. It will produce a fluent account of its reasoning on demand, but treat that as a plausible story about what the agent might have done, written in the same confident register as the answer itself, and wrong in precisely the ways that matter.

The trouble bring is narrower than distrust in principle: they remove the one component that used to supply trust by hand, and leave nothing in its place unless you put something there on purpose. What used to live in a head now has to live in a system, or it does not exist at all.

"I trust it" was never an acceptable answer

I learned this discipline on a public pension reform programme, doing QA and delivery control around the part of the delivery . In that world, "I trust the number" is not a sentence you are allowed to say. Pensions are computed from rules, the rules are auditable, and at every stage someone could be asked to show how a figure was derived and to produce the test evidence that it was derived correctly. , a test apparatus during production test periods, documented deployment and contingency plans: at the time it felt like heavy process wrapped around what was, underneath, an job.

It was heavy. It was also the exact muscle that regulated work builds and most commercial analytics never has to. The instinct that you must be able to show the derivation rather than assert it turned out to be the template for what every organisation needs the moment an agent, rather than a known person, is producing the answer. The public sector got there first because it had no choice.

There was a second reason that trail held, and I did not appreciate it at the time. One delivery owned the whole chain. The ingestion, the transformations, the reconciliation, the test evidence, the reporting were a single accountable line, not a relay between five vendors. When you have to show a derivation end to end, it helps enormously that one owner is responsible for every hop of it. The moment the chain is stitched together from separately-owned pieces, the trail acquires seams, and seams are where trails break.

Two lineages people conflate

There is a distinction the tooling conversation keeps blurring.

There is design-time lineage, the picture of how the model is built. Which sources feed which staging tables, which marts derive from which, which measure is defined on which column. This is static. It describes the system as designed, and it is what a gives you. It is useful, and most teams still do not have it.

There is , the record of which specific rows, which definition version, which filter, and which join produced this answer to this question, at the time it was asked. This is dynamic. It describes one event rather than the system, and it is what an agent's answer needs, because the auditor is asking about the number on slide nine, and only about that.

The same model, two different things asked of it. The catalog draws the whole graph on the left; the auditor wants the single navy path on the right, the one this answer took. Author's illustration.

Most catalogs give you the left panel and call the job done. Agents need the right one, and the right one is harder by an order of magnitude, because it has to be captured at the moment the answer is produced rather than described once when the model is designed.

A plausible answer without provenance

An AI answer you cannot trace is fine for a curiosity, a hunch, a thing you will sanity-check yourself anyway. It is unacceptable for anything a regulator, an auditor, or a board will later ask you to stand behind. The two can be equally good answers. What separates them is whether you can show the derivation when someone asks.

The trail dies at the seams

Put those two ideas together. Lineage has to span the whole path, and the improvising consumer at the end of it is a machine that remembers nothing. Where does the trail come from?

The answer depends entirely on how you built the platform underneath.

Assemble the fashionable stack and you get lineage in slices. The loader knows what it ingested. knows the transformations it ran, and can emit column-level lineage for them through . The catalog knows the table graph. The BI tool knows which dashboard reads which dataset. Each of these is real, each is partial, and none of them owns the handoff to the next. The trail exists in four vendors' logs, in four schemas, with a gap at every boundary where one tool stops recording and the next has not started.

You can close those gaps. The does have an answer: OpenLineage plus a catalog whose entire job is to collect everyone's lineage events and stitch them into one graph. It works. It is also a standing, never-finished piece of engineering that you have to staff and maintain, and even then the coverage is uneven. OpenLineage's column-level support is solid for dbt, partial for and , and absent for some engines entirely . Stitching slices into a whole is a tax you pay, over and over, for the privilege of having picked the sharpest tool at each layer.

Each tool records only its own slice, with a gap at every seam and the agent's improvised query recorded by none. The continuous trail an answer needs is the one thing no single component owns. Author's illustration.

To recover the end-to-end trail that a single platform would have produced as a byproduct, you have added another product to the stack, another vendor, another integration surface. The lineage tool is itself a seam. You solved the problem of too many seams by introducing one more and pointing it at all the others.

When lineage is just there

The integrated platform makes a different bargain. Because one system owns ingestion, transformation, the catalog, and the analytics layer, the lineage arrives as a byproduct of the platform knowing what it did, with no after-the-fact stitching. You do not build the graph; you read it. Nobody is assigned to maintain the seams, because the seams are internal and the vendor owns them.

I work with this platform, so discount for that if you like. I still think Qlik has the right idea with , and so far it has executed unusually well on it. The catalog tracks end-to-end lineage across integration, transformation, and analytics, down to column and feature level, without anyone wiring it up . The agentic-data-engineering release that went generally available at the end of June 2026 puts AI agents to work inside that same governed environment, with -enabled workflows that, in Qlik's framing, preserve governance, lineage, and control rather than bolting an assistant onto the side . Its CEO put the problem more bluntly than most vendors manage: too many AI tools, in his words, "attempt to provide answers without explaining how or why" . That is the whole argument in a sentence, from a company whose architecture can back it up.

Qlik is not the only one who sees this, and each of the others places the bet differently.

Palantir takes the idea to its logical extreme. is about as integrated as software gets: ingestion, transformation, an that acts as the , and the operational apps on top, all in one system where every object is versioned and auditable and lineage is how the platform works . Its AIP layer runs agents against that Ontology, and its 2026 releases went further than anyone else by putting the agents themselves into the lineage graph as first-class nodes, so an agent's action becomes something the dependency graph records rather than something that happens off to the side . It is the closest anyone has come to solving the run-time- problem, and it is possible precisely because Palantir owns the entire estate. The catch is that Foundry is heavy, expensive, and opinionated, a platform you commit an organisation to rather than one you slot in beside what you have. It proves the architectural point without being the thing most companies will deploy.

The rest sit at different distances from that ideal. captures column-level lineage automatically in , across notebooks, jobs, and , with no configuration ; the instinct is exactly right, but Databricks grew up from the engineering side of the house and is not yet the packaged ingestion-to-analyst platform Qlik is. is integrated by design and improving fast, still maturing into the promise. And , of them all, already had every piece: catalog, integration, governance, quality, master data management, the full kit. closed its roughly $8 billion acquisition of it in November 2025 and pitched it as the data foundation for agentic AI . On the record of what Salesforce has done with nearly everything it has bought on the data and analytics side over the years, I would not bet on it. I would be glad to be wrong.

Line them up and the pattern has little to do with which vendor is cleverest. What matters is who owns the path. Where a single platform owns ingestion through analytics, lineage tends to be automatic and complete. Where the path is assembled from separately-owned parts, lineage is achievable but manual, and manual work is exactly what does not get done.

The part still unsolved

Even with the right platform, one piece stays hard.

Run-time provenance is tractable when the agent stays on rails. If it calls a defined metric through a governed semantic layer and returns the result, you can log which metric, which parameters, which load. That path is recoverable, and an integrated platform gives you a real chance of recovering it, because the metric and the log live in the same system.

The trouble starts when the agent improvises, which is the thing it is good at and the reason you wanted it. It writes a join nobody modelled. It picks the raw table over the reconciled one because the reconciled one was a few hours stale, and it reasoned, not unreasonably, that fresher was better. None of that improvised path exists in any design-time lineage graph, because it was invented at query time and will be invented slightly differently next time. The agent has created lineage that no catalog captured, and unless something recorded the actual calls, the joins, the filters, the table versions at the instant of answering, that derivation is gone the moment the response comes back.

When I first drafted this I wrote that no product on the market captured it. That has already dated. A whole discipline has grown up around exactly this problem: MCP audit logging that records every tool call an agent makes, conventions for tracing agent steps, a crop of agent-observability tools whose pitch is a full trace of what the agent touched . The plumbing is arriving fast.

The gap now is not capture, it is connection.

Capture is not connection

You can buy a full trace of every tool call an agent makes, and separately maintain a lineage graph of your data. Having both is not the same as having one view that reconstructs how a specific answer was produced. Today those two records live in different systems that do not talk to each other.

That instrumentation grew up in the observability and security world, built by different teams for different buyers than the data catalog was. So you end up with no single view that says this answer came from these rows, through these definitions, via these agent steps. Palantir putting agents into the dependency graph is the rare case where the two are joined, and it is available to you only once you have handed Palantir the whole estate. For everyone else the joining is the open problem, and it is far more open in an assembled stack, because there is no single graph to join anything into.

The temptation to skip it, and why that is backwards

Lineage loses every prioritisation argument it enters. It ships no feature, it demos badly, and the cost of not having it is invisible right up until the day it is not. So it gets deferred, and the agentic-analytics demo goes ahead without it, because the demo only ever asks questions whose answers nobody is going to challenge.

That ordering was defensible when a human sat in the gap and could reconstruct any number on request. It is backwards once the decisions are automated. The more you let an agent act without a person re-checking its work, the more the traceability of that work is the only thing standing between you and an answer you cannot defend. Automation raises the value of the audit trail in direct proportion to how much human checking it removes.

The platform choice folds into the same logic. Pick the assembled stack and you defer the bill twice: once on lineage itself, and once on the standing work of stitching lineage across the seams you chose to own. Both come due at the same meeting, the one where someone important asks how the agent got its number. Skipping lineage because it is unglamorous is choosing to discover, in front of an auditor, that you built a system that cannot explain itself.

Nobody's holding the trail now

Lineage gets filed under compliance, a tax bolted onto analytics after the fact. Its real job is the audit trail that turns a plausible AI answer into a defensible one, and the two are indistinguishable on the surface, which is the danger.

A defensible answer and a lucky guess look identical until someone asks you to show your work.

For thirty years that trail lived in a person's head, and the arrangement worked because there was always a person. The quiet event of the last two years is that we took the person out of the middle of the work and did not replace what they were silently doing, which was holding the thread from the source to the number. Something has to hold it now. It can be a lineage graph you build and maintain by hand across a pile of tools, or it can be a platform that holds it for you because it owns every hop. What it cannot be is nobody, and nobody is the default you back into by not choosing.

The cost of not choosing is real but deferred, and deferred costs are the ones organisations price worst. You pay it in full the first time someone important asks "are you sure?" and the only thing you have to offer is the agent's word, delivered with total confidence and no way to check it.

Final thought

Build the trail before that meeting, not during it.

Lineage Is the Audit Trail for AI Work · July 2026 · An opinion piece. The pension delivery-control engagement is from my own work; the design-time versus run-time distinction, the automatic-versus-tedious read on integrated platforms, and the call on where the unsolved work sits are my own argument. Vendor capabilities are cited; the judgement about who executes on them is mine.

References12
  1. 1Henning Naarlien-Tolpinrud, "Public Pension Delivery Control", experience page. henning.tolpinrud.no ↗ Accessed 2026-07-01
  2. 2Qlik / Talend, "Lineage", Talend Data Catalog User Guide (8.1). help.qlik.com ↗ Accessed 2026-07-01
  3. 3Business Wire, "Qlik Delivers Agentic Data Engineering in Qlik Cloud to Help Enterprises Build Trusted Data for AI", June 30, 2026. businesswire.com ↗
  4. 4SiliconANGLE, "Qlik debuts new agentic capabilities, aiming to enhance AI trust and transparency", April 15, 2026. siliconangle.com ↗
  5. 5OpenLineage, "OpenLineage: an open framework for data lineage collection and analysis" (LF AI & Data Foundation graduate project). openlineage.io ↗ Accessed 2026-07-01
  6. 6Databricks, "Data lineage in Unity Catalog", product documentation. docs.databricks.com ↗ Accessed 2026-07-01
  7. 7Salesforce, "Salesforce Completes Acquisition of Informatica", November 18, 2025. salesforce.com ↗
  8. 8Palantir, "Data Lineage: Overview" and "AIP: Overview", Foundry documentation. palantir.com ↗ Accessed 2026-07-01
  9. 9Tetrate, "MCP Audit Logging: Tracing AI Agent Actions for Compliance". tetrate.io ↗ Accessed 2026-07-01
  10. 10Qlik, "Qlik Acquires NodeGraph to Enhance End-to-End Analytics Data Pipelines With Interactive Data Lineage and Drive 'Explainable BI'", August 12, 2021. qlik.com ↗
  11. 11Microsoft, "Data lineage" (Power BI service documentation). learn.microsoft.com ↗ Accessed 2026-07-01
  12. 12Tableau, "Use Lineage for Impact Analysis", Tableau Help. help.tableau.com ↗ Accessed 2026-07-01