Book a 30-min call
← cd ../blogs
$ cat posts/iceberg-v3-deletes-your-downstream-cdc.mdx

Iceberg v3 row lineage: the equality-delete trap waiting in your CDC consumer

June 16, 2026 · ImmovableTech Team

  • Data Engineering
  • Real-Time Systems

The sentence in the spec that should change your design

Apache Iceberg v3 makes row lineage mandatory. Every row in a v3 table carries a _row_id, a unique long assigned when the row is first added, and a _last_updated_sequence_number, the sequence number of the commit that last modified it. Read those two fields and you have change data capture straight out of table metadata, with no external tooling to detect what moved. That is the pitch, and it is largely true.

Then there is this, from the Row Lineage section of the spec: “Row lineage does not track lineage for rows updated via Equality Deletes, because engines using equality deletes avoid reading existing data before writing changes and can’t provide the original row ID for the new rows. These updates are always treated as if the existing row was completely removed and a unique new row was added.”

That single paragraph decides whether the feature works for you. The obvious way to consume row lineage is to treat _row_id as a durable identity — the thing you join on, the thing your slowly-changing-dimension logic keys off. On any table whose writer uses equality deletes, that identity is not durable. An update becomes a delete plus an insert with a brand-new _row_id, and nothing in the metadata flags the difference. Your consumer does not fail. It quietly produces the wrong history.

What v3 actually changed

Version 3 of the spec is a types-and-capabilities release. It adds nanosecond-precision timestamps (timestamp_ns and timestamptz_ns), an unknown type, variant for semi-structured payloads, geometry and geography for geospatial data, default values for columns, multi-argument partition and sort transforms, table encryption keys, mandatory row lineage and binary deletion vectors.

Two of those matter operationally on day one. Row lineage is the one everybody writes about. Deletion vectors are the one your storage bill notices.

Deletion vectors are the change you will feel first

In v2, a merge-on-read delete produced a position delete file, and every subsequent commit could add another one against the same data file. Read planning then had to open and merge all of them. v3 replaces that with a deletion vector: a Roaring bitmap of deleted positions for a single data file, stored as a deletion-vector-v1 blob inside a Puffin file, referenced from the delete manifest by file location, offset and length.

The spec makes the useful guarantee explicit — there can be at most one deletion vector for a given data file in a snapshot, and writers must merge new deletes into the existing vector or existing position delete files rather than stacking another file alongside. Deletes against one data file therefore stay O(1) in files rather than growing with commit count, which is exactly the pathology that makes streaming writes expensive. Positions are 64-bit but encoded as 32-bit Roaring bitmaps keyed by the high four bytes, so the common case where a file has fewer than four billion rows costs very little.

AWS’s own framing when it shipped support was that deletion vectors “produce fewer delete files than positional deletes in Iceberg V2” and that compaction of a v3 table should therefore be quicker and cheaper than v2 for a comparable change pattern. Databricks claims up to 10x faster data manipulation than copy-on-write in its announcement, but has not published the workload behind that figure, so treat it as a direction rather than a number you can plan capacity against.

The hole, precisely

Deletion vectors are position deletes. Position deletes identify a row by data file and row position, which means the engine already knows which physical row it is touching — so it can copy the existing _row_id forward and set _last_updated_sequence_number to null for reassignment. The spec spells out those rules for any row moved to a different data file: copy the non-null _row_id, null the sequence number if the write modified the row, keep it if it did not. Lineage survives.

Equality deletes work differently by design. They identify rows by column value — id = 5 — precisely so the writer never has to read the existing data to know what it is deleting. That is what makes them cheap for streaming upserts, and it is also why the writer cannot know the original _row_id. The spec does not paper over this; it says the update is recorded as a removal and an unrelated insertion.

This is not an exotic corner. Flink’s Iceberg sink implements upsert on top of equality fields, and the write path a Debezium-to-Flink pipeline naturally lands on — write.upsert.enabled, primary key declared, one record per changed row — is an equality-delete path. The pipeline shape we described in streaming-first data architecture with CDC and Iceberg is the one most exposed to this, because it was built to avoid reading before writing.

So the table that most wants free CDC out of row lineage is often the table least able to provide it.

What this looked like when we hit it

We upgraded a staging copy of a CDC-fed customer table to v3 expecting to retire a chunk of downstream change-detection logic. The dimension build keyed on _row_id: new id means new member, same id with a higher _last_updated_sequence_number means close the current version and open a new one. Clean, and about forty lines shorter than what it replaced.

The row counts came out right. The history did not. Every customer the Flink upsert path touched appeared in the dimension as a member that had ended and a different member that had begun, with no link between them. A record of a changed address became two people. Because totals reconciled and no job failed, the only reason we caught it in staging was that a “customers who changed tier” count that should have been small was roughly the size of the daily change volume.

The mistake was ours and it was conceptual, not operational. We had read row lineage as a property of the table, when it is a property of the table plus the engine writing to it. Two engines writing the same v3 table with different delete strategies give a downstream consumer two different notions of identity, and the metadata does not distinguish them. We went back to keying on the business key, and now use _last_updated_sequence_number for what it reliably does — telling us which rows a commit touched, so incremental reads do not rescan the table.

Where engine support stood through the first half of 2026

AWS moved first. Amazon EMR 7.12, with Iceberg 1.10, added v3 table format support on 21 November 2025, and on 26 November 2025 AWS announced deletion vectors and row lineage across Spark on EMR 7.12, AWS Glue, SageMaker notebooks, Amazon S3 Tables and the Glue Data Catalog. Glue 5.1, generally available the same day, ships Iceberg 1.10.0 with default column values, deletion vectors for merge-on-read tables, multi-argument transforms and row lineage.

Snowflake made v3 generally available on 7 May 2026, covering variant, geometry, geography, nanosecond timestamps, default values, deletion vectors and row lineage, for both Snowflake-managed and externally managed tables. Reading Snowflake-managed v3 tables from an external engine through its Iceberg REST catalog went GA at the same time; external writes through that catalog did not.

Databricks put Iceberg v3 into public preview in April 2026, on Databricks Runtime 18.0 and above with Unity Catalog enabled, and took it to general availability on 29 May 2026 — deletion vectors, row tracking and VARIANT across managed Iceberg, foreign Iceberg and UniForm-enabled managed tables, alongside GA for Managed Iceberg and Foreign Iceberg in Unity Catalog.

For open source, the Spark 4.0 plus Iceberg 1.10.x combination is the most complete path we have used. Iceberg 1.10.0, released 11 September 2025, added Spark 4.0 support, row lineage in the Avro and Parquet vectorised readers, row lineage preservation through compaction and a fix for lineage inheritance under distributed planning. Iceberg 1.11.0 followed on 19 May 2026 and is the current release. Check your own engine rather than assuming: v3 is a specification, and support for it is per-engine and partial in more places than the announcements suggest.

Before you set format-version = 3

Flipping the property is the easy part, and it is metadata-only — AWS documents that setting format-version = 3 on an existing v2 table upgrades it atomically without rewriting data. To actually get deletion vectors you also need the merge-on-read modes set:

ALTER TABLE catalog.db.orders SET TBLPROPERTIES (
  'format-version' = '3',
  'write.delete.mode' = 'merge-on-read',
  'write.update.mode' = 'merge-on-read',
  'write.merge.mode' = 'merge-on-read'
);

The checks worth doing first are on the consumer, not the table. Audit which delete types your writers actually produce — content type 2 is equality deletes, and their presence tells you row lineage will be discontinuous for those rows:

SELECT content, count(*) AS files
FROM catalog.db.orders.all_delete_files
GROUP BY content;

Then handle the upgrade discontinuity. When a table is upgraded to v3, next-row-id is initialised to 0 and existing snapshots are not modified, so _row_id reads as null for every row in a pre-upgrade snapshot. Row IDs get assigned to existing files by the first snapshot committed after the upgrade. Any consumer that assumes _row_id is non-null will break on historical data, and any consumer that treats the first post-upgrade commit as a change event will see the whole table light up once.

What we’d do differently

We would have read the Row Lineage section of the spec in full before designing anything on top of it, rather than reading a summary of what v3 adds and inferring the semantics. The constraint is stated plainly, in one paragraph, in the primary source. Every secondary description of row lineage we read beforehand described _row_id as an identifier that never changes, which is true only for writers that do not use equality deletes, and none of them said so.

We would also have run the old and new change-detection logic side by side over the same commits before deleting anything, rather than replacing the logic and checking the output. Comparing outputs catches disagreements. Checking output catches only failures, and this failure mode does not produce one.

References

We design lakehouse and change-data-capture pipelines as part of our Data Engineering practice. Talk to us if you are planning an Iceberg v3 upgrade with downstream consumers that depend on row identity.