Apache Iceberg Table Format Interview Preparation Guide

The primary difference lies in how table state and metadata are managed. Apache Hive relies on directory listings and file globbing on HDFS or cloud object storage, which makes operations like schema alterations, partitioning updates, and atomic transactions fragile and prone to data corruption. In contrast, Apache Iceberg uses a hierarchical metadata tree consisting of table metadata JSON files, manifest lists, and manifest files, enabling true ACID transactions, hidden partitioning, safe schema evolution, and time travel without directory-level renames.

In traditional Hive tables, users must explicitly include partition columns in query predicates (e.g., WHERE date = '2026-04-01') and ensure data is inserted into matching directory paths. If a business requirement changes from daily to hourly partitioning, the entire table must be rewritten. Iceberg solves this via hidden partitioning by automatically managing partition transforms (such as days, hours, or buckets) derived from source columns in the table schema. Compute engines automatically evaluate partition pruning based on logical filters, and partition specs can evolve over time without rewriting historical data files.

Schema evolution in Iceberg allows users to instantly add, drop, rename, or reorder table columns without rewriting underlying data files. Iceberg assigns a unique integer ID to every column when it is created. When queries execute, Iceberg resolves column identity using these field IDs rather than relying on column positions or names. This ensures that historical Parquet files lacking newly added columns or containing older column names are read safely, returning null for missing fields without causing read failures or silent data corruption.

Optimistic Concurrency Control allows multiple writers to append data and manifest files to an Iceberg table simultaneously without locking the storage layer. When a writer finishes writing data files, it attempts to commit its transaction by updating the table metadata pointer in the catalog. If another writer committed a conflicting update in the interim, Iceberg's commit conflict resolution logic checks whether the newly added files overlap with the current table state. If there is no conflict, the commit succeeds; otherwise, the transaction retries with exponential backoff.

Copy-on-Write rewrites entire data files whenever rows are updated or deleted, resulting in higher write latency and write amplification but delivering maximum read performance because no extra files need evaluation. Merge-on-Read prioritizes fast write throughput by appending small delta files (positional or equality deletes) when records change, resolving updates and deletes lazily at read time using compute engine vectorization, which can introduce read amplification.

Manifest files are discrete Avro files that track individual data files (Parquet or ORC) belonging to a table snapshot, storing column-level min/max statistics and partition value bounds. Manifest lists are higher-level Avro files that group multiple manifest files together for a specific table snapshot, summarizing their partition bounds. During query planning, compute engines read the manifest list to quickly prune irrelevant manifest files, and then read the surviving manifests to skip irrelevant data files without scanning raw storage blocks.

Time travel in Iceberg is made possible by maintaining an immutable lineage of table snapshots recorded in the table metadata JSON file. Each snapshot has a unique ID and commit timestamp. Users can query the exact state of a table at a specific historical timestamp or snapshot ID using clauses like FOR SYSTEM_TIME AS OF or by referencing snapshot IDs. Rolling back a table is an instantaneous metadata-only operation where the current table metadata pointer is updated to point to a prior valid snapshot without modifying or moving any physical data files.

Small file accumulation occurs frequently in streaming ingest pipelines (such as Apache Flink micro-batch writers) that commit thousands of tiny Parquet files every few seconds. This bloats the metadata tree, overwhelms query planners with large manifest lists, and causes object storage API throttling. This is resolved by scheduling regular background maintenance jobs using Spark procedures like rewrite_data_files with binpack or sort strategies to consolidate small files into optimal 128MB to 512MB blocks.

The Iceberg REST Catalog provides a standardized, lightweight service protocol that abstracts catalog operations from underlying storage systems and database implementations. It has become an industry standard in 2026 because it guarantees multi-engine interoperability (allowing Spark, Trino, Flink, and Athena to share the exact same table state seamlessly), enforces strict atomic commit protocols, and prevents vendor lock-in by separating catalog logic from proprietary cloud metastores.

When running expire_snapshots and remove_orphan_files, engineers must carefully configure retention grace periods (typically 7 days or more). Setting excessively short retention windows can lead to catastrophic data loss if active concurrent queries, long-running streaming jobs, or staging branches attempt to access data files or snapshots that have been prematurely purged from cloud object storage. Always ensure maintenance windows account for maximum query execution times and multi-day data pipeline schedules.