Introduction

Managing petabyte-scale data lakes often leads to a choice between the rigidity of traditional data warehouses and the chaos of raw file-based storage. Apache Iceberg solves this by bringing the reliability and simplicity of SQL tables to big data, allowing multiple compute engines to safely work with the same tables simultaneously. With thousands of GitHub stars and widespread industry adoption, Apache Iceberg has become the foundational open table format for the modern data lakehouse architecture.

What Is Apache Iceberg?

Apache Iceberg is an open table format for huge analytic datasets that provides a high-performance way to manage large-scale tables in a data lake. It is designed to be engine-agnostic, meaning it allows compute engines like Apache Spark, Trino, Apache Flink, Presto, Hive, and Impala to interact with the same data using a standardized SQL-like interface. Maintained by the Apache Software Foundation and licensed under the Apache License 2.0, it replaces the legacy Hive table format by tracking data at the file level rather than the folder level.

Unlike traditional data lakes where a table is defined as a directory of files, Iceberg defines a table as a canonical list of files with metadata on those files themselves. This architecture allows for atomic commits, full read isolation, and multiple concurrent writers using optimistic concurrency control.

Why Apache Iceberg Matters

Before Iceberg, data engineers struggled with “zombie” data, broken schema evolution, and the need for manual partition management. In a traditional Hive-style data lake, adding a column or changing a partition required rewriting entire tables or risking data corruption. This made it nearly impossible to maintain data integrity at the petabyte scale.

Apache Iceberg matters because it decouples the table definition from the physical layout of the data. By introducing a metadata layer, it enables ACID transactions on top of object stores like S3, GCS, and Azure Blob Storage. This allows organizations to treat their data lake as a true database, enabling features like time travel and atomic updates without the cost of a proprietary data warehouse.

The project’s traction is evident in its massive adoption by companies like Netflix (where it was created) and its integration into almost every major cloud data tool. It fills the critical gap between the flexibility of a data lake and the reliability of a relational database.

Key Features

• Full Schema Evolution: Iceberg allows you to add, rename, update, or drop columns without rewriting the entire table. These changes are metadata-only operations that do not affect the underlying data files.
• Hidden Partitioning: Iceberg handles the production of partition values automatically. Users do not need to know the partitioning scheme to write fast queries, and the layout can be updated as data volume or query patterns change without requiring a table rewrite.
• Time Travel and Rollbacks: Because Iceberg tracks snapshots of the table, users can query the state of a table at a specific point in time or quickly roll back a table to a previous healthy state after a failed write.
• ACID Transactions: Iceberg ensures that table changes are atomic. Readers never see partial or uncommitted changes, and multiple concurrent writers can commit changes safely using optimistic concurrency.
• High-Performance Scan Planning: Scan planning is fast because it uses table metadata to prune data files based on partition and column-level statistics, avoiding the need for expensive directory listings in cloud object stores.
• Engine Agnostic: Iceberg provides a specification that ensures compatibility across different languages and implementations, allowing Spark, Flink, Trino, and others to work together on the same dataset.

How Apache Iceberg Compares

Apache Iceberg competes primarily with Delta Lake and Apache Hudi. While all three provide ACID transactions and time travel, they differ in their architectural approach to metadata and partitioning.

The primary differentiator for Apache Iceberg is its partition evolution. In Delta Lake or Hudi, changing how a table is partitioned usually requires a full rewrite of the data. In Iceberg, this is a metadata-only operation. This is critical for organizations whose data patterns change over time.

While Delta Lake is deeply integrated with the Databricks ecosystem, Iceberg is designed from the ground up to be a community standard, making it a more attractive choice for those avoiding vendor lock-in. Apache Hudi is often preferred for near real-time streaming and Change Data Capture (CDC) pipelines due to its specialized indexing and upsert capabilities.

Getting Started: Installation

Apache Iceberg is a library, not a standalone server. You integrate it into your compute engine of choice. The most common way to get started is via Apache Spark.

Using Apache Spark

To include Iceberg in your Spark installation, add the iceberg-spark-runtime JAR to your Spark jars folder or use the --packages flag when starting Spark:

spark-shell --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.0

You must also configure the Spark session to use the Iceberg Spark extensions and a catalog. For example, to create a local Hadoop-based catalog:

--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \n--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \n--conf spark.sql.catalog.local.type=hadoop \n--conf spark.sql.catalog.local.warehouse=/tmp/warehouse

Using Docker Compose

The fastest way to experience Iceberg is using the official tabulario/spark-iceberg image, which provides a pre-configured environment with Spark, a REST catalog, and MinIO for storage.

docker compose up -d

This spins up a local lakehouse in a box, including a REST catalog for metadata management and MinIO as an S3-compatible storage layer.

How to Use Apache Iceberg

Once your environment is configured, you can interact with Iceberg tables using standard SQL. The workflow typically involves creating a table, writing data to it, and then querying it.

The basic workflow is as follows: first, you define the table schema and partitioning. Then, you use INSERT INTO or MERGE INTO to populate the data. Because Iceberg handles partitioning automatically (hidden partitioning), you don’t need to create separate columns for your partition values.

If you are using a REST catalog, the compute engine communicates with the REST server to get the current snapshot of the table. The engine then reads the manifest files from the storage layer to identify which data files are needed for a specific query, drastically reducing the amount of data scanned.

Code Examples

The following examples demonstrate common Iceberg operations using Spark SQL.

Creating a Table

CREATE TABLE local.db.taxis ( 
  id bigint, 
  vendor_id int, 
  trip_distance double, 
  fare amount double 
) USING iceberg 
PARTITIONED BY (days(trip_start))

This example creates an Iceberg table partitioned by day. Note that the days() function is used for hidden partitioning, which means the users don’t need to create a separate date column for the partition.

Updating Data with MERGE INTO

MERGE INTO local.db.taxis t 
USING (SELECT * FROM staging.nyc_taxis) s 
ON t.id = s.id 
WHEN MATCHED THEN UPDATE SET * 
WHEN NOT MATCHED THEN INSERT *

The MERGE INTO command is a powerful tool for handling upserts. It allows you to combine new data from a staging table into the production table, updating existing records and inserting new ones in a single atomic transaction.

Time Travel Query

SELECT * FROM local.db.taxis FOR SYSTEM_TIME AS OF '2023-10-01 12:00:00'

This allows you to query the table as it existed at a specific point in time, which is essential for auditing, reproducing results, and debugging data pipelines.

Real-World Use Cases

Apache Iceberg shines in environments where data volume is massive and data integrity is non-negotiable.

• Regulatory Compliance (GDPR/CCPA): For companies needing to perform targeted deletes of specific user data to comply with privacy laws, Iceberg’s ACID transactions and row-level deletes make this process efficient without requiring a full table rewrite.
• Machine Learning Feature Stores: ML teams use Iceberg to version their training datasets. By using time travel, they can ensure that a model is trained on the exact same snapshot of data, ensuring reproducibility of ML experiments.
• Financial Auditing: In financial services, the ability to roll back a table to a previous state or query a historical snapshot is critical for auditing and regulatory reporting.
• Petabyte-Scale Analytics: Organizations managing tens of petabytes of data use Iceberg to avoid the “directory listing” problem of Hive, where querying a table with millions of files would crash the metastore.

Contributing to Apache Iceberg

Apache Iceberg is an active open-source project governed by the Apache Software Foundation. Contributions are welcome and the project prefers to receive contributions as GitHub pull requests.

To get started, you can look for “good first issues” on GitHub to find approachable tasks for first-time contributors. The project maintains a detailed set of contributing guidelines and a Code of Conduct to ensure a professional environment. Specification changes—such as changes to the format or the REST catalog API—are handled through the Iceberg Improvement Proposal (IIP) process to ensure community consensus before code is merged.

Community and Support

The Apache Iceberg community is vibrant and active. Official support and documentation are available through the following channels:

• Official Documentation: The primary source of truth for the project is Iceberg Documentation.
• GitHub Discussions: Technical questions and bug reports are tracked via the Apache Iceberg GitHub Repository.
• Mailing Lists: Community discussions and development decisions are coordinated through the dev@iceberg.apache.org mailing list.
• Slack: The community maintains an active Slack channel for real-time support and peer-to-peer help.

Conclusion

Apache Iceberg is the right choice for organizations that have outgrown the traditional Hive-style data lake and need the reliability of a SQL database on top of their object storage. It is particularly powerful for those who need seamless schema evolution, atomic commits, and the ability to query data across multiple different compute engines without vendor lock-in.

While it is a highly scalable format, it is not a replacement for a traditional relational database for OLTP workloads. It is designed for OLAP (analytic) workloads where read performance and metadata management at scale are the primary concerns.

Star the repo, try the quickstart with Docker Compose, and join the community to start building your modern data lakehouse.

Resources

• Apache Iceberg GitHub Repository
• Official Iceberg Documentation
• Apache Iceberg 101 Guide