Iceberg Rust

iceberg-rust is a Rust implementation for managing Apache Iceberg tables.

What is Apache Iceberg?

Apache Iceberg is a modern, high-performance open table format for huge analytic datasets that brings SQL-like tables to processing engines including Spark, Trino, PrestoDB, Flink, Hive and Impala.

Iceberg provides a metadata layer that sits on top of formats like Parquet and ORC, ensuring data is organized, accessible, and safe to work with at scale. It introduces features long expected in databases such as transactional consistency, schema evolution, and time travel into environments where files are stored directly on systems like Amazon S3.

Install

Cargo 1.75.0 or later is required to build.

Add iceberg and iceberg-catalog-rest into Cargo.toml dependencies:

iceberg = "0.6.0"
iceberg-catalog-rest = "0.6.0"

using cargo add:

$ cargo add iceberg iceberg-catalog-rest

iceberg is under active development, you may want to use the git version instead:

iceberg = { git = "https://github.com/apache/iceberg-rust", rev = "commit-hash" }

Apache Iceberg™ Rust Downloads

The official Apache Iceberg-Rust releases are provided as source artifacts.

Releases

Find the latest source release at the Apache CDN. For release notes, see the GitHub releases page.

For older releases, please check the archive.

Notes

  • When downloading a release, please verify the OpenPGP compatible signature (or failing that, check the SHA-512); these should be fetched from the main Apache site.
  • The KEYS file contains the public keys used for signing release. It is recommended that (when possible) a web of trust is used to confirm the identity of these keys.
  • Please download the KEYS as well as the .asc signature files.

To verify the signature of the release artifact

You will need to download both the release artifact and the .asc signature file for that artifact. Then verify the signature by:

  • Download the KEYS file and the .asc signature files for the relevant release artifacts.

  • Import the KEYS file to your GPG keyring:

    gpg --import KEYS

  • Verify the signature of the release artifact using the following command:

    gpg --verify <artifact>.asc <artifact>

To verify the checksum of the release artifact

You will need to download both the release artifact and the .sha512 checksum file for that artifact. Then verify the checksum by:

shasum -a 512 -c <artifact>.sha512

Catalog

Catalog is the entry point for accessing iceberg tables. You can use a catalog to:

  • Create and list namespaces.
  • Create, load, and drop tables

There is support for the following catalogs:

CatalogDescription
Restthe Iceberg REST catalog
Gluethe AWS Glue Data Catalog
Memorya memory-based Catalog
HMSApache Iceberg HiveMetaStore catalog
S3TablesAWS S3 Tables
SQLSQL-based catalog

Not all catalog implementations are complete.

RestCatalog

Here is an example of how to create a RestCatalog:

// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.

use std::collections::HashMap;

use iceberg::{Catalog, CatalogBuilder, NamespaceIdent};
use iceberg_catalog_rest::{REST_CATALOG_PROP_URI, RestCatalogBuilder};

static REST_URI: &str = "http://localhost:8181";

/// This is a simple example that demonstrates how to use [`RestCatalog`] to create namespaces.
///
/// The demo creates a namespace and prints it out.
///
/// A running instance of the iceberg-rest catalog on port 8181 is required. You can find how to run
/// the iceberg-rest catalog with `docker compose` in the official
/// [quickstart documentation](https://iceberg.apache.org/spark-quickstart/).
#[tokio::main]
async fn main() {
    // Create the REST iceberg catalog.
    let catalog = RestCatalogBuilder::default()
        .load(
            "rest",
            HashMap::from([(REST_CATALOG_PROP_URI.to_string(), REST_URI.to_string())]),
        )
        .await
        .unwrap();

    // List all namespaces already in the catalog.
    let existing_namespaces = catalog.list_namespaces(None).await.unwrap();
    println!("Namespaces alreading in the existing catalog: {existing_namespaces:?}");

    // Create a new namespace identifier.
    let namespace_ident =
        NamespaceIdent::from_vec(vec!["ns1".to_string(), "ns11".to_string()]).unwrap();

    // Drop the namespace if it already exists.
    if catalog.namespace_exists(&namespace_ident).await.unwrap() {
        println!("Namespace already exists, dropping now.",);
        catalog.drop_namespace(&namespace_ident).await.unwrap();
    }

    // Create the new namespace in the catalog.
    let _created_namespace = catalog
        .create_namespace(
            &namespace_ident,
            HashMap::from([("key1".to_string(), "value1".to_string())]),
        )
        .await
        .unwrap();
    println!("Namespace {namespace_ident:?} created!");

    let loaded_namespace = catalog.get_namespace(&namespace_ident).await.unwrap();
    println!("Namespace loaded!\n\nNamespace: {loaded_namespace:#?}",);
}

You can run following code to list all root namespaces:

// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.

use std::collections::HashMap;

use iceberg::{Catalog, CatalogBuilder, NamespaceIdent};
use iceberg_catalog_rest::{REST_CATALOG_PROP_URI, RestCatalogBuilder};

static REST_URI: &str = "http://localhost:8181";

/// This is a simple example that demonstrates how to use [`RestCatalog`] to create namespaces.
///
/// The demo creates a namespace and prints it out.
///
/// A running instance of the iceberg-rest catalog on port 8181 is required. You can find how to run
/// the iceberg-rest catalog with `docker compose` in the official
/// [quickstart documentation](https://iceberg.apache.org/spark-quickstart/).
#[tokio::main]
async fn main() {
    // Create the REST iceberg catalog.
    let catalog = RestCatalogBuilder::default()
        .load(
            "rest",
            HashMap::from([(REST_CATALOG_PROP_URI.to_string(), REST_URI.to_string())]),
        )
        .await
        .unwrap();

    // List all namespaces already in the catalog.
    let existing_namespaces = catalog.list_namespaces(None).await.unwrap();
    println!("Namespaces alreading in the existing catalog: {existing_namespaces:?}");

    // Create a new namespace identifier.
    let namespace_ident =
        NamespaceIdent::from_vec(vec!["ns1".to_string(), "ns11".to_string()]).unwrap();

    // Drop the namespace if it already exists.
    if catalog.namespace_exists(&namespace_ident).await.unwrap() {
        println!("Namespace already exists, dropping now.",);
        catalog.drop_namespace(&namespace_ident).await.unwrap();
    }

    // Create the new namespace in the catalog.
    let _created_namespace = catalog
        .create_namespace(
            &namespace_ident,
            HashMap::from([("key1".to_string(), "value1".to_string())]),
        )
        .await
        .unwrap();
    println!("Namespace {namespace_ident:?} created!");

    let loaded_namespace = catalog.get_namespace(&namespace_ident).await.unwrap();
    println!("Namespace loaded!\n\nNamespace: {loaded_namespace:#?}",);
}

Then you can run following code to create namespace:

// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.

use std::collections::HashMap;

use iceberg::{Catalog, CatalogBuilder, NamespaceIdent};
use iceberg_catalog_rest::{REST_CATALOG_PROP_URI, RestCatalogBuilder};

static REST_URI: &str = "http://localhost:8181";

/// This is a simple example that demonstrates how to use [`RestCatalog`] to create namespaces.
///
/// The demo creates a namespace and prints it out.
///
/// A running instance of the iceberg-rest catalog on port 8181 is required. You can find how to run
/// the iceberg-rest catalog with `docker compose` in the official
/// [quickstart documentation](https://iceberg.apache.org/spark-quickstart/).
#[tokio::main]
async fn main() {
    // Create the REST iceberg catalog.
    let catalog = RestCatalogBuilder::default()
        .load(
            "rest",
            HashMap::from([(REST_CATALOG_PROP_URI.to_string(), REST_URI.to_string())]),
        )
        .await
        .unwrap();

    // List all namespaces already in the catalog.
    let existing_namespaces = catalog.list_namespaces(None).await.unwrap();
    println!("Namespaces alreading in the existing catalog: {existing_namespaces:?}");

    // Create a new namespace identifier.
    let namespace_ident =
        NamespaceIdent::from_vec(vec!["ns1".to_string(), "ns11".to_string()]).unwrap();

    // Drop the namespace if it already exists.
    if catalog.namespace_exists(&namespace_ident).await.unwrap() {
        println!("Namespace already exists, dropping now.",);
        catalog.drop_namespace(&namespace_ident).await.unwrap();
    }

    // Create the new namespace in the catalog.
    let _created_namespace = catalog
        .create_namespace(
            &namespace_ident,
            HashMap::from([("key1".to_string(), "value1".to_string())]),
        )
        .await
        .unwrap();
    println!("Namespace {namespace_ident:?} created!");

    let loaded_namespace = catalog.get_namespace(&namespace_ident).await.unwrap();
    println!("Namespace loaded!\n\nNamespace: {loaded_namespace:#?}",);
}

Table

After creating Catalog, we can manipulate tables through Catalog.

You can use following code to create a table:

// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.

use std::collections::HashMap;

use iceberg::spec::{NestedField, PrimitiveType, Schema, Type};
use iceberg::{Catalog, CatalogBuilder, NamespaceIdent, TableCreation, TableIdent};
use iceberg_catalog_rest::{REST_CATALOG_PROP_URI, RestCatalogBuilder};

static REST_URI: &str = "http://localhost:8181";
static NAMESPACE: &str = "default";
static TABLE_NAME: &str = "t1";

/// This is a simple example that demonstrates how to use [`RestCatalog`] to create tables.
///
/// The demo creates a table creates a table and then later retrieves the same table.
///
/// A running instance of the iceberg-rest catalog on port 8181 is required. You can find how to run
/// the iceberg-rest catalog with `docker compose` in the official
/// [quickstart documentation](https://iceberg.apache.org/spark-quickstart/).
#[tokio::main]
async fn main() {
    // Create the REST iceberg catalog.
    let catalog = RestCatalogBuilder::default()
        .load(
            "rest",
            HashMap::from([(REST_CATALOG_PROP_URI.to_string(), REST_URI.to_string())]),
        )
        .await
        .unwrap();

    // Create the table identifier.
    let namespace_ident = NamespaceIdent::from_vec(vec![NAMESPACE.to_string()]).unwrap();
    let table_ident = TableIdent::new(namespace_ident.clone(), TABLE_NAME.to_string());

    // You can also use the `from_strs` method on `TableIdent` to create the table identifier.
    // let table_ident = TableIdent::from_strs([NAMESPACE, TABLE_NAME]).unwrap();

    // Drop the table if it already exists.
    if catalog.table_exists(&table_ident).await.unwrap() {
        println!("Table {TABLE_NAME} already exists, dropping now.");
        catalog.drop_table(&table_ident).await.unwrap();
    }

    // Build the table schema.
    let table_schema = Schema::builder()
        .with_fields(vec![
            NestedField::optional(1, "foo", Type::Primitive(PrimitiveType::String)).into(),
            NestedField::required(2, "bar", Type::Primitive(PrimitiveType::Int)).into(),
            NestedField::optional(3, "baz", Type::Primitive(PrimitiveType::Boolean)).into(),
        ])
        .with_schema_id(1)
        .with_identifier_field_ids(vec![2])
        .build()
        .unwrap();

    // Build the table creation parameters.
    let table_creation = TableCreation::builder()
        .name(table_ident.name.clone())
        .schema(table_schema.clone())
        .properties(HashMap::from([("owner".to_string(), "testx".to_string())]))
        .build();

    // Create the table.
    let _created_table = catalog
        .create_table(&table_ident.namespace, table_creation)
        .await
        .unwrap();
    println!("Table {TABLE_NAME} created!");

    // Ensure that the table is under the correct namespace.
    assert!(
        catalog
            .list_tables(&namespace_ident)
            .await
            .unwrap()
            .contains(&table_ident)
    );

    // Load the table back from the catalog. It should be identical to the created table.
    let loaded_table = catalog.load_table(&table_ident).await.unwrap();
    println!("Table {TABLE_NAME} loaded!\n\nTable: {loaded_table:?}");
}

Also, you can load a table directly:

// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.

use std::collections::HashMap;

use iceberg::spec::{NestedField, PrimitiveType, Schema, Type};
use iceberg::{Catalog, CatalogBuilder, NamespaceIdent, TableCreation, TableIdent};
use iceberg_catalog_rest::{REST_CATALOG_PROP_URI, RestCatalogBuilder};

static REST_URI: &str = "http://localhost:8181";
static NAMESPACE: &str = "default";
static TABLE_NAME: &str = "t1";

/// This is a simple example that demonstrates how to use [`RestCatalog`] to create tables.
///
/// The demo creates a table creates a table and then later retrieves the same table.
///
/// A running instance of the iceberg-rest catalog on port 8181 is required. You can find how to run
/// the iceberg-rest catalog with `docker compose` in the official
/// [quickstart documentation](https://iceberg.apache.org/spark-quickstart/).
#[tokio::main]
async fn main() {
    // Create the REST iceberg catalog.
    let catalog = RestCatalogBuilder::default()
        .load(
            "rest",
            HashMap::from([(REST_CATALOG_PROP_URI.to_string(), REST_URI.to_string())]),
        )
        .await
        .unwrap();

    // Create the table identifier.
    let namespace_ident = NamespaceIdent::from_vec(vec![NAMESPACE.to_string()]).unwrap();
    let table_ident = TableIdent::new(namespace_ident.clone(), TABLE_NAME.to_string());

    // You can also use the `from_strs` method on `TableIdent` to create the table identifier.
    // let table_ident = TableIdent::from_strs([NAMESPACE, TABLE_NAME]).unwrap();

    // Drop the table if it already exists.
    if catalog.table_exists(&table_ident).await.unwrap() {
        println!("Table {TABLE_NAME} already exists, dropping now.");
        catalog.drop_table(&table_ident).await.unwrap();
    }

    // Build the table schema.
    let table_schema = Schema::builder()
        .with_fields(vec![
            NestedField::optional(1, "foo", Type::Primitive(PrimitiveType::String)).into(),
            NestedField::required(2, "bar", Type::Primitive(PrimitiveType::Int)).into(),
            NestedField::optional(3, "baz", Type::Primitive(PrimitiveType::Boolean)).into(),
        ])
        .with_schema_id(1)
        .with_identifier_field_ids(vec![2])
        .build()
        .unwrap();

    // Build the table creation parameters.
    let table_creation = TableCreation::builder()
        .name(table_ident.name.clone())
        .schema(table_schema.clone())
        .properties(HashMap::from([("owner".to_string(), "testx".to_string())]))
        .build();

    // Create the table.
    let _created_table = catalog
        .create_table(&table_ident.namespace, table_creation)
        .await
        .unwrap();
    println!("Table {TABLE_NAME} created!");

    // Ensure that the table is under the correct namespace.
    assert!(
        catalog
            .list_tables(&namespace_ident)
            .await
            .unwrap()
            .contains(&table_ident)
    );

    // Load the table back from the catalog. It should be identical to the created table.
    let loaded_table = catalog.load_table(&table_ident).await.unwrap();
    println!("Table {TABLE_NAME} loaded!\n\nTable: {loaded_table:?}");
}

Contributing

First, thank you for contributing to Iceberg Rust! The goal of this document is to provide everything you need to start contributing to iceberg-rust. The following TOC is sorted progressively, starting with the basics and expanding into more specifics.

Your First Contribution

  1. Fork the iceberg-rust repository into your own GitHub account.
  2. Create a new Git branch.
  3. Make your changes.
  4. Submit the branch as a pull request to the main iceberg-rust repo. An iceberg-rust team member should comment and/or review your pull request within a few days. Although, depending on the circumstances, it may take longer.

Workflow

Git Branches

All changes must be made in a branch and submitted as pull requests. iceberg-rust does not adopt any type of branch naming style, but please use something descriptive of your changes.

GitHub Pull Requests

Once your changes are ready you must submit your branch as a pull request.

Title

The pull request title must follow the format outlined in the conventional commits spec. Conventional commits is a standardized format for commit messages. iceberg-rust only requires this format for commits on the main branch. And because iceberg-rust squashes commits before merging branches, this means that only the pull request title must conform to this format.

The following are all good examples of pull request titles:

feat(schema): Add last_updated_ms in schema
docs: add hdfs classpath related troubleshoot
ci: Mark job as skipped if owner is not apache
fix(schema): Ignore prefix if it's empty
refactor: Polish the implementation of read parquet

Reviews & Approvals

All pull requests should be reviewed by at least one iceberg-rust committer.

Merge Style

All pull requests are squash merged. We generally discourage large pull requests that are over 300-500 lines of diff. If you would like to propose a change that is larger we suggest coming onto Iceberg's DEV mailing list or Slack #rust Channel and discuss it with us. This way we can talk through the solution and discuss if a change that large is even needed! This will produce a quicker response to the change and likely produce code that aligns better with our process.

When a pull request is under review, please avoid using force push as it makes it difficult for reviewer to track changes. If you need to keep the branch up to date with the main branch, consider using git merge instead.

CI

Currently, iceberg-rust uses GitHub Actions to run tests. The workflows are defined in .github/workflows.

Setup

For small or first-time contributions, we recommend the dev container method. Prefer to do it yourself? That's fine too!

Using a dev container environment

iceberg-rust provides a pre-configured dev container that could be used in Github Codespaces, VSCode, JetBrains, JupyterLab. Please pick up your favourite runtime environment.

The fastest way is:

Open in GitHub Codespaces

Bring your own toolbox

Install rust

iceberg-rust is primarily a Rust project. To build iceberg-rust, you will need to set up Rust development first. We highly recommend using rustup for the setup process.

For Linux or MacOS, use the following command:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

For Windows, download rustup-init.exe from here instead.

Rustup will read iceberg-rust's rust-toolchain.toml and set up everything else automatically. To ensure that everything works correctly, run cargo version under iceberg-rust's root directory:

$ cargo version
cargo 1.69.0 (6e9a83356 2023-04-12)

Install Docker or Podman

Currently, iceberg-rust uses Docker to set up environment for integration tests. See Container Runtimes for setup instructions.

Build

  • To compile the project: make build
  • To check code styles: make check
  • To run unit tests only: make unit-test
  • To run all tests: make test

Dependencies

Cargo.lock is committed, and regularly updated by dependabot to make sure the latest dependency versions are tested in CI and developers have reproducible builds.

In Cargo.toml, we specify the minimum version required to use iceberg-rust. This allows users to choose their dependency versions without always upgrading to the latest.

Code of Conduct

We expect all community members to follow our Code of Conduct.

This document explains how the release manager releases Apache Iceberg Rust in accordance with Apache requirements.

Introduction

Source Release is the key point which Apache values, and is also necessary for an ASF release.

Please remember that publishing software has legal consequences.

This guide complements the foundation-wide policies and guides:

Terminology

In this guide:

  • iceberg_version: the final Iceberg Rust version, like 0.9.1.
  • rc: the numeric release candidate voting round, like 2.
  • rc_tag: the git tag for a release candidate, like v0.9.1-rc.2.
  • rc_dist_dir: the ASF dev distribution directory, like apache-iceberg-rust-0.9.1-rc2.
  • source_archive: the source tarball, like apache-iceberg-rust-0.9.1.tar.gz.

The RC tag includes rc.<number>. The ASF dev distribution directory uses rc<number>. The source archive name uses only the final version.

Preparation

This section is the requirements for individuals who are new to the role of release manager.

Refer to Setup GPG Key to make sure the GPG key has been set up. The RC creation script requires a local GPG secret key when artifact signing or tag creation is enabled.

Install the release tooling used by the local scripts:

  • cargo-deny
  • docker
  • gpg
  • svn

The local release helpers are under dev/release/. They log every step before it runs and after it succeeds. If a step fails, the script prints the failed step and stops.

Start a tracking issue about the next release

Start a tracking issue on GitHub for the upcoming release to track all tasks that need to be completed.

Title:

Tracking issues of Iceberg Rust ${iceberg_version} Release

Content:

This issue is used to track tasks of the iceberg rust ${iceberg_version} release.

## Tasks

### Blockers

> Blockers are the tasks that must be completed before the release.

### Build Release

#### GitHub Side

- [ ] Bump version in project
- [ ] Update docs
- [ ] Generate dependencies list
- [ ] Create and push release candidate tag

#### ASF Side

- [ ] Create ASF source release artifacts
- [ ] Upload artifacts to the SVN dist repo

### Voting

- [ ] Start VOTE at iceberg community

### Official Release

- [ ] Push the release git tag
- [ ] Publish artifacts to SVN RELEASE branch
- [ ] Change Iceberg Rust Website download link
- [ ] Send the announcement

For details of each step, please refer to: https://rust.iceberg.apache.org/release

GitHub Side

Bump version in project

Bump all components' version in the project to the new Iceberg Rust version. This version is the final version, not the release candidate version.

  • Rust core and Python binding: bump version in root Cargo.toml under [workspace.package].

Update docs

Update CHANGELOG.md by drafting a new release note on GitHub Releases.

Generate dependencies list

Download and set up cargo-deny. You can refer to cargo-deny. For example:

cargo install cargo-deny

Run the following command to update the dependencies list of every package:

dev/release/dependencies.sh generate

Run the following command to verify the updated dependencies' license:

dev/release/dependencies.sh check

Create release candidate tag and artifacts

After the version bump PR gets merged, check out the exact commit to release and run:

dev/release/create_rc.sh ${iceberg_version} ${rc}

For example:

dev/release/create_rc.sh 0.9.1 2

Useful options include:

  • --release_ref HEAD: git commit-ish to archive and tag.
  • --dist_dir dist: artifact output root.
  • --create_rc_tag 1: create the signed annotated RC tag as the final release step.
  • --check_headers 1: check Apache license headers against the source archive.
  • --check_deps 1: run dependency license checks before artifact creation.
  • --sign 1: create and verify the detached GPG signature.
  • --upload_svn 0: upload RC artifacts to the ASF dev dist SVN repository.
  • --svn_dist_url https://dist.apache.org/repos/dist/dev/iceberg: SVN directory URL where the RC artifact directory will be uploaded.

This script creates:

  • Local artifact directory: dist/apache-iceberg-rust-${iceberg_version}-rc${rc}/
  • Source archive: apache-iceberg-rust-${iceberg_version}.tar.gz
  • Signature: apache-iceberg-rust-${iceberg_version}.tar.gz.asc
  • SHA-512 checksum: apache-iceberg-rust-${iceberg_version}.tar.gz.sha512
  • Signed annotated RC tag: v${iceberg_version}-rc.${rc}

The script checks license headers against the generated source archive, not the live Git worktree. If enabled, SVN upload runs after local artifact verification and before RC tag creation. The script creates the signed RC tag as the final release step, then prints a draft VOTE email for dev@iceberg.apache.org.

To upload artifacts to ASF dev dist as part of RC creation, pass:

dev/release/create_rc.sh ${iceberg_version} ${rc} --upload_svn 1

The script does not push the RC tag. Review the output, then push the tag manually:

git push origin "v${iceberg_version}-rc.${rc}"

If an RC has a problem, abandon that RC and increment the RC number.

ASF Side

If any step in the ASF release process fails and requires code changes, abandon that RC and prepare a new RC number. Our release page displays ASF releases instead of GitHub Releases.

Verify the release candidate locally

Before uploading artifacts to ASF dev dist, verify the local artifacts:

dev/release/verify_rc.sh ${iceberg_version} ${rc} --download 0

To skip expensive build steps during a quick local check:

dev/release/verify_rc.sh ${iceberg_version} ${rc} --download 0 --build 0 --python 0

Upload artifacts to the SVN dist repo

SVN is required for this step.

The SVN repository of the dev branch is: https://dist.apache.org/repos/dist/dev/iceberg/

First, check out Iceberg to a local directory:

svn co https://dist.apache.org/repos/dist/dev/iceberg/ /tmp/iceberg-dist-dev

If the artifacts were not uploaded by dev/release/create_rc.sh --upload_svn 1, upload them manually:

rc_dist_dir="apache-iceberg-rust-${iceberg_version}-rc${rc}"

mkdir "/tmp/iceberg-dist-dev/${rc_dist_dir}/"
cp "./dist/${rc_dist_dir}/"* "/tmp/iceberg-dist-dev/${rc_dist_dir}/"

cd /tmp/iceberg-dist-dev/
svn status
svn add "${rc_dist_dir}"
svn commit -m "Prepare Apache Iceberg Rust ${iceberg_version} RC${rc}"

Visit https://dist.apache.org/repos/dist/dev/iceberg/ to make sure the artifacts are uploaded correctly.

Verify the uploaded release candidate

After uploading the artifacts, verify them from ASF dev dist:

dev/release/verify_rc.sh ${iceberg_version} ${rc}

Rescue

If you accidentally publish wrong or unexpected artifacts, like wrong signature files or checksum files, cancel the current RC, increment the RC number, and initiate a new release candidate. Remember to delete the wrong artifacts from the SVN dist repo.

Voting

Send the Iceberg community VOTE email to dev@iceberg.apache.org.

Title:

[VOTE] Release Apache Iceberg Rust ${iceberg_version} RC${rc}

Content:

Hello Apache Iceberg Rust Community,

This is a call for a vote to release Apache Iceberg Rust version ${iceberg_version}.

The tag to be voted on is: v${iceberg_version}-rc.${rc}.

The release candidate:

https://dist.apache.org/repos/dist/dev/iceberg/apache-iceberg-rust-${iceberg_version}-rc${rc}/

Keys to verify the release candidate:

https://downloads.apache.org/iceberg/KEYS

Git tag for the release:

https://github.com/apache/iceberg-rust/releases/tag/v${iceberg_version}-rc.${rc}

Please download, verify, and test the release candidate.

This vote will be open for at least 72 hours and will remain open until the required number of votes is reached.

Please vote accordingly:
[ ] +1 Approve
[ ] +0 No opinion
[ ] -1 Disapprove (please provide a reason)

To learn more about Apache Iceberg, please visit:
https://rust.iceberg.apache.org/

Checklist for reference:
[ ] Download links are valid
[ ] Checksums and signatures are correct
[ ] LICENSE and NOTICE files are present
[ ] No unexpected binary files are included
[ ] All source files have ASF headers
[ ] The project builds successfully from source
[ ] pyiceberg-core builds and tests successfully

For more details, please refer to:
https://rust.iceberg.apache.org/release.html#how-to-verify-a-release

Thanks,
${name}

Example: https://lists.apache.org/thread/c211gqq2yl15jbxqk4rcnq1bdqltjm5l

After at least 3 +1 binding votes from Iceberg PMC members, claim the vote result.

Title:

[RESULT][VOTE] Release Apache Iceberg Rust ${iceberg_version} RC${rc}

Content:

Hello Apache Iceberg Rust Community,

The vote to release Apache Iceberg Rust ${iceberg_version} RC${rc} has passed.

The vote PASSED with 3 +1 binding and 1 +1 non-binding votes, no +0 or -1 votes:

Binding votes:

- xxx
- yyy
- zzz

Non-Binding votes:

- aaa

Vote thread: ${vote_thread_url}

Thanks,
${name}

Example: https://lists.apache.org/thread/xk5myl10mztcfotn59oo59s4ckvojds6

How to verify a release

Validate with the helper script

Run:

dev/release/verify_rc.sh ${iceberg_version} ${rc}

The helper downloads the source archive, signature, and checksum from ASF dev dist, verifies the signature with the local GPG keyring, verifies the checksum, extracts the archive, checks source headers, and runs Rust and Python build/tests.

To import Apache Iceberg release keys before signature verification, run:

dev/release/verify_rc.sh ${iceberg_version} ${rc} --import_gpg_keys 1

Validate manually

A release candidate contains links to following things:

  • A source tarball
  • A signature (.asc)
  • A checksum (.sha512)

After downloading them, here are the instructions on how to verify them.

  • Import keys:

    curl https://downloads.apache.org/iceberg/KEYS -o KEYS
gpg --import KEYS

  • Verify the .asc file:

    gpg --verify apache-iceberg-rust-*.tar.gz.asc

    Expects: gpg: Good signature from ...

  • Verify the checksum:

    shasum -a 512 -c apache-iceberg-rust-*.tar.gz.sha512

    Expects: "apache-iceberg-rust-...tar.gz: OK"

  • Verify build and test:

    tar -xzf apache-iceberg-rust-*.tar.gz
cd apache-iceberg-rust-*/
make build && make test

  • Verify pyiceberg-core build and tests:

    (
  cd bindings/python
  make install
  make test
)

  • Verify license headers:

    docker run --rm -v $(pwd):/github/workspace apache/skywalking-eyes header check

    Expects: INFO Totally checked _ files, valid: _, invalid: 0, ignored: _, fixed: 0

Official Release

Promote the RC

After the VOTE passes, create the final release tag and move the ASF artifacts from dev dist to release dist:

dev/release/release.sh ${iceberg_version} ${rc}

Useful options include:

  • --create_release_tag 1: create the signed annotated final release git tag.
  • --move_svn 1: move the RC artifacts from ASF dev dist to ASF release dist.
  • --tag_ref <rc tag commit>: git commit-ish to tag as the final release.
  • --dev_dist_url https://dist.apache.org/repos/dist/dev/iceberg: SVN directory URL containing RC artifact directories.
  • --release_dist_url https://dist.apache.org/repos/dist/release/iceberg: SVN directory URL where final release artifact directories are published.

The release script does not push the final release tag. Review the output, then push the tag manually:

git push origin "v${iceberg_version}"

Pushing the final release tag triggers the publish workflow for crates and pyiceberg-core.

Create a GitHub Release

  • Click here to create a new release.
  • Pick the git tag of this release version from the dropdown menu.
  • Make sure the branch target is main.
  • Generate the release note by clicking the Generate release notes button.
  • Add the release note from every component's upgrade.md if there are breaking changes before the content generated by GitHub. Check them carefully.
  • Publish the release.

Send the announcement

Send the release announcement to dev@iceberg.apache.org and CC announce@apache.org.

Instead of adding breaking changes, include the new features as "notable changes" in the announcement.

Title:

[ANNOUNCE] Release Apache Iceberg Rust ${iceberg_version}

Content:

Hi all,

The Apache Iceberg Rust community is pleased to announce
that Apache Iceberg Rust ${iceberg_version} has been released!

Iceberg is a data access layer that allows users to easily and efficiently
retrieve data from various storage services in a unified way.

The notable changes since the previous release include:
1. xxxxx
2. yyyyyy
3. zzzzzz

Please refer to the change log for the complete list of changes:
https://github.com/apache/iceberg-rust/releases/tag/v${iceberg_version}

Apache Iceberg Rust website: https://rust.iceberg.apache.org/

Download Links: https://rust.iceberg.apache.org/download

From official ASF distribution: https://dist.apache.org/repos/dist/release/iceberg/apache-iceberg-rust-${iceberg_version}/

Iceberg Resources:
- Issue: https://github.com/apache/iceberg-rust/issues
- Mailing list: dev@iceberg.apache.org

Thanks
On behalf of Apache Iceberg Community

Example: https://lists.apache.org/thread/oy77n55brvk72tnlb2bjzfs9nz3cfd0s

Container Runtimes

Iceberg-rust uses containers for integration tests, where docker and docker compose start containers for MinIO and various catalogs. You can use any of the following container runtimes.

Docker Desktop

Docker Desktop is available for macOS, Windows, and Linux.

  1. Install Docker Desktop by downloading the installer or using Homebrew on macOS.

    brew install --cask docker

  2. Launch Docker Desktop and complete the setup.

  3. Verify the installation.

    docker --version
docker compose version

  4. Try some integration tests!

    make test

OrbStack (macOS)

OrbStack is a lightweight alternative to Docker Desktop on macOS.

  1. Install OrbStack by downloading the installer or using Homebrew.

    brew install orbstack

  2. Migrate Docker data (if switching from Docker Desktop).

    orb migrate docker

  3. (Optional) Add registry mirrors.

    You can edit the config directly at ~/.orbstack/config/docker.json and restart the engine with orb restart docker.

    {
    "registry-mirrors": ["<mirror_addr>"]
}

  4. Try some integration tests!

    make test

Podman

Podman is a daemonless container engine. The instructions below set up "rootful podman" with docker's official docker-compose plugin.

  1. Have podman v4 or newer.

    $ podman --version
podman version 4.9.4-rhel

  2. Create a docker wrapper script:

    Create a fresh /usr/bin/docker file and add the below contents:

    #!/bin/sh
[ -e /etc/containers/nodocker ] || \
echo "Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg." >&2
exec sudo /usr/bin/podman "$@"

    Set new /usr/bin/docker file to executable.

    sudo chmod +x /usr/bin/docker

  3. Install the docker compose plugin. Check for successful installation.

    $ docker compose version
Docker Compose version v2.28.1

  4. Append the below to ~/.bashrc or equivalent shell config:

    export DOCKER_HOST=unix:///run/podman/podman.sock

  5. Start the "rootful" podman socket.

    sudo systemctl start podman.socket
sudo systemctl status podman.socket

  6. Check that the following symlink exists.

    $ ls -al /var/run/docker.sock
lrwxrwxrwx 1 root root 27 Jul 24 12:18 /var/run/docker.sock -> /var/run/podman/podman.sock

    If the symlink does not exist, create it.

    sudo ln -s /var/run/podman/podman.sock /var/run/docker.sock

  7. Check that the docker socket is working.

    sudo curl -H "Content-Type: application/json" --unix-socket /var/run/docker.sock http://localhost/_ping

  8. Try some integration tests!

    cargo test -p iceberg --test file_io_s3_test

Note on rootless containers

As of podman v4, "To be succinct and simple, when running rootless containers, the container itself does not have an IP address". This causes issues with iceberg-rust's integration tests, which rely upon IP-addressable containers via docker-compose. As a result, podman "rootful" containers are required to ensure containers have IP addresses.

Podman troubleshooting

Error: short-name "apache/iceberg-rest-fixture" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"

Fix: Add or modify the /etc/containers/registries.conf file:

[[registry]]
prefix = "docker.io"
location = "docker.io"

Podman references

Setup GPG key

This section is a brief from the Cryptography with OpenPGP guideline.

Install GPG

For more details, please refer to GPG official website. Here shows one approach to install GPG with apt:

sudo apt install gnupg2

Generate GPG Key

Attentions:

  • Name is best to keep consistent with your full name of Apache ID;
  • Email should be the Apache email;
  • Name is best to only use English to avoid garbled.

Run gpg --full-gen-key and complete the generation interactively:

gpg (GnuPG) 2.2.20; Copyright (C) 2020 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
  (14) Existing key from card
Your selection? 1 # input 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096 # input 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 0 # input 0
Key does not expire at all
Is this correct? (y/N) y # input y

GnuPG needs to construct a user ID to identify your key.

Real name: Hulk Lin               # input your name
Email address: hulk@apache.org    # input your email
Comment:                          # input some annotations, optional
You selected this USER-ID:
    "Hulk <hulk@apache.org>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O # input O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

# Input the security key
┌──────────────────────────────────────────────────────┐
│ Please enter this passphrase                         │
│                                                      │
│ Passphrase: _______________________________          │
│                                                      │
│       <OK>                              <Cancel>     │
└──────────────────────────────────────────────────────┘
# key generation will be done after your inputting the key with the following output
gpg: key E49B00F626B marked as ultimately trusted
gpg: revocation certificate stored as '/Users/hulk/.gnupg/openpgp-revocs.d/F77B887A4F25A9468C513E9AA3008E49B00F626B.rev'
public and secret key created and signed.

pub   rsa4096 2022-07-12 [SC]
      F77B887A4F25A9468C513E9AA3008E49B00F626B
uid           [ultimate] hulk <hulk@apache.org>
sub   rsa4096 2022-07-12 [E]

Upload your key to public GPG keyserver

Firstly, list your key:

gpg --list-keys

The output is like:

-------------------------------
pub   rsa4096 2022-07-12 [SC]
      F77B887A4F25A9468C513E9AA3008E49B00F626B
uid           [ultimate] hulk <hulk@apache.org>
sub   rsa4096 2022-07-12 [E]

Then, send your key id to key server:

gpg --keyserver keys.openpgp.org --send-key <key-id> # e.g., F77B887A4F25A9468C513E9AA3008E49B00F626B

Among them, keys.openpgp.org is a randomly selected keyserver, you can use keyserver.ubuntu.com or any other full-featured keyserver.

Check whether the key is created successfully

Uploading takes about one minute; after that, you can check by your email at the corresponding keyserver.

Uploading keys to the keyserver is mainly for joining a Web of Trust.

Add your GPG public key to the KEYS document

:::info

SVN is required for this step.

:::

The svn repository of the release branch is: https://dist.apache.org/repos/dist/release/iceberg

Please always add the public key to KEYS in the release branch:

svn co https://dist.apache.org/repos/dist/release/iceberg iceberg-dist
# As this step will copy all the versions, it will take some time. If the network is broken, please use svn cleanup to delete the lock before re-execute it.
cd iceberg-dist
(gpg --list-sigs YOUR_NAME@apache.org && gpg --export --armor YOUR_NAME@apache.org) >> KEYS # Append your key to the KEYS file
svn add .   # It is not needed if the KEYS document exists before.
svn ci -m "add gpg key for YOUR_NAME" # Later on, if you are asked to enter a username and password, just use your apache username and password.

Upload the GPG public key to your GitHub account

  • Enter https://github.com/settings/keys to add your GPG key.
  • Please remember to bind the email address used in the GPG key to your GitHub account (https://github.com/settings/emails) if you find "unverified" after adding it.