Skip to main content

Crate iceberg_storage_opendal

Crate iceberg_storage_opendal 

Source
Expand description

OpenDAL-based storage implementation for Apache Iceberg.

This crate provides OpenDalStorage and OpenDalStorageFactory, which implement the [Storage] and [StorageFactory] traits from the iceberg crate using OpenDAL as the backend.

Structs§

AwsCredential
AWS credentials: access key ID, secret access key, and optional session token. Credential that holds the access_key and secret_key.
CustomAwsCredentialLoader
Custom AWS credential loader.
HdfsNativeStorage
State of OpenDalStorage::HdfsNative: the parsed configuration and the per-NameNode operator cache. Only the storage factories build it.
OpenDalClientConfig
Backend-independent client settings shared by every OpenDalStorage variant.
OpenDalResolvingStorage
A resolving storage that auto-detects the scheme from a path and delegates to the appropriate OpenDalStorage variant.
OpenDalResolvingStorageFactory
A resolving storage factory that creates OpenDalResolvingStorage instances.

Enums§

OpenDalStorage
OpenDAL-based storage implementation.
OpenDalStorageFactory
OpenDAL-based storage factory.

Constants§

OPENDAL_IO_TIMEOUT_MS
Timeout in milliseconds for each IO call on a reader, writer, lister or deleter, applied per retry attempt via OpenDAL’s TimeoutLayer::with_io_timeout. Defaults to 10 seconds.
OPENDAL_RETRY_MAX_WAIT_MS
Maximum wait in milliseconds between two retries on any backend. Unset keeps OpenDAL’s default. For S3, S3_RETRY_MAX_WAIT_MS takes precedence.
OPENDAL_RETRY_MIN_WAIT_MS
Minimum wait in milliseconds before retrying a failed operation on any backend. Unset keeps OpenDAL’s default. For S3, S3_RETRY_MIN_WAIT_MS takes precedence.
OPENDAL_RETRY_NUM_RETRIES
Number of times to retry a failed operation on any backend, 0 to disable retrying. Unset keeps OpenDAL’s default. For S3, S3_RETRY_NUM_RETRIES takes precedence.

Traits§

ProvideCredential
Trait for types that can asynchronously supply AwsCredential to a CustomAwsCredentialLoader. ProvideCredential is the trait used by signer to load the credential from the environment. ` Service may require different credential to sign the request, for example, AWS require access key and secret key, while Google Cloud Storage require token.