Struct FileIO 

pub struct FileIO { /* private fields */ }

FileIO implementation, used to manipulate files in underlying storage.

FileIO wraps a dyn Storage with lazy initialization via StorageFactory. The storage is created on first use and cached for subsequent operations.

Note

All paths passed to FileIO must be absolute paths starting with the scheme string appropriate for the storage backend being used.

This crate provides native support for local filesystem (file://) and memory (memory://) storage. For extensive storage backend support (S3, GCS, OSS, Azure, etc.), use the iceberg-storage-opendal crate.

Example

use iceberg::io::{FileIO, FileIOBuilder};
use iceberg::io::{LocalFsStorageFactory, MemoryStorageFactory};
use std::sync::Arc;

// Create FileIO with memory storage for testing
let file_io = FileIO::new_with_memory();

// Create FileIO with local filesystem storage
let file_io = FileIO::new_with_fs();

// Create FileIO with custom factory
let file_io = FileIOBuilder::new(Arc::new(LocalFsStorageFactory))
    .with_prop("key", "value")
    .build();

Implementations

impl FileIO

pub fn new_with_memory() -> Self

Create a new FileIO backed by in-memory storage.

This is useful for testing scenarios where persistent storage is not needed.

pub fn new_with_fs() -> Self

Create a new FileIO backed by local filesystem storage.

This is useful for local development and testing with real files.

pub fn serialize_all(&self) -> Result<Vec<u8>>

Serializes all portable state of this FileIO into a byte vector.

This includes the storage configuration and factory, but not the cached storage instance. The storage cache is rebuilt lazily on first use after calling FileIO::deserialize_all.

The serialized representation is not a stable format and may change between crate versions. Applications should not rely on it for long-term storage or exchange it between incompatible versions of this crate.

All storage configuration properties are included in the serialized representation. These properties may contain credentials or other sensitive values, so the returned bytes must be protected in transit and at rest by the application embedding this crate.

Storage factories are serialized through typetag. Third-party factories must use #[typetag::serde] on their StorageFactory implementation.

pub fn deserialize_all(bytes: &[u8]) -> Result<Self>

Deserializes a FileIO previously produced by FileIO::serialize_all.

The receiving binary must use a compatible crate version and link the concrete factory implementation so it is registered with typetag. Backend-specific requirements are documented by each storage factory implementation.

pub fn config(&self) -> &StorageConfig

Get the storage configuration.

pub async fn delete(&self, path: impl AsRef<str>) -> Result<()>

Deletes file.

Arguments

• path: It should be absolute path starting with scheme string used to construct FileIO.

pub async fn delete_prefix(&self, path: impl AsRef<str>) -> Result<()>

Remove the path and all nested dirs and files recursively.

Arguments

• path: It should be absolute path starting with scheme string used to construct FileIO.

Behavior

• If the path is a file or not exist, this function will be no-op.
• If the path is a empty directory, this function will remove the directory itself.
• If the path is a non-empty directory, this function will remove the directory and all nested files and directories.

pub async fn delete_stream( &self, paths: impl Stream<Item = String> + Send + 'static, ) -> Result<()>

Delete multiple files from a stream of paths.

Arguments

• paths: A stream of absolute paths starting with the scheme string used to construct FileIO.

pub async fn exists(&self, path: impl AsRef<str>) -> Result<bool>

Check file exists.

Arguments

• path: It should be absolute path starting with scheme string used to construct FileIO.

pub fn new_input(&self, path: impl AsRef<str>) -> Result<InputFile>

Creates input file.

Arguments

• path: It should be absolute path starting with scheme string used to construct FileIO.

pub fn new_output(&self, path: impl AsRef<str>) -> Result<OutputFile>

Creates output file.

Arguments

• path: It should be absolute path starting with scheme string used to construct FileIO.

Trait Implementations

impl Clone for FileIO

fn clone(&self) -> FileIO

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for FileIO

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.