Module arrow

Expand description

Conversion between Iceberg and Arrow schema

Re-exports§

delete_file_loader: Delete File loader
partition_value_calculator: Partition value calculator for computing partition values Partition value calculation for Iceberg tables.
record_batch_partition_splitter: Record batch partition splitter for partitioned tables
record_batch_projector: RecordBatch projection utilities

ArrowArrayAccessor: Partner type representing accessing and walking arrow arrays alongside iceberg schema
ArrowFileReader: ArrowFileReader is a wrapper around a FileRead that impls parquets AsyncFileReader.
ArrowReader: Reads data from Parquet files
ArrowReaderBuilder: Builder to create ArrowReader
ScanMetrics: Metrics collected during an Iceberg scan.
ScanResult: Result of ArrowReader::read, containing the record batch stream and metrics collected during the scan.

FieldMatchMode: Defines how Arrow fields are matched with Iceberg fields when converting data.

DEFAULT_MAP_FIELD_NAME: When iceberg map type convert to Arrow map type, the default map field name is “key_value”.
UTC_TIME_ZONE: UTC time zone for Arrow timestamp type.

arrow_primitive_to_literal: Convert arrow primitive array to iceberg primitive value array. This function will assume the schema of arrow struct array is the same as iceberg struct type.
arrow_schema_to_schema: Convert Arrow schema to Iceberg schema.
arrow_schema_to_schema_auto_assign_ids: Convert Arrow schema to Iceberg schema with automatically assigned field IDs.
arrow_struct_to_literal: Convert arrow struct array to iceberg struct value array. This function will assume the schema of arrow struct array is the same as iceberg struct type.
arrow_type_to_type: Convert Arrow type to iceberg type.
datum_to_arrow_type_with_ree: Converts a Datum (Iceberg type + primitive literal) to its corresponding Arrow DataType with Run-End Encoding (REE).
schema_to_arrow_schema: Convert iceberg schema to an arrow schema.
strip_metadata_from_schema: Strips all metadata from an Arrow schema and its nested fields.
type_to_arrow_type: Convert iceberg type to an arrow type.