pub fn read_manifest_entries<B: AsRef<[u8]>>(
bytes: B,
manifest_file: &ManifestFile,
) -> Result<ManifestEntryBatchReader<B>>Expand description
Reads the entries of a manifest file as batches with one row per entry.
bytes is the manifest file and manifest_file is its entry in the
manifest list. The schema is the spec’s manifest_entry struct for the
latest format version, with field IDs in the Arrow field metadata, so
manifests of every format version read into the same schema apart from
data_file.partition, which follows the manifest’s partition spec.
Fields are matched to the manifest’s Avro schema by field ID. A v1
manifest reads with the spec’s v2+ defaults, and null snapshot IDs,
sequence numbers, and first row IDs are inherited from manifest_file.
Lower and upper bounds keep the binary single-value serialization they were
written with.
If manifest_file.key_metadata is set, the manifest is encrypted and
bytes must be decrypted first.
Table::manifest_reader returns a
reader that decrypts.
§Errors
Returns an error if the manifest’s header can’t be read or names an unsupported format version. Encrypted bytes fail at the header. Errors in the entries come from the reader, which yields no batches after one.
§Example
use iceberg::arrow::read_manifest_entries;
use iceberg::spec::{ManifestContentType, ManifestFile};
let bytes = std::fs::read(concat!(
env!("CARGO_MANIFEST_DIR"),
"/testdata/manifests/pyiceberg-v2-data.avro"
))?;
let manifest_file = ManifestFile {
manifest_path: "s3://bucket/metadata/manifest.avro".to_string(),
manifest_length: bytes.len() as i64,
partition_spec_id: 0,
content: ManifestContentType::Data,
sequence_number: 1,
min_sequence_number: 1,
added_snapshot_id: 1,
added_files_count: None,
existing_files_count: None,
deleted_files_count: None,
added_rows_count: None,
existing_rows_count: None,
deleted_rows_count: None,
partitions: None,
key_metadata: None,
first_row_id: None,
};
let mut rows = 0;
for batch in read_manifest_entries(&bytes, &manifest_file)? {
rows += batch?.num_rows();
}
assert_eq!(rows, 2);