Skip to main content

strip_metadata_from_schema

Function strip_metadata_from_schema 

Source
pub fn strip_metadata_from_schema(schema: &Schema) -> Result<Schema>
Expand description

Strips all metadata from an Arrow schema and its nested fields.

This function recursively removes metadata from all fields at every level of the schema, including nested struct, list, and map fields. This is useful for schema comparison where metadata differences should be ignored.

Some other differences are normalized away too, so that schemas that differ only in them compare equal once stripped. The fields inside lists and maps are renamed as schema_to_arrow_schema names them, and dictionaries are unwrapped:

  • A list’s element field is renamed to LIST_FIELD_NAME, so item (the arrow-rs default) becomes element.
  • A map’s entries field is renamed to DEFAULT_MAP_FIELD_NAME and made non-nullable.
  • A map’s key and value fields are renamed to MAP_KEY_FIELD_NAME and MAP_VALUE_FIELD_NAME, so keys and values (the arrow-rs MapBuilder default) become key and value, and the key is made non-nullable.
  • A dictionary-encoded field becomes a field of the dictionary’s value type.

The names of the schema’s fields and of struct fields are kept. Extension type metadata (ARROW:extension:*) is dropped with the rest, so a variant field becomes a plain struct of its metadata and value fields.

§Arguments

  • schema - The Arrow schema to strip metadata from

§Returns

A new Arrow schema with all metadata removed, or an error if the schema structure is invalid.

§Example

use std::collections::HashMap;

use arrow_schema::{DataType, Field, Schema as ArrowSchema};
use iceberg::arrow::strip_metadata_from_schema;

let mut metadata = HashMap::new();
metadata.insert("key".to_string(), "value".to_string());

let field = Field::new("col1", DataType::Int32, false).with_metadata(metadata);
let schema = ArrowSchema::new(vec![field]);

let stripped = strip_metadata_from_schema(&schema).unwrap();
assert!(stripped.field(0).metadata().is_empty());