Module iceberg::spec

source ·
Expand description

Spec for Iceberg.

Structs§

  • DataFile
    Data file carries data file path, partition tuple, metrics, …
  • DataFileBuilder
    Builder for DataFile.
  • Datum
    Literal associated with its type. The value and type pair is checked when construction, so the type and value is guaranteed to be correct when used.
  • FieldSummary
    Field summary for partition field in the spec.
  • ListType
    A list is a collection of values with some element type. The element field has an integer id that is unique in the table schema. Elements can be either optional or required. Element types may be any type.
  • Manifest
    A manifest contains metadata and a list of entries.
  • ManifestEntry
    A manifest is an immutable Avro file that lists data files or delete files, along with each file’s partition data tuple, metrics, and tracking information.
  • ManifestFile
    Entry in a manifest list.
  • ManifestList
    Snapshots are embedded in table metadata, but the list of manifests for a snapshot are stored in a separate manifest list file.
  • ManifestListWriter
    A manifest list writer.
  • ManifestMetadata
    Meta data of a manifest that is stored in the key-value metadata of the Avro file
  • ManifestWriter
    A manifest writer.
  • Map
    Map is a collection of key-value pairs with a key type and a value type. It used in Literal::Map, to make it hashable, the order of key-value pairs is stored in a separate vector so that we can hash the map in a deterministic way. But it also means that the order of key-value pairs is matter for the hash value.
  • MapType
    A map is a collection of key-value pairs with a key type and a value type. Both the key field and value field each have an integer id that is unique in the table schema. Map keys are required and map values can be either optional or required. Both map keys and map values may be any type, including nested types.
  • MetadataLog
    Encodes changes to the previous metadata files for the table
  • NestedField
    A struct is a tuple of typed values. Each field in the tuple is named and has an integer id that is unique in the table schema. Each field can be either optional or required, meaning that values can (or cannot) be null. Fields may be any type. Fields may have an optional comment or doc string. Fields can have default values.
  • PartitionField
    Partition fields capture the transform from table data to partition values.
  • PartitionSpec
    Partition spec that defines how to produce a tuple of partition values from a record.
  • PartitionSpecBuilder
    Create valid partition specs for a given schema.
  • RawLiteral
    Raw literal representation used for serde. The serialize way is used for Avro serializer.
  • Schema
    Defines schema in iceberg.
  • SchemaBuilder
    Schema builder.
  • Snapshot
    A snapshot represents the state of a table at some time and is used to access the complete set of data files in the table.
  • SnapshotLog
    A log of when each snapshot was made.
  • SnapshotReference
    Iceberg tables keep track of branches and tags using snapshot references.
  • SortField
    Entry for every column that is to be sorted
  • SortOrder
    A sort order is defined by a sort order id and a list of sort fields. The order of the sort fields within the list defines the order in which the sort is applied to the data.
  • SortOrderBuilder
    Builder for SortOrder.
  • SqlViewRepresentation
    The SQL representation stores the view definition as a SQL SELECT, with metadata such as the SQL dialect.
  • Struct
    The partition struct stores the tuple of partition values for each file. Its type is derived from the partition fields of the partition spec used to write the manifest file. In v2, the partition struct’s field ids must match the ids from the partition spec.
  • StructType
    DataType for a specific struct
  • StructValueIntoIter
    An iterator that moves out of a struct.
  • Summary
    Summarises the changes in the snapshot.
  • TableMetadata
    Fields for the version 2 of the table metadata.
  • TableMetadataBuilder
    Manipulating table metadata.
  • UnboundPartitionField
    Unbound partition field can be built without a schema and later bound to a schema.
  • UnboundPartitionSpec
    Unbound partition spec can be built without a schema and later bound to a schema.
  • UnboundPartitionSpecBuilder
    Create a new UnboundPartitionSpec
  • ViewMetadata
    Fields for the version 1 of the view metadata.
  • ViewMetadataBuilder
    Manipulating view metadata.
  • ViewRepresentations
    A list of view representations.
  • ViewVersion
    A view versions represents the definition of a view at a specific point in time.
  • ViewVersionLog
    A log of when each snapshot was made.

Enums§

  • DataContentType
    Type of content stored by the data file: data, equality deletes, or position deletes (all v1 files are data files)
  • DataFileBuilderError
    Error type for DataFileBuilder
  • DataFileFormat
    Format of this data.
  • FormatVersion
    Iceberg format version
  • Literal
    Values present in iceberg type
  • ManifestContentType
    The type of files tracked by the manifest, either data or delete files; Data(0) for all v1 manifests
  • ManifestStatus
    Used to track additions and deletions in ManifestEntry.
  • NullOrder
    Describes the order of null values when sorted.
  • Operation
    The operation field is used by some operations, like snapshot expiration, to skip processing certain snapshots.
  • PrimitiveLiteral
    Values present in iceberg type
  • PrimitiveType
    Primitive data types
  • SnapshotRetention
    The snapshot expiration procedure removes snapshots from table metadata and applies the table’s retention policy.
  • SortDirection
    Sort direction in a partition, either ascending or descending
  • SortOrderBuilderError
    Error type for SortOrderBuilder
  • Transform
    Transform is used to transform predicates to partition predicates, in addition to transforming data values.
  • Type
    All data types are either primitives or nested types, which are maps, lists, or structs.
  • ViewFormatVersion
    Iceberg format version
  • ViewRepresentation
    View definitions can be represented in multiple ways. Representations are documented ways to express a view definition.

Constants§

Traits§

Functions§

Type Aliases§