Enum iceberg::spec::Transform

pub enum Transform {
    Identity,
    Bucket(u32),
    Truncate(u32),
    Year,
    Month,
    Day,
    Hour,
    Void,
    Unknown,
}

Transform is used to transform predicates to partition predicates, in addition to transforming data values.

Deriving partition predicates from column predicates on the table data is used to separate the logical queries from physical storage: the partitioning can change and the correct partition filters are always derived from column predicates.

This simplifies queries because users don’t have to supply both logical predicates and partition predicates.

All transforms must return null for a null input value.

Variants

Identity

Source value, unmodified

• Source type could be any type.
• Return type is the same with source type.

Bucket(u32)

Hash of value, mod N.

Bucket partition transforms use a 32-bit hash of the source value. The 32-bit hash implementation is the 32-bit Murmur3 hash, x86 variant, seeded with 0.

Transforms are parameterized by a number of buckets, N. The hash mod N must produce a positive value by first discarding the sign bit of the hash value. In pseudo-code, the function is:

def bucket_N(x) = (murmur3_x86_32_hash(x) & Integer.MAX_VALUE) % N

• Source type could be int, long, decimal, date, time, timestamp, timestamptz, string, uuid, fixed, binary.
• Return type is int.

Truncate(u32)

Value truncated to width W

For int:

• v - (v % W) remainders must be positive
• example: W=10: 1 ￫ 0, -1 ￫ -10
• note: The remainder, v % W, must be positive.

For long:

• v - (v % W) remainders must be positive
• example: W=10: 1 ￫ 0, -1 ￫ -10
• note: The remainder, v % W, must be positive.

For decimal:

• scaled_W = decimal(W, scale(v)) v - (v % scaled_W)
• example: W=50, s=2: 10.65 ￫ 10.50

For string:

• Substring of length L: v.substring(0, L)

• example: L=3: iceberg ￫ ice

• note: Strings are truncated to a valid UTF-8 string with no more than L code points.

• Source type could be int, long, decimal, string

• Return type is the same with source type.

Year

Extract a date or timestamp year, as years from 1970

• Source type could be date, timestamp, timestamptz
• Return type is int

Month

Extract a date or timestamp month, as months from 1970-01-01

• Source type could be date, timestamp, timestamptz
• Return type is int

Day

Extract a date or timestamp day, as days from 1970-01-01

• Source type could be date, timestamp, timestamptz
• Return type is int

Hour

Extract a timestamp hour, as hours from 1970-01-01 00:00:00

• Source type could be timestamp, timestamptz
• Return type is int

Void

Always produces null

The void transform may be used to replace the transform in an existing partition field so that the field is effectively dropped in v1 tables.

• Source type could be any type..
• Return type is Source type.

Unknown

Used to represent some customized transform that can’t be recognized or supported now.

Implementations

impl Transform

pub fn result_type(&self, input_type: &Type) -> Result<Type>

Get the return type of transform given the input type. Returns None if it can’t be transformed.

pub fn preserves_order(&self) -> bool

Whether the transform preserves the order of values.

pub fn dedup_name(&self) -> String

Return the unique transform name to check if similar transforms for the same source field are added multiple times in partition spec builder.

pub fn satisfies_order_of(&self, other: &Self) -> bool

Whether ordering by this transform’s result satisfies the ordering of another transform’s result.

For example, sorting by day(ts) will produce an ordering that is also by month(ts) or year(ts). However, sorting by day(ts) will not satisfy the order of hour(ts) or identity(ts).

pub fn project( &self, name: &str, predicate: &BoundPredicate, ) -> Result<Option<Predicate>>

Projects a given predicate according to the transformation specified by the Transform instance.

This allows predicates to be effectively applied to data that has undergone transformation, enabling efficient querying and filtering based on the original, untransformed data.

Example

Suppose, we have row filter a = 10, and a partition spec bucket(a, 37) as bs, if one row matches a = 10, then its partition value should match bucket(10, 37) as bs, and we project a = 10 to bs = bucket(10, 37)

Trait Implementations

impl Clone for Transform

fn clone(&self) -> Transform

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Transform

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for Transform

fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>

where D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Display for Transform

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl FromStr for Transform

type Err = Error

The associated error which can be returned from parsing.

fn from_str(s: &str) -> Result<Self>

Parses a string s to return a value of this type.

impl PartialEq for Transform

fn eq(&self, other: &Transform) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for Transform

fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>

where S: Serializer,

Serialize this value into the given Serde serializer.

impl Copy for Transform

impl Eq for Transform

impl StructuralPartialEq for Transform