readParquet

Read Parquet data from existing urls by using Arrow Dataset

Nested Arrow Struct columns are read as ColumnGroups: an optional (nullable) struct becomes a column group whose child columns are nullable and hold null where the struct is absent (a column group is never null per row, so an absent struct and a present all-null struct read the same). The same applies to readArrowIPC and readArrowFeather; see issue #536.

Timestamp columns flagged isAdjustedToUTC = true count time units since 1970-01-01T00:00:00Z, so they identify a single point on the time-line and are read as kotlin.time.Instant in every supported precision (MILLIS, MICROS, NANOS). Zone-less timestamps identify no such point and stay kotlinx.datetime.LocalDateTime.

A zone name may still be stored alongside such a column — Parquet's logical type itself keeps only isAdjustedToUTC, but PyArrow puts the original zone into the file's ARROW:schema metadata, and in Arrow IPC/Feather it is part of the schema as Timestamp(unit, tz). It is display metadata: the values are already normalized to UTC, so applying it would change nothing about which instant they name, and it is deliberately neither applied nor returned. Only the instant comes back; use convert { … }.with { it.toLocalDateTime(zone) } for wall-clock values in a zone you pick.

Read Parquet data from existing strUrls by using Arrow Dataset

Nested Arrow Struct columns are read as ColumnGroups: an optional (nullable) struct becomes a column group whose child columns are nullable and hold null where the struct is absent (a column group is never null per row, so an absent struct and a present all-null struct read the same). The same applies to readArrowIPC and readArrowFeather; see issue #536.

Timestamp columns flagged isAdjustedToUTC = true count time units since 1970-01-01T00:00:00Z, so they identify a single point on the time-line and are read as kotlin.time.Instant in every supported precision (MILLIS, MICROS, NANOS). Zone-less timestamps identify no such point and stay kotlinx.datetime.LocalDateTime.

A zone name may still be stored alongside such a column — Parquet's logical type itself keeps only isAdjustedToUTC, but PyArrow puts the original zone into the file's ARROW:schema metadata, and in Arrow IPC/Feather it is part of the schema as Timestamp(unit, tz). It is display metadata: the values are already normalized to UTC, so applying it would change nothing about which instant they name, and it is deliberately neither applied nor returned. Only the instant comes back; use convert { … }.with { it.toLocalDateTime(zone) } for wall-clock values in a zone you pick.

Read Parquet data from existing paths by using Arrow Dataset

Nested Arrow Struct columns are read as ColumnGroups: an optional (nullable) struct becomes a column group whose child columns are nullable and hold null where the struct is absent (a column group is never null per row, so an absent struct and a present all-null struct read the same). The same applies to readArrowIPC and readArrowFeather; see issue #536.

Timestamp columns flagged isAdjustedToUTC = true count time units since 1970-01-01T00:00:00Z, so they identify a single point on the time-line and are read as kotlin.time.Instant in every supported precision (MILLIS, MICROS, NANOS). Zone-less timestamps identify no such point and stay kotlinx.datetime.LocalDateTime.

A zone name may still be stored alongside such a column — Parquet's logical type itself keeps only isAdjustedToUTC, but PyArrow puts the original zone into the file's ARROW:schema metadata, and in Arrow IPC/Feather it is part of the schema as Timestamp(unit, tz). It is display metadata: the values are already normalized to UTC, so applying it would change nothing about which instant they name, and it is deliberately neither applied nor returned. Only the instant comes back; use convert { … }.with { it.toLocalDateTime(zone) } for wall-clock values in a zone you pick.

Read Parquet data from existing files by using Arrow Dataset

Nested Arrow Struct columns are read as ColumnGroups: an optional (nullable) struct becomes a column group whose child columns are nullable and hold null where the struct is absent (a column group is never null per row, so an absent struct and a present all-null struct read the same). The same applies to readArrowIPC and readArrowFeather; see issue #536.

Timestamp columns flagged isAdjustedToUTC = true count time units since 1970-01-01T00:00:00Z, so they identify a single point on the time-line and are read as kotlin.time.Instant in every supported precision (MILLIS, MICROS, NANOS). Zone-less timestamps identify no such point and stay kotlinx.datetime.LocalDateTime.

A zone name may still be stored alongside such a column — Parquet's logical type itself keeps only isAdjustedToUTC, but PyArrow puts the original zone into the file's ARROW:schema metadata, and in Arrow IPC/Feather it is part of the schema as Timestamp(unit, tz). It is display metadata: the values are already normalized to UTC, so applying it would change nothing about which instant they name, and it is deliberately neither applied nor returned. Only the instant comes back; use convert { … }.with { it.toLocalDateTime(zone) } for wall-clock values in a zone you pick.