dataframe-arrow/org.jetbrains.kotlinx.dataframe.io/ArrowWriter

ArrowWriter

Save dataFrame content in Apache Arrow format (can be written to File, ByteArray, OutputStream or raw Channel) with targetSchema. If dataFrame content does not match targetSchema, the behaviour is specified by mode, and mismatches would be sent to mismatchSubscriber

Inheritors

ArrowWriterImpl

Types

Companion

object Companion

Mode

data class Mode(val restrictWidening: Boolean, val restrictNarrowing: Boolean, val strictType: Boolean, val strictNullable: Boolean)

If restrictWidening is true, dataFrame columns not described in targetSchema would not be saved (otherwise, would be saved as is). If restrictNarrowing is true, targetSchema fields that are not nullable and do not exist in dataFrame will produce an exception (otherwise, would not be saved). If strictType is true, dataFrame columns described in targetSchema with non-compatible type will produce an exception (otherwise, would be saved as is). If strictNullable is true, targetSchema fields that are not nullable and contain nulls in dataFrame will produce an exception (otherwise, would be saved as is with nullable = true).

Properties

dataFrame

abstract val dataFrame: DataFrame<*>

mismatchSubscriber

abstract val mismatchSubscriber: (ConvertingMismatch) -> Unit

mode

abstract val mode: ArrowWriter.Mode

targetSchema

abstract val targetSchema: Schema

Functions

allocateVectorSchemaRoot

abstract fun allocateVectorSchemaRoot(): VectorSchemaRoot

Create Arrow VectorSchemaRoot with dataFrame content cast to targetSchema according to the mode.

abstract fun close()

saveArrowFeatherToByteArray

open fun saveArrowFeatherToByteArray(): ByteArray

Save data to Arrow random access format, write to new ByteArray

saveArrowIPCToByteArray

open fun saveArrowIPCToByteArray(): ByteArray

Save data to Arrow interprocess streaming format, write to new ByteArray

writeArrowFeather

open fun writeArrowFeather(file: File)

Save data to Arrow random access format, write to new or existing file. If file exists, it would be recreated.

open fun writeArrowFeather(stream: OutputStream)

Save data to Arrow random access format, write to opened stream.

open fun writeArrowFeather(channel: WritableByteChannel)

Save data to Arrow random access format, write to opened channel.

open fun writeArrowFeather(path: Path)

Save data to Arrow random access format, write to a new or existing file on the given path. If file on the given path exists, it would be recreated.

writeArrowIPC

open fun writeArrowIPC(stream: OutputStream)

Save data to Arrow interprocess streaming format, write to opened stream.

open fun writeArrowIPC(channel: WritableByteChannel)

Save data to Arrow interprocess streaming format, write to opened channel.

open fun writeArrowIPC(file: File, append: Boolean = true)

Save data to Arrow interprocess streaming format, write to new or existing file. If file exists, it can be recreated or expanded.

open fun writeArrowIPC(path: Path, append: Boolean = true)

Save data to Arrow interprocess streaming format, write to a new or existing file on the given path. If file on the given path exists, it can be recreated or expanded.