kotlin-spark-api_3.3.2_2.13/org.jetbrains.kotlinx.spark.api/mapGroupsWithState

mapGroupsWithState

inline fun <K, V, S, U> KeyValueGroupedDataset<K, V>.mapGroupsWithState(    noinline func: (key: K, values: Iterator<V>, state: GroupState<S>) -> U): Dataset<U>

(Kotlin-specific) Applies the given function to each group of data, while maintaining a user-defined per-group state. The result Dataset will represent the objects returned by the function. For a static batch Dataset, the function will be invoked once per group. For a streaming Dataset, the function will be invoked for each group repeatedly in every trigger, and updates to each group's state will be saved across invocations. See org.apache.spark.sql.streaming.GroupState for more details.

Parameters

S

The type of the user-defined state. Must be encodable to Spark SQL types.

U

The type of the output objects. Must be encodable to Spark SQL types.

func

Function to be called on every group.

See Encoder for more details on what types are encodable to Spark SQL.

inline fun <K, V, S, U> KeyValueGroupedDataset<K, V>.mapGroupsWithState(    timeoutConf: GroupStateTimeout,     noinline func: (key: K, values: Iterator<V>, state: GroupState<S>) -> U): Dataset<U>

(Kotlin-specific) Applies the given function to each group of data, while maintaining a user-defined per-group state. The result Dataset will represent the objects returned by the function. For a static batch Dataset, the function will be invoked once per group. For a streaming Dataset, the function will be invoked for each group repeatedly in every trigger, and updates to each group's state will be saved across invocations. See org.apache.spark.sql.streaming.GroupState for more details.

Parameters

S

The type of the user-defined state. Must be encodable to Spark SQL types.

U

The type of the output objects. Must be encodable to Spark SQL types.

func

Function to be called on every group.

timeoutConf

Timeout configuration for groups that do not receive data for a while.

See Encoder for more details on what types are encodable to Spark SQL.