mapWithState
fun <K, V, StateType, MappedType> JavaDStream<Tuple2<K, V>>.mapWithState( spec: StateSpec<K, V, StateType, MappedType>): JavaMapWithStateDStream<K, V, StateType, MappedType>
Return a JavaMapWithStateDStream by applying a function to every key-value element of this stream, while maintaining some state data for each unique key. The mapping function and other specification (e.g. partitioners, timeouts, initial state data, etc.) of this transformation can be specified using StateSpec class. The state data is accessible in as a parameter of type State in the mapping function.
Example of using mapWithState:
// A mapping function that maintains an integer state and return a String
fun mappingFunction(key: String, value: Optional<Int>, state: State<Int>): Optional<String> {
// Use state.exists(), state.get(), state.update() and state.remove()
// to manage state, and return the necessary string
}
val spec = StateSpec.function(::mappingFunction).numPartitions(10)
val mapWithStateDStream = keyValueDStream.mapWithState<StateType, MappedType>(spec)Content copied to clipboard
Parameters
spec
Specification of this transformation