KSparkStreamingSession

class KSparkStreamingSession(val ssc: JavaStreamingContext) : Serializable

This wrapper over SparkSession and JavaStreamingContext provides several additional methods to create org.apache.spark.sql.Dataset

Constructors

constructor(ssc: JavaStreamingContext)

val ssc: JavaStreamingContext

fun getSpark(sc: SparkConf): SparkSession

Creates new spark session from given sc.

fun getSpark(rddForConf: JavaRDDLike<*, *>): SparkSession

Creates new spark session from context of given JavaRDD, rddForConf.

fun getSpark(sscForConf: JavaStreamingContext): SparkSession

Creates new spark session from context of given JavaStreamingContext, sscForConf

block will be run after the streaming session has started from a new context (so not when loading from a checkpoint) and before it's terminated.

fun <T> withSpark(sc: SparkConf, func: KSparkSession.() -> T): T

Helper function to enter Spark scope from sc like

fun <T> withSpark(rddForConf: JavaRDDLike<*, *>, func: KSparkSession.() -> T): T

Helper function to enter Spark scope from a provided like when using the foreachRDD function.

fun <T> withSpark(sscForConf: JavaStreamingContext, func: KSparkSession.() -> T): T

Helper function to enter Spark scope from sscForConf like