Data Schemas
The Kotlin DataFrame library provides typed data access via generation of extension properties for the type DataFrame<T> (as well as for DataRow<T>), where T is a marker class representing the DataSchema of the DataFrame.
A schema of a DataFrame is a mapping from column names to column types.
This data schema can be expressed as a Kotlin class or interface.
If the DataFrame is hierarchical — contains a column group or a column of dataframes — the data schema reflects this structure, with a separate class representing the schema of each column group or nested DataFrame.
For example, consider a simple hierarchical DataFrame from example.csv.
This DataFrame consists of two columns:
name, which is aStringcolumninfo, which is a column group containing two nested value columns:ageof typeIntheightof typeDouble
name | info | |
|---|---|---|
age | height | |
Alice | 23 | 175.5 |
Bob | 27 | 160.2 |
The data schema corresponding to this DataFrame can be represented as:
// Data schema of the "info" column group
@DataSchema
data class Info(
val age: Int,
val height: Float
)
// Data schema of the entire DataFrame
@DataSchema
data class Person(
val info: Info,
val name: String
)Extension properties for DataFrame<Person>
are generated based on this schema and allow accessing columns or using them in operations:
// Assuming `df` has type `DataFrame<Person>`
// Get "age" column from "info" group
df.info.age
// Select "name" and "height" columns
df.select { name and info.height }
// Filter rows by "age"
df.filter { age >= 18 }See Extension Properties API for more information.
Defining a data schema manually can be difficult, especially for dataframes with many columns or deeply nested structures, and may lead to mistakes in column names or types. Kotlin DataFrame provides several methods for generating data schemas.
generate..()methods are extensions forDataFrame(or for itsschema) that generate a code string representing itsDataSchema.Kotlin DataFrame Compiler Plugin cannot automatically infer a data schema from external sources such as files or URLs. However, it can infer the schema if you construct the
DataFramemanually — that is, by explicitly declaring the columns using the API. It will also automatically update the schema during operations that modify the structure of the DataFrame.
tip
For best results when working with the Compiler Plugin, it's recommended to generate the initial schema using one of the
generate..()methods. Once generated, the Compiler Plugin will automatically keep the schema up to date after any operations that change the structure of the DataFrame.
warning
The current Gradle plugin is under consideration for deprecation and may be officially marked as deprecated in future releases.
The KSP plugin is not compatible with KSP2 and may not work properly with Kotlin 2.1 or newer.
At the moment, data schema generation is handled via dedicated methods instead of relying on the plugins.
The Gradle plugin allows generating a data schema automatically by specifying a source file path in the Gradle build script.
The KSP plugin allows generating a data schema automatically using Kotlin Symbol Processing by specifying a source file path in your code file.
Once you have a data schema, you can generate extension properties.
The easiest and most convenient way is to use the Kotlin DataFrame Compiler Plugin, which generates extension properties on the fly for declared data schemas and automatically keeps them up to date after operations that modify the structure of the DataFrame.
warning
Extension properties generation was deprecated from the Gradle plugin in favor of the Compiler Plugin.
When using Kotlin DataFrame inside Kotlin Notebook, the schema and extension properties are generated automatically after each cell execution for all
DataFramevariables declared in that cell. See extension properties example in Kotlin Notebook.
tip
Compiler Plugin is coming to Kotlin Notebook soon.
If you're not using the Compiler Plugin, you can still generate extension properties for a
DataFramemanually by calling one of thegenerate..()methods with theextensionProperties = trueargument.