Interop with Collections
Kotlin DataFrame and Kotlin Collection represent two different approaches to data storage:
DataFramestores data by fields/columnsCollectionstores data by records/rows
Although DataFrame doesn't implement the Collection or Iterable interface, it has many similar operations, such as filter, take, first, map, groupBy etc.
DataFrame has two-way compatibility with Map and List:
List<T>->DataFrame<T>: toDataFrameDataFrame<T>->List<T>: toListMap<String, List<*>>->DataFrame<*>: toDataFrameDataFrame<*>->Map<String, List<*>>: toMapList<List<T>>->DataFrame<*>: toDataFrame
Columns, rows, and values of DataFrame can be accessed as List, Iterable and Sequence accordingly:
Interop with data classes
DataFrame can be used as an intermediate object for transformation from one data structure to another.
Assume you have a list of instances of some data class that you need to transform into some other format.
You can convert this list into DataFrame using toDataFrame() extension:
Mark the original data class with DataSchema annotation to get extension properties and perform data transformations.
After your data is transformed, DataFrame instances can be exported eagerly into List of another data class using toListOf:
toListOf works whatever the type argument of the DataFrame is, so use it in most cases. toList creates instances of the type argument itself, so use it only when this type argument is known to be a data class. With the compiler plugin, this is not the case here: the plugin gives df2 a generated type argument, which is not a data class, so df2.toList() throws.
A ColumnGroup becomes a nested data class:
Alternatively, toSequence and toSequenceOf create a lazy Sequence. The objects are created only when the sequence is iterated, so you do not need to hold all of them in memory at once.
The rules for matching columns to data class properties are in How rows become objects.
Converting columns with object instances to ColumnGroup
unfold turns columns of objects inside an existing dataframe into ColumnGroups, the way toDataFrame() builds a dataframe from a list of objects.