concat
concat on DataFrame
Appends the rows of one or more DataFrame objects or an iterable of DataRow objects to a DataFrame. Rows from every input keep their order, and the inputs are processed in the order in which they are passed or iterated.
Returns a DataFrame containing the receiver's rows followed by all appended rows. The input objects are not modified.
In the overload with the single frame parameter, T is the schema type of the receiver DataFrame, and T1 is the schema type of the frame parameter. When enabled, the compiler plugin refines the declared DataFrame<Any> return type to a generated schema type that combines both input schemas.
This overload can be called using either infix or regular call syntax. Infix notation reads naturally when concatenation is the complete expression, such as val fullDf = df1 concat df2 or return df1 concat df2. When chaining another operation, prefer regular call syntax, such as df1.concat(df2).head(), instead of parenthesizing the infix expression.
When concat combines dataframes, it unifies their schemas:
The result contains the union of the input columns, ordered by their first appearance.
When two or more input schemas are combined, values from columns with the same name form one result column. It keeps the contributing columns' shared runtime type, except for its nullability. If their runtime types differ, the result type is inferred from the concatenated values. If all contributing columns are empty, their runtime types determine the result type. The result type is nullable only if the resulting values contain
null.If an input does not contain a result
ValueColumn, that column is filled withnullfor the input's rows and becomes nullable. A missingListcolumn is filled with empty lists, and a missingFrameColumnwith emptyDataFrames instead; these columns do not become nullable. A missingColumnGroupremains aColumnGroup, and its nested columns are filled according to these same rules recursively.
For overloads that append DataFrame objects, an appended DataFrame with no rows adds no rows, but its columns still participate in schema unification. Calling the vararg overload without any frames or passing an empty iterable of DataFrame objects returns the receiver as the same DataFrame instance.
For the overload that appends DataRow objects, a receiver with no rows contributes no columns to the result schema. An empty iterable of DataRow objects adds no rows and provides no additional schema to unify.
See also:
add, which adds columns to aDataFrame.append, which appends rows supplied as values or schema objects.join, which combines rows from two dataframes using matching keys or a condition.Multiple DataFrames, an overview of operations that combine dataframes.
Parameters
vararg frames: DataFrame | frames: Iterable<DataFrame>— theDataFrames whose rows are appended. Varargframesare processed in argument order; iterableframesare processed in iteration order.frame: DataFrame— a singleDataFrameappended using infix or regular call syntax.rows: Iterable<DataRow>— the rows to append, processed in iteration order.
Examples
The following DataFrame is used in examples that share one input:
Pass several dataframes as vararg arguments to append their rows in argument order:
Schema unification examples
The following two dataframes have different schemas. Their age columns also have different runtime types:
concat combines both schemas. The resulting age column contains Int and Double values and has the common runtime type Number. The missing city values are filled with null, so city becomes nullable.
New data can be added to a DataFrame as an iterable of rows:
An iterable of DataFrames can be appended in the same way:
concat on DataRow
Appends one or more rows to a DataRow. Returns a DataFrame containing the receiver followed by the supplied rows in argument order. If the rows have different schemas, they are unified according to the same rules as DataFrame.concat.
Parameters
vararg rows: DataRow— the rows to append in argument order.
Examples
DataRow.concat is useful when working with operations that produce rows:
concat on DataColumn
The result of concat on a DataColumn depends on the values stored in the column:
For value columns, concat appends the values from other in argument order and preserves the receiver's name. With one input column, its runtime type is preserved. With two or more input columns that have the same runtime type, the result keeps this type except for its nullability. If their types differ, the result type is inferred from the concatenated values. If all input columns are empty, their runtime types determine the result type. The result runtime type is nullable only if the resulting values contain null.
For a column of DataFrame values, concat combines the rows of all stored dataframes and performs schema unification. Dataframes are processed in their order of appearance, and row order within every dataframe is preserved. An empty column of DataFrame values produces an empty DataFrame. A stored dataframe with no rows adds no rows, but its columns still participate in schema unification.
For a column of Collection<T> values, it flattens the collections into a List<T> while preserving the order of both the column values and the elements in each collection.
Parameters
vararg other: DataColumn<T>— the columns whose values are appended to the receiver in argument order.
Examples
Concatenating value columns keeps the name of the receiver and finds a common type for the values:
A FrameColumn can store nested dataframes. concat() combines those dataframes in their order of appearance:
Concatenating a column of collections flattens all collection values. Empty collections add no elements:
concat on GroupBy
A GroupBy stores its grouping keys separately from the group DataFrames. GroupBy.concat() combines only those group dataframes, in group order, and preserves the row order within each group. It does not add the separate key columns. If a group already contains a column with the same name as a key, that original column remains unchanged.
concatWithKeys() also adds every grouping key column that is absent from a group. Each added key value is repeated for every row in its group. An existing group column with the same name is not overwritten.
ReducedGroupBy.concat() applies the stored reducer to every group and returns one result row per group, in group order.
If the reduced rows have different schemas, the result contains the union of their schemas. Columns are ordered by their first appearance in the reduced rows. Values missing from a reduced row follow the schema-unification rules for DataFrames.
If the reducer returns null for a group, the result still contains one row for that group. This row is treated as missing every result column, so its values are filled according to the same rules.
Examples
Here the grouping key ageGroup is created by an expression and is stored separately from the groups. The adult group appears before the teen group, so concatWithKeys() changes the row order to Alice, Charlie, Bob. It adds the ageGroup key column and repeats each key value for the rows in its group. The added column is highlighted in the result, and values of this column are colored according to the grouping expression:
concat() does not include the ageGroup key column:
After a reducing operation such as first, concat() applies the reducer and combines one selected row from every group:
concat on Iterable
concat() combines an Iterable of dataframes, columns, or nullable rows in iteration order:
For Iterable<DataFrame>, concat uses the same schema-unification rules as DataFrame.concat. A DataFrame with no rows adds no rows, but its columns still participate in schema unification. An empty iterable returns an empty DataFrame.
For Iterable<DataColumn>, the values form one result column. The result keeps the name of the first column and uses the same type rules as DataColumn.concat. An empty iterable returns an empty column.
For Iterable<DataRow?>, concat returns a DataFrame containing rows from the iterable in iteration order. Non-null rows follow the same schema-unification rules described for DataRow.concat. A null element contributes one result row that is treated as missing every result column, so its values follow the same missing-column rules. An empty iterable returns an empty DataFrame.
Examples
An iterable of dataframes is concatenated in iteration order:
An iterable of columns is concatenated into one column. Its name comes from the first input column:
A null element in an iterable of rows becomes a row with null values: