Aggregation

GroupBy aggregation

Each GroupBy can be directly transformed into a new DataFrame by applying one or more aggregation operations to its groups.

Aggregation is a generalization of reducing.

The following aggregation methods are available:

• concat — concatenates all rows from all groups into a single DataFrame, without preserving grouping keys;

• toDataFrame — returns this GroupBy as DataFrame with the grouping keys and corresponding groups in FrameColumn.

• concatWithKeys — a variant of concat that also includes grouping keys that were not present in the original DataFrame;

• into — creates a new column containing a list of values computed with a RowExpression for each group, or a new frame column containing the groups themselves;

• values — creates a DataFrame containing values collected into a single List from all rows of each group for the selected columns.

• count — creates a DataFrame containing the grouping key columns and an additional column with the number of rows in each corresponding group;

• countDistinct — creates a DataFrame containing the grouping key columns and an additional column with the number of distinct rows in each corresponding group;

• aggregate — performs a set of custom aggregations using AggregateDsl, allowing you to compute one or more derived values per group;

• Various aggregation statistics — predefined shortcuts for common statistical aggregations such as sum, mean, median, and others.

Each of these methods returns a new DataFrame that includes the grouping key columns (except for concat) along with the columns of values aggregated from the corresponding groups.

Check out GroupBy grammar.

For more information: See "GroupBy Aggregation" on the documentation website.