Density Plot

Usage

"Density" statistic is useful when you have a large dataset, and you want to understand the underlying probability distribution. Density plot visualizes the PDF and also allows you to compare the distribution of different samples. This is a useful alternative to the histogram for continuous data that comes from an underlying smooth distribution.

Arguments

• Input (mandatory):

  • x — numeric sample on which the statistics are calculated

• Weights (optional):

  • weights — set of weights of the same size as the input sample. null (by default) means all weights are equal to 1.0 and the weighted density is equal to the normal one

• Parameters (optional):

  • n: Int — number of sampled points;

  • trim: Boolean — if false, each density is computed on the full range of the data, if true, each density is computed over the range of that group (only for grouped inputs).

  • adjust: Double — adjusts the value of bandwidth by multiplying it; changes how smooth the frequency curve is.

  • kernel: Kernel — the kernel used to calculate the density function:

    • Kernel.GAUSSIAN

    • Kernel.RECTANGULAR

    • Kernel.TRIANGULAR

    • Kernel.BIWEIGHT

    • Kernel.EPANECHNIKOV

    • Kernel.OPTCOSINE

  • fullScanMax: Int — maximum size of data to use density computation with 'full scan'. For bigger data, less accurate but more efficient density computation is applied

  • bandWidth: BandWidth — the method (or exact value) of bandwidth:

    • BandWidth.Method.NRD

    • BandWidth.Method.NRD0

    • BandWidth.value(value: Double)

Output statistics

StatDensity plots

df has a signature

Let's take a look at StatDensity output DataFrame:

It has the following signature:

As you can see, we got a DataFrame with one ColumnGroup called Stat which contains several columns with statics. For statDensity, each row corresponds to one PDF point. Stat.x is the column with this point x coordinate. Stat.density contains the estimated density. Stat.densityWeighted — weighted version of density. Stat.scaled is a density scaled to a maximum of 1.0. Stat.scaledWeighted — weighted version of scaled. DataFrame with "density" statistics is called StatDensityFrame

statDensity transform

statDensity(statDensityArgs) { /*new plotting context*/ } modifies a plotting context — instead of original data (no matter was it empty or not) new StatDensity dataset (calculated on given arguments, inputs and weights can be provided as Iterable or as dataset column reference — by name as a String, as a ColumnReference or as a DataColumn) is used inside a new context (original dataset and primary context are not affected — you can add layers using initial dataset outside the statDensity context). Since the old dataset is irrelevant, we cannot use references for its columns. But we can refer to the new ones. They are all contained in the Stat group and can be called inside the new context:

plot { statDensity(depthList, adjust = 0.2) { // New `StatDensity` dataset here line { // Use `Stat.*` columns for mappings x(Stat.x) y(Stat.density) color(Stat.density) } } }

DensityPlot layer

Density plot is a statistical plot used for visualizing the distribution of continuous variables. It's an area graph of kernel-estimated PDF. So basically, we can build a histogram with statDensity as follows:

val statDensityAndAreaPlot = df.plot { statDensity("depth") { area { x(Stat.x) y(Stat.density) } } layout.title = "`statDensity()` + `area()` layer" } statDensityAndAreaPlot

But we can do it even faster with densityPlot(statDensityArgs) method:

val densityLayerPlot = plot { densityPlot(depthList) layout.title = "`densityPlot()` layer" } densityLayerPlot

Let's compare them:

plotGrid(listOf(statDensityAndAreaPlot, densityLayerPlot))

These two plots are identical. Indeed, densityPlot just uses statDensity and area and performs coordinate mappings under the hood. And we can customize densityPlot layer: densityPlot() optionally opens a new context, where we can configure bars (as in the usual context opened by area { ... }) — even change coordinate mappings from default ones. StatDensity dataset of densityPlot is also can be accessed here.

df.plot { densityPlot(depth) { // Change a column mapped on `y` to `Stat.scaled` y(Stat.scaled) alpha = 0.7 fillColor = Color.RED borderLine.color = Color.BLACK } }

If we specify weights, Stat.densityWeighted is mapped to y by default:

df.plot { densityPlot(depth, coeff, n = 700, adjust = 0.8, bandWidth = BandWidth.value(17.0)) // We can add other layers as well. // Let's add a horizontal mark line with constant y intercept: vLine { // Count sample mean val mean = depth.mean() xIntercept.constant(mean) tooltips { line("Depth mean: ${String.format("%.2f", mean)}m") } color = Color.RED; width = 2.0 } x.axis.name = "depth, m" }

densityPlot plot

densityPlot(statDensityArgs) and DataFrame.densityPlot(statDensityArgs) are a family of functions for fast plotting a density plot.

densityPlot(depthList, kernel = Kernel.COSINE)

df.densityPlot("depth")

In case you want to provide input and weights using column selection DSL, it's a bit different from the usual one — you should assign x input and (optionally) weight throw invocation eponymous functions:

df.densityPlot(adjust = 0.5) { x(depth) weight(coeff) }

densityPlot plot can be configured with .configure {} extension — it opens context that combines area, StatDensity and plot context. That means you can configure bars settings, mappings using StatDensity dataset and any plot adjustments:

df.densityPlot { x(depth) }.configure { // Area + StatDensity + PlotBuilder // Can't add new layer // Can add area mapping, including for `Stat.*` columns fillColor(Stat.scaled) // doesn't work properly for now alpha = 0.6 // Can configure general plot adjustments layout { title = "Configured `densityPlot` plot" size = 600 to 350 } }

Grouped statDensity

statDensity can be applied for grouped data — statistics will be calculated on each group independently but with equal categories. This application returns a new GroupBy dataset with the same keys as the old one but with StatDensity groups instead of old ones.

It has the following signature:

Now we have a GroupBy with a signature

After statDensity applying it's still a GroupBy but with different signature of group - all groups have the same signature as usual DataFrame after statDensity applying (i.e. StatDensityFrame):

As you can see, we did indeed do a statDensity transformation within groups, the grouping keys did not change. The plotting process doesn't change much — we do everything the same.

As you can see, there are two lines because we have two groups of data. To distinguish them, we need to add mapping to the color from the key. This is convenient — the key is available in the context

The densityPlot() layer also works. Moreover, if we have exactly one grouping key, a mapping from it to fillColor and borderLine.color will be created by default.

We can customize it like we used to. From the differences — access to key columns:

Also, we can stack areas (for that we need x coordinates to match — use trim = true):

densityPlot plot for GroupBy (i.e. GroupBy.densityPlot(statDensityArgs) extensions) works as well:

... and can be configured the same way:

Inside groupBy{} plot context

We can apply groupBy modification to the initial dataset and build a density plot with grouped data the same way:

rangesDF.plot { groupBy(category) { densityPlot(range) } }