QuantileEstimationMethods

Quantile Estimation Methods

Both the median and the percentile are quantiles; the median is simply the 50th percentile. Unless a quantile lands exactly on one of the values, it needs to be estimated from the values around it. There are several common methods to do this. DataFrame follows the definitions of Hyndman, R. J. & Fan, Y. (1996). Sample Quantiles in Statistical Packages and Apache Commons Statistics, which call these methods R1 to R9.

DataFrame uses:

R8 for primitive numbers (Byte, Short, Int, Long, Float, and Double). It interpolates linearly between the two values around the quantile, so the result is a Double. R8 is the method recommended by Hyndman and Fan. Other libraries, like Numpy, default to R7, so slightly different results are to be expected. For the median, R8 gives the middle value, or the mean of the two middle values if the number of values is even.
R3 for all other self-comparable values (like strings or dates), since those cannot be interpolated, and for all -By operations (like medianBy and percentileBy), since those select an existing row or element, even if they compare numbers. R3 selects the value whose rank (its 1-based position among the sorted values) is the closest to N * p, where N is the number of values and p is the requested quantile, like 0.5 for the median. Ties are rounded to the even rank. The result is thus always one of the values itself.

Currently, only the default quantile estimation methods can be used. In the future, it might become possible to choose a different one, see #1121.