parse
Returns a DataFrame
in which the given String
columns are parsed into other types.
This is a special case of the convert operation.
This parsing operation is sometimes executed implicitly, for example, when reading from CSV or type converting from String
columns. You can recognize this by the locale
or parserOptions
arguments in these functions.
To parse only particular columns use a column selector:
Parsing Order
parse
tries to parse every String
column into one of supported types in the following order:
Int
Long
Instant
(kotlinx.datetime
andjava.time
)LocalDateTime
(kotlinx.datetime
andjava.time
)LocalDate
(kotlinx.datetime
andjava.time
)Duration
(kotlin.time
andjava.time
)LocalTime
(java.time
)URL
(java.net
)Boolean
BigDecimal
JSON
(arrays and objects) (requires theorg.jetbrains.kotlinx:dataframe-json
dependency)
Parser Options
DataFrame supports multiple parser options that can be used to customize the parsing behavior. These can be supplied to the parse
function (or any other function that can implicitly parse Strings
) as an argument.
For each option you don't supply (or supply null
) DataFrame will take the value from the Global Parser Options.
Available parser options:
locale: Locale
is used to parse doublesGlobal default locale is
Locale.getDefault()
dateTimePattern: String
is used to parse date and timeGlobal default supports ISO (local) date-time
dateTimeFormatter: DateTimeFormatter
is used to parse date and timeIs derived from
dateTimePattern
and/orlocale
ifnull
nullStrings: List<String>
is used to treat particular strings asnull
valueGlobal default null strings are "null" and "NULL"
When reading from CSV, we include even more defaults, like "", and "NA". See the KDocs there for the exact details
skipTypes: Set<KType>
types that should be skipped during parsingEmpty set by global default; parsing can result in any supported type
useFastDoubleParser: Boolean
is used to enable or disable the new fast double parserEnabled by global default
Global Parser Options
As mentioned before, you can change the default global parser options that will be used by read
, convert
, and other parse
operations. Whenever you don't explicitly provide parser options to a function call, DataFrame will use these global options instead.
For example, to change the locale to French and add a custom date-time pattern for all following DataFrame calls, do:
For locale
, this means that the one being used by the parser is defined as:
↪ The locale given as function argument directly, or in parserOptions
, if it is not null
, else
↪ The locale set by DataFrame.parser.locale = ...
, if it is not null
, else
↪ Locale.getDefault()
, which is the system's default locale that can be changed with Locale.setDefault()
.
Parsing Doubles
DataFrame has a new fast and powerful double parser enabled by default. It is based on the FastDoubleParser library for its high performance and configurability (in the future, we might expand this support to Float
, BigDecimal
, and BigInteger
as well).
The parser is locale-aware; it will use the locale set by the (global) parser options to parse the doubles. It also has a fallback mechanism built in, meaning it can recognize characters from all other locales (and some from Wikipedia) and parse them correctly as long as they don't conflict with the current locale.
For example, if your locale uses ',' as decimal separator, it will not recognize ',' as thousands separator, but it will recognize ''', ' ', '٬', '_', ' ', etc. as such. The same holds for characters like "e", "inf", "×10^", "NaN", etc. (ignoring case).
This means you can safely parse "123'456 789,012.345×10^6"
with a US locale but not "1.234,5"
.
Aside from this, DataFrame also explicitly recognizes "∞", "inf", "infinity", and "infty" as Double.POSITIVE_INFINITY
(as well as their negative counterparts), "nan", "na", and "n/a" as Double.NaN
, and all forms of whitespace are treated equally.
If FastDoubleParser
fails to parse a String
as Double
, DataFrame will try to parse it using the standard NumberFormat.parse()
function as a last resort.
If you experience any issues with the new parser, you can turn it off by setting useFastDoubleParser = false
, which will use the old NumberFormat.parse()
function instead.
Please report any issues you encounter.