Being constructed on high of numpy made it arduous for pandas to deal with lacking values in a hassle-free, versatile manner, since numpy doesn’t help null values for some knowledge sorts.
As an example, integers are mechanically transformed to floats, which isn’t preferrred:
Be aware how factors mechanically modifications from int64 to float64 after the introduction of a singleNone worth.
There may be nothing worst for a knowledge circulate than incorrect typesets, particularly inside a data-centric AI paradigm.
Inaccurate typesets straight impression knowledge preparation selections, trigger incompatibilities between completely different chunks of knowledge, and even when passing silently, they may compromise sure operations that output nonsensical leads to return.
For instance, on the Data-Centric AI Community, we’re currenlty engaged on a venture round synthetic data for data privacy. One of many options, NOC (variety of youngsters), has lacking values and subsequently it’s mechanically transformed to float when the information is loaded. The, when passing the information right into a generative mannequin as a float , we’d get output values as decimals comparable to 2.5 — until you’re a mathematician with 2 children, a new child, and a bizarre humorousness, having 2.5 youngsters will not be OK.
In pandas 2.0, we will leverage dtype = 'numpy_nullable', the place lacking values are accounted for with none dtype modifications, so we will maintain our unique knowledge sorts (int64 on this case):
It’d seem to be a refined change, however below the hood it implies that now pandas can natively use Arrow’s implementation of coping with lacking values. This makes operations rather more environment friendly, since pandas doesn’t need to implement its personal model for dealing with null values for every knowledge kind.
