Missing Data Filter

[phorne-uncharted]

Depending on the column type, having a filter on the column could result in rows with missing data also being excluded. For example, if field A is a timestamp and has missing values, then excluding data with values between 2010-01-01 and 2011-01-01 would also exclude rows where A is missing. It is unclear if that is desired behaviour or not. Letting the user specify what to do can also get messy quickly as generating the appropriate queries all the time may not be the easiest thing to do, especially on type switches.

There is also a more fundamental problem with filters and missing data. The primitives themselves do not support missing data and crash if ANY row has missing data (datetime and numerical definitely, others unsure). The only way to really handle this would be to add an initial filter on all fields where missing data could be present to explicitly exclude empty rows BEFORE parsing the data to the right type. But then that becomes very annoying very quickly, and for potentially little gain.

A decision needs to be made at some point to determine exactly the kind of missing data support Distil needs to offer.