How to improve reading speed?

[shenlei149]

We have a lot of sample data for training, and we want to use nimble to replace parquet to 1) reduce file size and 2) reduce reading cost.

If we read some columns, for instance 10% or 20% columns, nimble is much faster than parquet. But if reading all columns, VeloxReader::next cost is the same as parquet reader, but we have to convert to arrow to integrate other processing system, so the total cost is more than cost of reading parquet file.

Here is example code. Is the way I use nimble right? If not, what's the best practice?

nimble::VeloxReader veloxReader(
    *leafPool.get(),
    readFile.get(),
    read_all ? nullptr
             : std::make_shared<velox::dwio::common::ColumnSelector>(
                   std::make_shared<velox::RowType>(
                       std::move(names), std::move(types))));

ArrowArray array;
ArrowSchema schema;

velox::VectorPtr data;
auto& table_reader = veloxReader.tabletReader();
auto nstripes = table_reader.stripeCount();
for (size_t i = 0; i < nstripes; i++) {
  auto nrows = table_reader.stripeRowCount(i);

  veloxReader.next(nrows, data);

  velox::exportToArrow(data, schema);
  velox::exportToArrow(data, array, leafPool.get());

  auto arrow_array = arrow::ImportArray(&array, &schema);
  auto ok = arrow_array.status();
}