CSVIter crashes claiming different row length

[xafilox]

Hi.

I'm trying to use CSVIter to load a dataset I have in CSV format. The data file has 25 columns (some of the can be empty, especially the last columns).

While executing, it outputs the following error trace:

julia> using MXNet;

julia> train_provider = mx.CSVIter(data_csv = "/home/laketide/work/Alejandro/output_training_data.csv", data_shape =(1,25), label_csv = "/home/laketide/work/Alejandro/output_training_labels.csv", label_shape = (1,2), batch_size = 1)
[15:45:59] /home/laketide/mxnet/dmlc-core/include/dmlc/./logging.h:300: [15:45:59] src/io/iter_csv.cc:105: Check failed: row.length == shape.Size() (24 vs. 25) The data size in CSV do not match size of shape: specified shape=(25,1), the csv row-length=24

Stack trace returned 9 entries:
[bt] (0) /home/laketide/mxnet/lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f5f4df6bfac]
[bt] (1) /home/laketide/mxnet/lib/libmxnet.so(_ZN5mxnet2io7CSVIter7AsTBlobERKN4dmlc3RowIjEERKN4nnvm6TShapeE+0x3aa) [0x7f5f4e73ae2a]
[bt] (2) /home/laketide/mxnet/lib/libmxnet.so(_ZN5mxnet2io7CSVIter4NextEv+0x19c) [0x7f5f4e73e5ac]
[bt] (3) /home/laketide/mxnet/lib/libmxnet.so(_ZN5mxnet2io11BatchLoader4NextEv+0x87) [0x7f5f4e74d617]
[bt] (4) /home/laketide/mxnet/lib/libmxnet.so(ZZN5mxnet2io14PrefetcherIter4InitERKSt6vectorISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_ESaISA_EEENKUlPPNS_9DataBatchEE_clESH+0x39) [0x7f5f4e7406d9]
[bt] (5) /home/laketide/mxnet/lib/libmxnet.so(_ZZN4dmlc12ThreadedIterIN5mxnet9DataBatchEE4InitESt8functionIFbPPS2_EES4_IFvvEEENKUlvE_clEv+0x1ff) [0x7f5f4e746d0f]
[bt] (6) /usr/bin/../lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f61912c9c80]
[bt] (7) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f619304e6ba]
[bt] (8) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f6192d8482d]

terminate called after throwing an instance of 'dmlc::Error'
what(): [15:45:59] src/io/iter_csv.cc:105: Check failed: row.length == shape.Size() (24 vs. 25) The data size in CSV do not match size of shape: specified shape=(25,1), the csv row-length=24

Stack trace returned 9 entries:
[bt] (0) /home/laketide/mxnet/lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f5f4df6bfac]
[bt] (1) /home/laketide/mxnet/lib/libmxnet.so(_ZN5mxnet2io7CSVIter7AsTBlobERKN4dmlc3RowIjEERKN4nnvm6TShapeE+0x3aa) [0x7f5f4e73ae2a]
[bt] (2) /home/laketide/mxnet/lib/libmxnet.so(_ZN5mxnet2io7CSVIter4NextEv+0x19c) [0x7f5f4e73e5ac]
[bt] (3) /home/laketide/mxnet/lib/libmxnet.so(_ZN5mxnet2io11BatchLoader4NextEv+0x87) [0x7f5f4e74d617]
[bt] (4) /home/laketide/mxnet/lib/libmxnet.so(ZZN5mxnet2io14PrefetcherIter4InitERKSt6vectorISt4pairINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES9_ESaISA_EEENKUlPPNS_9DataBatchEE_clESH+0x39) [0x7f5f4e7406d9]
[bt] (5) /home/laketide/mxnet/lib/libmxnet.so(_ZZN4dmlc12ThreadedIterIN5mxnet9DataBatchEE4InitESt8functionIFbPPS2_EES4_IFvvEEENKUlvE_clEv+0x1ff) [0x7f5f4e746d0f]
[bt] (6) /usr/bin/../lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f61912c9c80]
[bt] (7) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f619304e6ba]
[bt] (8) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f6192d8482d]

signal (6): Aborted
while loading no file, in expression starting on line 0
Allocations: 3195248 (Pool: 3193533; Big: 1715); GC: 3
Aborted (core dumped)

I've checked all the lines. All of them have 24 commas, so it has 25 columns.

Thanks in advance.

[pluskid]

Can you produce a minimum CSV file that will trigger the bug? Something like 2 lines of CSV, and paste the contents here.

[xafilox]

Hi.
Thanks for taking care of this problem.
Here is an example of data file:

tower_1_latitude,tower_1_longitude,tower_1_height,tower_1_azimuth,tower_1_horizontal_beam_width,tower_1_vertical_beam_width,tower_1_pathloss,tower_2_latitude,tower_2_longitude,tower_2_height,tower_2_azimuth,tower_2_horizontal_beam_width,tower_2_vertical_beam_width,tower_3_latitude,tower_3_longitude,tower_3_height,tower_3_azimuth,tower_3_horizontal_beam_width,tower_3_vertical_beam_width,tower_4_latitude,tower_4_longitude,tower_4_height,tower_4_azimuth,tower_4_horizontal_beam_width,tower_4_vertical_beam_width
35.56638461303711,12.345677280273438,15.0,255,120,6,96.0,,,,,,,,,,,,,,,,,,
12.34567461303711,12.345677280273438,15.0,255,120,6,96.0,,,,,,,,,,,,,,,,,,

And labels file:

latitude,longitude
12.34567811594422,12.345671046080622
12.34567811594422,12.345671046080622

[xafilox]

Hi @pluskid . Did you manage to reproduce the problem with the files I gave you?
Thanks