[BUG] Cannot read properties of undefined (reading 'length')

[ozalieg]

Steps to reproduce

1. Just run the jv pipeline I added at the end over terminal (jv <filename>).

Description

• Expected: no problems
• Actual:

        error: An unknown error occurred: TypeError: Cannot read properties of undefined (reading 'length')
$In /Users/lea/Documents/Uni/WS24_25/Thesis/ws2425-buchner-jayvee-templating/General/TemplateGeneration/JvFiles/BiostatsPipeline.jv:16:11
        16 |     block BiostatsCSVInterpreter oftype CSVInterpreter {
           |    

Example Pipeline where the error occurs

pipeline BiostatsPipeline {

    BiostatsExtractor
    -> BiostatsTextFileInterpreter
    -> BiostatsCSVInterpreter
    -> BiostatsTableInterpreter
    -> BiostatsLoader;

    block BiostatsExtractor oftype HttpExtractor {
        url: "https://people.sc.fsu.edu/~jburkardt/data/csv/biostats.csv";
    }

    block BiostatsTextFileInterpreter oftype TextFileInterpreter {
    }

    block BiostatsCSVInterpreter oftype CSVInterpreter {
        enclosing: "";
    }

    block BiostatsTableInterpreter oftype TableInterpreter {
        header: true;
        columns: [
                "Name" oftype text,
                "Sex" oftype text,
                "Age" oftype integer,
                "Height (in)" oftype integer,
                "Weight (lbs)" oftype integer
        ];
    }

    block BiostatsLoader oftype SQLiteLoader {
        table: "Biostats";
        file: "./Biostats.sqlite";
    }

}

[TungstnBallon]

The above TypeError will be fixed by #642.
However, the pipeline still doesn't work as intended, because the "Age" column contains numbers with a leading whitespace.
Jayvee (currently) doesn't consider these values numbers:

jayvee/libs/language-server/src/lib/ast/wrappers/value-type/primitive/decimal-value-type.ts

Line 10 in 186dcb7

const NUMBER_REGEX = /^[+-]?([0-9]*[,.])?[0-9]+([eE][+-]?\d+)?$/;

@joluj What would you say is the intended behavior here?

[joluj]

AFAIK leading whitespaces are part of the cell in CSV. Therefore it's the problem with the CSV.

However, that does not remove the real-world problem of having leading (and also trailing) whitespaces. I'd suggest adding an attribute skipLeadingWhitespaces and skipTrailingWhitespaces to the CSVInterpreter (similar to Python). I wouldn't modify the number definition.

@rhazn @georg-schwarz Opinions?

[rhazn]

I'd consider parsing ' 5' as the number 5 correct, so I'd just silently strip leading and trailing whitespace from number values honestly.

I think that's pretty intuitive and I'm struggling to come up with edge cases that would make it incorrect.

[joluj]

I think that's pretty intuitive and I'm struggling to come up with edge cases that would make it incorrect.

Me neither, however, I'm not too big of a fan of silently parsing the number. I'd prefer setting skipLeadingWhitespaces to true by default.

[georg-schwarz]

Or we need a transform function (and a nicer Syntax to apply those) :)

[TungstnBallon]

@ozalieg When using #648, I'm able to execute the following pipeline without crashes/errors.

pipeline BiostatsPipeline {

	BiostatsExtractor
		-> BiostatsTextFileInterpreter
		-> BiostatsCSVInterpreter
		-> BiostatsTableInterpreter
		-> BiostatsLoader;

	block BiostatsExtractor oftype HttpExtractor {
		url: "https://people.sc.fsu.edu/~jburkardt/data/csv/biostats.csv";
	}

	block BiostatsTextFileInterpreter oftype TextFileInterpreter { }

	block BiostatsCSVInterpreter oftype CSVInterpreter {
		enclosing: "";
	}

	block BiostatsTableInterpreter oftype TableInterpreter {
		header: true;
		columns: [
			'"Name"' oftype text,
			'     "Sex"' oftype text,
			' "Age"' oftype integer,
			' "Height (in)"' oftype integer,
			' "Weight (lbs)"' oftype integer
		];
	}

	block BiostatsLoader oftype SQLiteLoader {
		table: "Biostats";
		file: "./Biostats.sqlite";
	}
}

it results in this table:

| "Name" |      "Sex" |  "Age" |  "Height (in)" |  "Weight (lbs)" |
|--------|------------|--------|----------------|-----------------|
| "Alex" |        "M" | 41     | 74             | 170             |
| "Bert" |        "M" | 42     | 68             | 166             |
| "Carl" |        "M" | 32     | 70             | 155             |
| "Dave" |        "M" | 39     | 72             | 167             |
| "Elly" |        "F" | 30     | 66             | 124             |
| "Fran" |        "F" | 33     | 66             | 115             |
| "Gwen" |        "F" | 26     | 64             | 121             |
| "Hank" |        "M" | 30     | 71             | 158             |
| "Ivan" |        "M" | 53     | 72             | 175             |
| "Jake" |        "M" | 32     | 69             | 143             |
| "Kate" |        "F" | 47     | 69             | 139             |
| "Luke" |        "M" | 34     | 72             | 163             |
| "Myra" |        "F" | 23     | 62             | 98              |
| "Neil" |        "M" | 36     | 75             | 160             |
| "Omar" |        "M" | 38     | 70             | 145             |
| "Page" |        "F" | 31     | 67             | 135             |
| "Quin" |        "M" | 29     | 71             | 176             |
| "Ruth" |        "F" | 28     | 65             | 131             |