How can one set the csv_delimiter option to a literal Tab?

[daviewales]

More generally, how can one enter invisible characters into the Visidata options page?

Motivation: On Windows, many programs produce output with CRLF line endings.
If the output is CSV, then all is well, the CSV reader will automatically handle this.

But if the output is TSV, then we have a problem, because the TSV reader expects a single character line ending.

I thought that I could have my cake and eat it by opening a TSV as CSV with csv_delimiter set to Tab.
But I can't figure out how to set the option...
\t doesn't seem to work.
And pasting a literal Tab into the cell tells me that the clipboard is empty...

This will be particularly useful when pasting TSV tables from the clipboard with gP.

[midichef]

To paste a special character into any cell, like the csv_delimiter option value, do e for edit mode then press Ctrl+V then the special character, in this case Tab.

Or you can set the option from the command line: do the Windows equivalent of the Linux vd -f csv --csv-delimiter="\t". I'm not familiar enough with Windows to know the precise way to format that command.

To set alternate line delimiters with the tsv parser, you'd want to set the row_delimiter option:
vd -f tsv --row-delimiter="\r\n"
The delimiter option also exists, to change the field separator: --delimiter='|'

But I'm surprised any special line ending handling is necessary. On my Ubuntu system, the tsv parser handles files where lines end with '\r\n', with no problem. This sample file tsv-crlf.tsv has lines separated by '\r\n'. Visidata opens it as I'd expect, in v3.0.2:
tsv-crlf.tsv.txt
The tsv loader uses open_text_source(), so the file should be read line by line as text, in universal newline mode, so that lines in a single file will end when any of these characters are seen: '\n', '\r', '\r\n'.

[midichef]

No, it does not work when I use gP to paste, from the system clipboard, a tsv that uses '\r\n' to separate lines. (Ubuntu 24.04, develop branch)

I'll look into it. There is likely a bug here that is not specific to Windows.

[midichef]

As a short term workaround, you can add this to .visidatarc:

# 2024-10-04 temporary workaround for open-syspaste problem with \r\n on Windows
@BaseSheet.api
def open_syspaste(sheet, filetype='tsv'):
    import io

    v = vd.sysclipValue().replace('\r\n','\n').strip() or vd.fail('nothing to open')
    p = Path('syspaste'+'.'+filetype, fp=io.BytesIO(v.encode('utf-8')))
    return vd.openSource(p, filetype=filetype)

The only change from the default open_syspaste() is that it reads the clipboard, then pretends any '\r\n' were actually '\n'.

[daviewales]

Thanks!

[daviewales]

Should we make this a bug rather than a discussion then?

[midichef]

Yes, please open an Issue.