bug: incorrect handling of multi-line strings

[pranasziaukas]

Meltano Version

3.0.0

Python Version

3.11

Bug scope

Configuration (settings parsing, validation, etc.)

Operating System

macOS - Venture 13.5.2

Description

I am passing a multi-line string to the tap in the form of an environment variable

TAP_POSTGRES_SSH_TUNNEL_PRIVATE_KEY='-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY——'

and it fails due to an invalid value.

If I check out the tap repository locally and provide the relevant constructor with the same key, it works fine.

As a result, I believe that multi-line strings are handled incorrectly somewhere along the way, specifically when Meltano builds and passes configuration downstream.

Related MeltanoLabs/tap-postgres#241

Code

> meltano config tap-postgres test

2023-09-20T12:05:19.866936Z [info     ] The default environment 'dev' will be ignored for `meltano config`. To configure a specific environment, please use the option `--environment=<environment name>`.
Need help fixing this problem? Visit http://melta.no/ for troubleshooting steps, or to
join our friendly Slack community.

Plugin configuration is invalid
Catalog discovery failed: command ['/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/bin/tap-postgres', '--config', '/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/run/tap-postgres/tap.598e019d-53d3-4dab-99ff-e2240091d9d6.config.json', '--discover'] returned 1 with stderr:
 Traceback (most recent call last):
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/bin/tap-postgres", line 8, in <module>
    sys.exit(TapPostgres.cli())
             ^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 1077, in main
    with self.make_context(prog_name, args, **extra) as ctx:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 943, in make_context
    self.parse_args(ctx, args)
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 1408, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 2400, in handle_parse_result
    value = self.process_value(ctx, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/click/core.py", line 2362, in process_value
    value = self.callback(ctx, self, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 527, in cb_discover
    tap.run_discovery()
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 287, in run_discovery
    catalog_text = self.catalog_json_text
                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/singer_sdk/tap_base.py", line 307, in catalog_json_text
    return json.dumps(self.catalog_dict, indent=2)
                      ^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 475, in catalog_dict
    result["streams"].extend(self.connector.discover_catalog_entries())
                             ^^^^^^^^^^^^^^
  File "/Users/pranas/.pyenv/versions/3.11.3/lib/python3.11/functools.py", line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 378, in connector
    url = self.ssh_tunnel_connect(ssh_config=ssh_config, url=url)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 429, in ssh_tunnel_connect
    ssh_private_key=self.guess_key_type(ssh_config["private_key"]),
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pranas/Repositories/balena-io-experimental/analytics-pipeline-v2/.meltano/extractors/tap-postgres/venv/lib/python3.11/site-packages/tap_postgres/tap.py", line 414, in guess_key_type
    raise ValueError(errmsg)
ValueError: Could not determine the key type.

[visch]

Multi line strings do work :D

.env files accept multiline see https://saurabh-kumar.com/python-dotenv/#multiline-values

Passing the value in via an env variable is doable as well it just takes pasting it in properly important. If there's a way to document this better after you figure it out please help out! :D

https://docs.meltano.com/cloud/platform/#ssh-private-keys-for-private-git-repository-package-access is how folks are directed to set this via cloud. Maybe we should have a tutorial for how to do this in the Meltano command line?

Personally I've just set it via the .env file

[edgarrmondragon]

Might be helped by something like #8177

[edgarrmondragon]

Closing now since we have https://docs.meltano.com/reference/command-line-interface#how-to-use-read-setting-value-from-a-file