Broken in 3.8 due to bad import from marrow/cgi

[indistinctTalk]

Looks like it's broken on python 3.8.1 currently.

Installs fine:

❯ pip install marrow.mailer --user                                                                                                         
Collecting marrow.mailer
  Using cached https://files.pythonhosted.org/packages/8c/37/961103b20bad1a8af86fa67a53748c2bf26e9e242a79fa89dc762d374ea6/marrow.mailer-4.0.3-py2.py3-none-any.whl
Collecting marrow.util<2.0
  Using cached https://files.pythonhosted.org/packages/d2/19/630a0984ba6d5dc4432ed45d2c9ab8752542fa6ba1590a0e72d1fb742c2e/marrow.util-1.2.3.tar.gz
Installing collected packages: marrow.util, marrow.mailer
    Running setup.py install for marrow.util ... done
Successfully installed marrow.mailer-4.0.3 marrow.util-1.2.3

Running a script with the example code:

from marrow.mailer import Mailer, Message

mailer = Mailer(dict(
        transport = dict(
                use = 'smtp',
                host = 'localhost')))
mailer.start()

message = Message(author="[email protected]", to="[email protected]")
message.subject = "Testing Marrow Mailer"
message.plain = "This is a test."
mailer.send(message)

mailer.stop()

Traceback (most recent call last):
  File "/home/liam/projects/soc_siem_bi_integration/soc_scripts/cleanup.py", line 8, in <module>
    from marrow.mailer import Mailer, Message
  File "/home/liam/.local/lib/python3.8/site-packages/marrow/mailer/__init__.py", line 12, in <module>
    from marrow.mailer.message import Message
  File "/home/liam/.local/lib/python3.8/site-packages/marrow/mailer/message.py", line 21, in <module>
    from marrow.mailer.address import Address, AddressList, AutoConverter
  File "/home/liam/.local/lib/python3.8/site-packages/marrow/mailer/address.py", line 12, in <module>
    from marrow.util.compat import basestring, unicode, unicodestr, native
  File "/home/liam/.local/lib/python3.8/site-packages/marrow/util/compat.py", line 33, in <module>
    from cgi import parse_qsl
ImportError: cannot import name 'parse_qsl' from 'cgi' (/usr/lib/python3.8/cgi.py)

If I go diving into compat.py and change from cgi import parse_qsl to from urllib.parse import parse_qsl, it works fine. This is makes sense, given Python 3.8 notes:

parse_qs, parse_qsl, and escape are removed from the cgi module. They are deprecated in Python 3.2 or older. They should be imported from the urllib.parse and html modules instead.

I don't know the ramifications of using urllib.parse over html, though.

Util is archived, otherwise I would have raised the issue there /shrug.

[nphilipp]

I don't know the ramifications of using urllib.parse over html, though.

It's just that parse_qs and parse_qsl live in urllib.parse, escape lives in html.

Util is archived, otherwise I would have raised the issue there /shrug.

Yeah, it's kinda odd that marrow.mailer still depends on it.

[amcgregor]

I don't know the ramifications of using urllib.parse over html, though.

Given it's not even used, the solution here is removal of that dependency, which is primarily there to support Python 2 compatibility, which is no longer a desired feature or concern. There are no ramifications.

[iamareebjamal]

Yes, parse_ is not present in this repo, because it is present in marrow.util. Which is indeed still used in quite a few places in the project https://github.com/marrow/mailer/search?q=marrow.util&unscoped_q=marrow.util

[amcgregor]

@iamareebjamal Reference my recent comment on #90 — with a properly quoted search you find the summary included at the end of that comment. That is: virtually none of marrow.util is required, except the compatibility for Python 2, which can go away, and Bunch… which is just an over-engineered attribute access dictionary. Splitting the discussion, though, is quite unfortunate, on this. #90 locked.

This issue is an actual issue, where the other is not.

[tyoc213]

Looks like it's broken on python 3.8.1 currently.

Installs fine:

Indeed it installs fine but importing it causes this

$ python
Python 3.8.2 (default, Mar 26 2020, 15:53:00) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from marrow.mailer import Message, Delivery
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tyoc213/miniconda3/envs/diccionario/lib/python3.8/site-packages/marrow/mailer/__init__.py", line 12, in <module>
    from marrow.mailer.message import Message
  File "/home/tyoc213/miniconda3/envs/diccionario/lib/python3.8/site-packages/marrow/mailer/message.py", line 21, in <module>
    from marrow.mailer.address import Address, AddressList, AutoConverter
  File "/home/tyoc213/miniconda3/envs/diccionario/lib/python3.8/site-packages/marrow/mailer/address.py", line 12, in <module>
    from marrow.util.compat import basestring, unicode, unicodestr, native
  File "/home/tyoc213/miniconda3/envs/diccionario/lib/python3.8/site-packages/marrow/util/compat.py", line 33, in <module>
    from cgi import parse_qsl
ImportError: cannot import name 'parse_qsl' from 'cgi' (/home/tyoc213/miniconda3/envs/diccionario/lib/python3.8/cgi.py)
>>> from marrow.mailer import Mailer, Message
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tyoc213/miniconda3/envs/diccionario/lib/python3.8/site-packages/marrow/mailer/__init__.py", line 12, in <module>
    from marrow.mailer.message import Message
  File "/home/tyoc213/miniconda3/envs/diccionario/lib/python3.8/site-packages/marrow/mailer/message.py", line 21, in <module>
    from marrow.mailer.address import Address, AddressList, AutoConverter
  File "/home/tyoc213/miniconda3/envs/diccionario/lib/python3.8/site-packages/marrow/mailer/address.py", line 12, in <module>
    from marrow.util.compat import basestring, unicode, unicodestr, native
  File "/home/tyoc213/miniconda3/envs/diccionario/lib/python3.8/site-packages/marrow/util/compat.py", line 33, in <module>
    from cgi import parse_qsl
ImportError: cannot import name 'parse_qsl' from 'cgi' (/home/tyoc213/miniconda3/envs/diccionario/lib/python3.8/cgi.py)

What would be the fastest way to go? for run in python 3.8? Install an older version? (guess not?)

[amcgregor]

@tyoc213 Quick fix while I work on removing marrow.util, isolate that module and patch out the import; being unused, this should have zero impact on marrow.mailer operation. To find the on-disk path to the file to modify pip install e (yup, just the letter e) then run python -me marrow.util.compat. That'll spit out the on-disk path. Edit that file, comment out (or correct) the from cgi line.

[tyoc213]

Indeed

It work now, I wait for the

[GertBurger]

Any news on this?

[ghost]

@GertBurger I have switched to this fork https://github.com/LexMachinaInc/mailer https://github.com/LexMachinaInc/util which contains the "trivial" fix we mere mortals are not worthy to enjoy in the official package. I suggest you do the same unless you want to wait another 4 months for a non-update

[amcgregor]

Curiouser and curiouser. That person made the fork, didn't bother opening a PR. Go figure. Edit: and did it wrong. That's just icing on this cavalcade.

[amcgregor]

@GertBurger Apologies for the fantastic signal:noise ratio on this issue today. Work is progressing on the next branch which is a complete modernization and update to make use of all modern Python 3 features, including type annotation, modern datatypes, and offers the opportunity for elimination of no longer needed legacy. As a singular developer working on a codebase in such wide use, careful attention is being paid towards making the update as non-breaking as possible.

The "hack and slash" fix (comment out one specific line in your copy) can be effective as a short-term solution, slightly longer-term is the bit of helpful pointer that bubbled out of the chatter, a fork that is patched. However, this is also stop-gap, and an approach entirely taken from the wrong direction.

[fiahil]

Hey everyone,

I ran into this issue today. Rather than switching to a fork, you can temporarily fix this error by adding these lines at the top of the file (before importing Mailer):

import sys  # isort:skip

sys.modules["cgi.parse_qsl"] = None

[GertBurger]

OT but it seems that there is also an issue with Python 3.9 as base64.encodestring() has been removed, should be a trivial fix.

I guess one can do a similar monkey patch to add it back...

[amcgregor]

Looks like my employer paid me to unarchive the repository and patch it. The develop branch of marrow.util successfully operates on Python 3.8 for our SMTP + Mailgun use of Mailer with this patch.

If you are utilizing requirements.txt to pin versions, you can pin marrow.util like this:

-e git+https://github.com/marrow/util.git@cbbc386f8411403953186bf78a9dfcd7cffc42fc#egg=marrow.util

---

Aside: I also just tested the import process retrieving references to Mailer, MailgunTransport, and SMTPTransport on Python 3.9.

Marking as closed.

[amcgregor]

@GertBurger The encodestring reference was corrected in 32dde28.

[davfive]

Lifesaver!!! Thanks :)

…

On Thu, Feb 17, 2022 at 1:41 PM Alice Zoë Bevan–McGregor < ***@***.***> wrote: @GertBurger <https://github.com/GertBurger> The encodestring reference was corrected in 32dde28 <32dde28> . — Reply to this email directly, view it on GitHub <#87 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGAYW7XL5XNPG2DINQORC3U3VTQLANCNFSM4KLAS3ZQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Marrin]

This seems to be patched but there is no release on the Python package index, so the original problem is still there when installing marrow.mailer via pip.

[bronikowski]

I just hit that issue today. It's still buggy on PyPI.