Dependency issues

[valpesendorfer]

Hi @vincentsarago, thanks for your work on this (and other things I use).

I'm trying to install a module for use alongside the lambda layer, following the steps outlined in the respective README section.

I've ran into some issues with dependencies that are available in the docker image (so no issues during build or package) but they are missing in the actual lambda function.

Here's an specific example:

The module I'm installing (replacing mercantile in your example) depends among other things on pyyaml. When running pip install I get the corresponding log message:

Requirement already satisfied: pyyaml in /var/lang/lib/python3.7/site-packages ...

and everything works fine locally. But the contents of /var/lang/lib/python3.7/site-packages don't make it into the zip, and they are not available in the lambda function which includes the layer.

Of course I could just try to find and install all dependencies into /var/task so they get zipped up. But that seems to be a sub-optimal solution. Any ideas? Apologies if I'm missing something obvious.

Thanks!

[vincentsarago]

which layer/docker image are you trying to use @valpesendorfer ?

[valpesendorfer]

Following the example line by line. So the layer is

remotepixel/amazonlinux:gdal3.0-py3.7-geo

arn:aws:lambda:eu-central-1:524387336408:layer:gdal30-py37-geo:7

[vincentsarago]

This is unlucky but lambCI base image provide pyyaml but I think just because of user dependencies. To fix this I'll need to compare what modules are actually in lambda and what are those in lambci/python3.7-build.

Temporary fix is to install pyyaml before you try to install your module.

FROM remotepixel/amazonlinux:gdal3.0-py3.7-geo

# Basically we don't want to replicated existant modules found in the layer ($PYTHONPATH)
# So we use the $PYTHONUSERBASE trick to set the output directory
ENV PYTHONUSERBASE=/var/task

RUN pip install pyyaml --user --ignore-installed
....

[valpesendorfer]

Yeah, I was afraid this will be the only option for now. Means I need to dig out all missing dependencies and install them manually.

If it’s of any help, I’ve previously printed the elements in /var/lang/lib/python3.7 to CloudWatch:

['cProfile.py', 'nturl2path.py', 'tempfile.py', 'sre_parse.py', 'ctypes', 'contextvars.py', 'operator.py', 'profile.py', 'pstats.py', '__phello__.foo.py', 'fnmatch.py', 'symbol.py', 'sndhdr.py', 'numbers.py', 'poplib.py', 'dataclasses.py', 'keyword.py', 'ftplib.py', 'imp.py', 'ipaddress.py', 'compileall.py', '_dummy_thread.py', 'fileinput.py', 'LICENSE.txt', 'getopt.py', '_bootlocale.py', 'symtable.py', 'config-3.7m-x86_64-linux-gnu', 'fractions.py', 'locale.py', 'sched.py', 'timeit.py', 'sunau.py', 'random.py', 'stringprep.py', 'shelve.py', 'bdb.py', 'curses', 'code.py', 'plistlib.py', 'copyreg.py', 'string.py', 'shlex.py', 'json', 'uu.py', 'gzip.py', '__future__.py', 'test', 're.py', 'codeop.py', 'tkinter', 'pdb.py', 'smtplib.py', 'email', 'pathlib.py', 'pkgutil.py', 'typing.py', 'configparser.py', 'platform.py', 'xmlrpc', 'functools.py', 'socket.py', 'getpass.py', 'py_compile.py', 'reprlib.py', 'concurrent', 'difflib.py', '_compression.py', 'zipapp.py', 'struct.py', 'tabnanny.py', 'pydoc.py', 'nntplib.py', 'pickletools.py', 'colorsys.py', '_pydecimal.py', 'abc.py', 'posixpath.py', 'imaplib.py', 'glob.py', 'netrc.py', 'filecmp.py', 'ensurepip', 'collections', 'tarfile.py', 'codecs.py', 'pyclbr.py', '_pyio.py', '_osx_support.py', 'turtledemo', 'queue.py', 'uuid.py', 'distutils', 'copy.py', 'chunk.py', 'asyncio', 'pty.py', 'opcode.py', 'xml', 'telnetlib.py', 'gettext.py', 'optparse.py', '_weakrefset.py', 'trace.py', 'tracemalloc.py', 'contextlib.py', 'dbm', 'ssl.py', 'mailcap.py', 'imghdr.py', 'pipes.py', 'logging', 'unittest', 'base64.py', 'shutil.py', 'heapq.py', 'site-packages', 'webbrowser.py', 'cgitb.py', 'hmac.py', '_sitebuiltins.py', 'crypt.py', 'rlcompleter.py', 'argparse.py', 'os.py', 'traceback.py', '_markupbase.py', 'sqlite3', '_strptime.py', 'binhex.py', 'pydoc_data', 'cmd.py', 'urllib', 'multiprocessing', '_collections_abc.py', 'tokenize.py', 'antigravity.py', 'io.py', 'turtle.py', '__pycache__', 'xdrlib.py', 'ntpath.py', 'wsgiref', 'csv.py', 'weakref.py', 'http', 'sysconfig.py', 'dis.py', 'mailbox.py', 'bz2.py', 'linecache.py', 'smtpd.py', 'ast.py', 'types.py', 'decimal.py', '_py_abc.py', 'encodings', '_compat_pickle.py', 'asynchat.py', 'tty.py', 'cgi.py', 'wave.py', 'stat.py', 'warnings.py', 'secrets.py', 'modulefinder.py', 'html', 'lib2to3', 'signal.py', 'formatter.py', 'inspect.py', 'statistics.py', 'pprint.py', 'textwrap.py', 'threading.py', 'sre_constants.py', 'aifc.py', 'datetime.py', 'lib-dynload', 'subprocess.py', '_sysconfigdata_m_linux_x86_64-linux-gnu.py', 'macpath.py', 'importlib', 'genericpath.py', 'calendar.py', '_threading_local.py', 'dummy_threading.py', 'socketserver.py', 'idlelib', 'selectors.py', 'token.py', 'lzma.py', 'enum.py', 'this.py', 'zipfile.py', 'hashlib.py', 'bisect.py', 'pickle.py', 'mimetypes.py', 'asyncore.py', 'quopri.py', 'venv', 'runpy.py', 'sre_compile.py', 'doctest.py', 'site.py']

Thanks again!

[vincentsarago]

the full list of pacakge in AWS Lambda function can be found over https://gist.github.com/gene1wood/4a052f39490fae00e0c3