Skip to content
This repository has been archived by the owner on Oct 21, 2019. It is now read-only.

Added Layer creation section #24

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 42 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,48 @@ You can find a more complex example in https://github.com/RemotePixel/remotepixe


## Create a Lambda layer
`TODO`
For this example, I will use a pyhton Lambda as the use case scenario.

According to AWS [docs](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html), a Lambda layer must have the following format:

```
package.zip
├── bin # executables
├── lib # libraries
├── lib64 # libraries 64-bit
├── python # python packages
└── share # shared libraries
```

### Creating the package

To create a layer, we just need to package all the thinfs we need into a `package.zip` with the format described above. To do that, we can use a slightly modified version of the preivous script:

```bash
docker run --name lambda -itd remotepixel/amazonlinux-gdal:3.0.1 /bin/bash
# This is just an example, installing gdal bindings for python
docker exec -it lambda bash -c 'mkdir python'
docker exec -it lambda bash -c 'pip install gdal==3.0.1 -no-binary :all: -t python -U'

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't work for py3.6 or py3.7 lambda layers, they need libs to be packaged into

  • python/lib/python${PY_VERSION}/site-packages

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...they also output the archived python libs into the correct path for py3.6 and py3.7 shared libs for a lambda layer (it's not /opt/python any more, that only works for py2.7).

Are you 100% sure about that? I am currently using a layer, created like I described, using python 3.7. According to AWS' docs, it appears that using only python/ works.

The following part is copied form the above link:

Python – python, python/lib/python3.7/site-packages (site directories)
Example Pillow

pillow.zip
│ python/PIL
└ python/Pillow-5.3.0.dist-info

The package.zip that I generated has the following format, for python packages:

package.zip
│ python/osgeo
└ python/other_package
 etc...

And I can import the packages normally in python. Like so:

from osgeo import gdal
...

Copy link

@dazza-codes dazza-codes Aug 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/shrug - didn't work for me but updates in my MR script are working OK and AFAICT the full path to the python packages in a layer becomes:

  • /opt/python/lib/python3.7/site-packages/**
  • not /opt/python/{packages_directly_here}

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It probably does become /opt/python/lib/python3.7/site-packages/**. I was only pointing out that the path, inside the package.zip, can be /python/your_lib. AWS probably does some post processing after unzipping the layer package.

docker exec -it lambda bash -c 'zip -r9 /tmp/package.zip python'
docker exec -it lambda bash -c 'zip -r9 --symlinks /tmp/package.zip lib/*.so*'
docker exec -it lambda bash -c 'zip -r9 --symlinks /tmp/package.zip lib64/*.so*'
docker exec -it lambda bash -c 'zip -r9 --symlinks /tmp/package.zip bin'
docker exec -it lambda bash -c 'zip -r9 /tmp/package.zip share'
docker cp lambda:/tmp/package.zip package.zip
docker stop lambda
docker rm lambda
```
Copy link

@dazza-codes dazza-codes Aug 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, see make rules and scripts in #22


The above script should result in a `package.zip` that can be uploaded as a Lambda layer. If you are using Chalice as the deployment tool, you do not need, per this example, to include `gdal` in the requirements file. It will be already available for use. But, you should have it installed locally, or Chalice won't deploy.

### Environment variables

You should, also, include some environment variables to your function. Mainly:

```bash
PROJ_LIB="/opt/share/proj"
GDAL_DATA="/opt/share/gdal"
```

## Package architecture and AWS Lambda config
:warning: AWS Lambda will need `GDAL_DATA` to be set to `/var/task/share/gdal` to be able to work :warning:
Expand Down