Skip to content

Commit 85279c1

Browse files
kshitijrajsharmakshitijrajsharma
authored
Merge pull request #12 from hotosm/feature/docker
Feature : Docker Installation
2 parents 0856582 + a2350fa commit 85279c1

File tree

568 files changed

+33776
-920
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

568 files changed

+33776
-920
lines changed

‎.github/workflows/build.yml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@ jobs:
1616
with:
1717
python-version: 3.8
1818

19+
- name: Remove solaris
20+
run: sudo rm -rf ./docker
21+
1922
- name: Clone Ramp
2023
run: git clone https://github.com/kshitijrajsharma/ramp-code-fAIr.git ramp-code
2124

‎.github/workflows/upload.yml

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ name: Build and Publish
33
on:
44
push:
55
tags:
6-
- 'v*'
6+
- "v*"
77

88
jobs:
99
build:
@@ -18,16 +18,20 @@ jobs:
1818
- name: Checkout code
1919
uses: actions/checkout@v2
2020

21+
- name: Remove solaris
22+
run: |
23+
sudo rm -rf docker
24+
2125
- name: Install dependencies
2226
run: |
2327
python -m pip install --upgrade pip
2428
pip install setuptools wheel twine
2529
2630
- name: Build package
27-
run: python setup.py sdist
31+
run: python setup.py sdist
2832

2933
- name: Publish to PyPI
3034
env:
3135
TWINE_USERNAME: __token__
3236
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
33-
run: twine upload dist/*
37+
run: twine upload dist/*

‎Dockerfile

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
FROM tensorflow/tensorflow:2.9.2-gpu-jupyter
2+
3+
RUN apt-get update && apt-get install -y python3-opencv
4+
RUN add-apt-repository ppa:ubuntugis/ppa && apt-get update
5+
RUN apt-get update
6+
RUN apt-get install -y gdal-bin
7+
RUN apt-get install -y libgdal-dev
8+
ENV CPLUS_INCLUDE_PATH=/usr/include/gdal
9+
ENV C_INCLUDE_PATH=/usr/include/gdal
10+
11+
#install numpy before gdal
12+
RUN pip install numpy==1.23.5
13+
14+
# pip install dependencies.
15+
RUN pip install --global-option=build_ext --global-option="-I/usr/include/gdal" GDAL==`gdal-config --version`
16+
17+
COPY docker/ramp/docker-requirements.txt docker-requirements.txt
18+
RUN pip install -r docker-requirements.txt
19+
20+
# pip install solaris -- try with tmp-free build
21+
# COPY docker/ramp/solaris /tmp/solaris
22+
COPY docker/solaris/solaris /tmp/solaris/solaris
23+
COPY docker/solaris/requirements.txt /tmp/solaris/requirements.txt
24+
COPY docker/solaris/setup.py /tmp/solaris/setup.py
25+
26+
RUN pip install /tmp/solaris --use-feature=in-tree-build
27+
28+
RUN pip install scikit-fmm --use-feature=in-tree-build
29+
30+
RUN pip install setuptools --upgrade
31+
32+
ENV RAMP_HOME=/tf
33+
34+
# Install the package in development mode
35+
COPY setup.py ./setup.py
36+
COPY hot_fair_utilities ./hot_fair_utilities
37+
RUN pip install -e .
38+
39+
# install ramp-fair
40+
RUN pip install ramp-fair mercantile pandas==1.5.3
41+
42+
## Copy Sample data
43+
COPY ramp-data ./ramp-data

‎README.md

Lines changed: 22 additions & 77 deletions
Original file line numberDiff line numberDiff line change
@@ -1,94 +1,39 @@
11
# hot_fair_utilities ( Utilities for AI Assisted Mapping fAIr )
2-
Initially lib was developed during Open AI Challenge with [Omdeena](https://omdena.com/). Learn more about challenge from [here](https://www.hotosm.org/tech-blog/hot-tech-talk-open-ai-challenge/)
32

4-
## `hot_fair_utilities` Installation
5-
6-
hot_fair_utilities is collection of utilities which contains core logic for model data prepration , training and postprocessing . It can support multiple models , Currently ramp is supported.
7-
8-
1. To get started clone this repo first :
9-
```
10-
git clone https://github.com/hotosm/fAIr-utilities.git
11-
```
12-
2. Setup your virtualenv with ```python 3.8``` ( Ramp is tested with python 3.8 )
13-
14-
3. Install tensorflow ```2.9.2``` from [here] (https://www.tensorflow.org/install/pip) According to your os
15-
16-
#### Setup Ramp :
17-
18-
4. Copy your basemodel : Basemodel is derived from ramp basemodel
19-
```
20-
git clone https://github.com/radiantearth/model_ramp_baseline.git
21-
```
22-
23-
5. Clone ramp working dir
24-
25-
```
26-
git clone https://github.com/kshitijrajsharma/ramp-code-fAIr.git ramp-code
27-
```
28-
29-
6. Copy base model to ramp-code
30-
```
31-
cp -r model_ramp_baseline/data/input/checkpoint.tf ramp-code/ramp/checkpoint.tf
32-
```
3+
Initially lib was developed during Open AI Challenge with [Omdeena](https://omdena.com/). Learn more about challenge from [here](https://www.hotosm.org/tech-blog/hot-tech-talk-open-ai-challenge/)
334

34-
7. Install native bindings
35-
- Install Numpy
36-
```
37-
pip install numpy==1.23.5
38-
```
39-
- Install [gdal](https://gdal.org/index.html) .
40-
41-
for eg : on Ubuntu
42-
```
43-
sudo add-apt-repository ppa:ubuntugis/ppa && sudo apt-get update
44-
sudo apt-get install gdal-bin
45-
sudo apt-get install libgdal-dev
46-
export CPLUS_INCLUDE_PATH=/usr/include/gdal
47-
export C_INCLUDE_PATH=/usr/include/gdal
48-
pip install --global-option=build_ext --global-option="-I/usr/include/gdal" GDAL==`gdal-config --version`
49-
```
50-
on conda :
51-
```
52-
conda install -c conda-forge gdal
53-
```
54-
- Install rasterio
5+
## `hot_fair_utilities` Installation
556

56-
for eg: on ubuntu :
57-
```
58-
sudo apt-get install -y python3-rasterio
59-
```
60-
on conda :
61-
```
62-
conda install -c conda-forge rasterio
63-
```
7+
Installing all libraries could be pain so we suggest you to use docker , If you like to do it bare , You can follow `.github/build.yml`
648

65-
8. Install ramp requirements
9+
Clone repo
6610

67-
Install necessary requirements for ramp and hot_fair_utilites
68-
69-
```
70-
cd ramp-code && cd colab && make install && cd ../.. && pip install -e .
71-
```
11+
```
12+
git clone https://github.com/hotosm/fAIr-utilities.git
13+
```
7214

15+
Build Docker
7316

17+
```
18+
docker build --tag fairutils .
19+
```
7420

75-
## Conda Virtual Environment
76-
Create from env fle
21+
Run Container with default Jupyter Notebook , Or add `bash` at end to see terminal
7722

7823
```
79-
conda env create -f environment.yml
24+
docker run -it --rm --gpus=all -p 8888:8888 fairutils
8025
```
81-
Create your own
26+
27+
By Default tf is set as Ramp_Home , You can change that attaching your volume to container as tf
8228

8329
```
84-
conda create -n fAIr python=3.8
85-
conda activate fAIr
86-
conda install -c conda-forge gdal
87-
conda install -c conda-forge geopandas
88-
pip install pyogrio rasterio tensorflow
89-
pip install -e hot_fair_utilities
30+
-v /home/hotosm/fAIr-utilities:/tf
9031
```
9132

92-
## Test Installation and workflow
33+
## Test Installation and workflow
34+
35+
You can run `package_test.ipynb` on your notebook from docker to test the installation and workflow with sample data provided , Or open with [collab and connect your runtime locally](https://research.google.com/colaboratory/local-runtimes.html#:~:text=In%20Colab%2C%20click%20the%20%22Connect,connected%20to%20your%20local%20runtime.)
36+
37+
## Get started with development
9338

94-
You can run ```package_test.ipynb``` to your pc to test the installation and workflow with sample data provided
39+
Now you can play with your data , use your own data , use different models for testing and also Help me Improve me !

‎docker/ramp/docker-requirements.txt

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
affine==2.3.1
2+
albumentations==1.0.3
3+
attrs==21.4.0
4+
branca==0.4.2
5+
brotlipy==0.7.0
6+
certifi==2021.10.8
7+
cffi==1.15.0
8+
charset-normalizer==2.0.4
9+
click==8.1.0
10+
click-plugins==1.1.1
11+
cligj==0.7.2
12+
cloudpickle==2.0.0
13+
cryptography==36.0.0
14+
cycler==0.11.0
15+
cytoolz==0.11.2
16+
dask
17+
efficientnet==1.0.0
18+
Fiona==1.8.21
19+
folium==0.12.1.post1
20+
fonttools==4.31.2
21+
fsspec==2022.2.0
22+
GDAL
23+
geojson==2.5.0
24+
geopandas==0.10.2
25+
h5py==3.6.0
26+
idna==3.3
27+
image-classifiers==1.0.0
28+
imagecodecs
29+
imageio==2.16.1
30+
imgaug==0.4.0
31+
Jinja2==3.1.1
32+
joblib==1.1.0
33+
Keras-Applications==1.0.8
34+
kiwisolver==1.4.2
35+
locket==0.2.0
36+
mapclassify==2.4.3
37+
MarkupSafe==2.1.1
38+
matplotlib==3.5.1
39+
munch==2.5.0
40+
munkres==1.1.4
41+
networkx
42+
numpy
43+
opencv-python==4.5.5.64
44+
packaging==21.3
45+
pandas
46+
partd==1.2.0
47+
Pillow==9.0.1
48+
pip==21.2.4
49+
pycosat==0.6.3
50+
pycparser==2.21
51+
pyOpenSSL==21.0.0
52+
pyparsing==3.0.7
53+
pyproj
54+
PySocks==1.7.1
55+
python-dateutil==2.8.2
56+
pytz==2022.1
57+
PyWavelets==1.3.0
58+
PyYAML==6.0
59+
rasterio==1.2.10
60+
requests==2.27.1
61+
Rtree==0.9.7
62+
scikit-image==0.19.2
63+
scikit-learn==1.0.2
64+
scipy
65+
segmentation-models==1.0.1
66+
setuptools==58.0.4
67+
Shapely==1.8.0
68+
six==1.16.0
69+
snuggs==1.4.7
70+
threadpoolctl==3.1.0
71+
tifffile
72+
tinydb==4.7.0
73+
toolz==0.11.2
74+
tqdm==4.62.3
75+
unicodedata2==14.0.0
76+
urllib3==1.26.7
77+
wheel==0.37.1
78+
xyzservices==2022.3.0

‎docker/solaris/.gitattributes

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
*.ipynb filter=nbstripout
2+
3+
*.ipynb diff=ipynb

‎docker/solaris/.github/ISSUE_TEMPLATE/bug_report.md

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
---
2+
name: Bug report
3+
about: Report an error or other suspected bug behavior
4+
title: "[BUG]: (your description here)"
5+
labels: 'Status: Review Needed, Type: bug'
6+
assignees: ''
7+
8+
---
9+
10+
_Thank you for helping us improve `solaris`!_
11+
12+
## Summary of the bug
13+
A clear and concise description of what the bug is.
14+
15+
## Steps to reproduce the bug
16+
Please either paste sample code used to generate the buggy behavior below, or provide step-by-step instructions to reproduce the problem.
17+
18+
```
19+
Paste bug-causing code here
20+
```
21+
22+
Steps to reproduce the behavior:
23+
1. Go to '...'
24+
2. Click on '....'
25+
3. Scroll down to '....'
26+
4. See error
27+
28+
## Buggy behavior and/or error message
29+
Please describe the buggy behavior and/or paste output here.
30+
```
31+
Paste output here
32+
```
33+
34+
## Expected behavior
35+
A clear and concise description of what you expected to happen.
36+
37+
## Screenshots
38+
If applicable, add screenshots to help explain your problem.
39+
40+
## Environment information
41+
- OS:
42+
- `solaris` version:
43+
- python version:
44+
- version of any relevant dependencies (optional - we may ask for this information later if not provided)
45+
46+
## Additional context
47+
Add any other context about the problem here.

‎docker/solaris/.github/ISSUE_TEMPLATE/documentation.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
name: Documentation
3+
about: Request for additional documentation, clarification of existing documentation, or tutorials
4+
title: "[DOCS]"
5+
labels: 'Status: Review Needed, Type: Documentation'
6+
assignees: ''
7+
8+
---
9+
10+
_Thank you for helping to improve `solaris`!_
11+
12+
## Documentation request summary
13+
14+
A short summary of the changes you'd like made to the documentation.
15+
16+
## Task detail and notes
17+
18+
List any additional information, links, or other content here that is needed to make the changes you're requesting.

‎docker/solaris/.github/ISSUE_TEMPLATE/feature_request.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
name: Feature request
3+
about: Suggest an idea for this project
4+
title: "[FEATURE]: (short description)"
5+
labels: 'Status: Review Needed, Type: Enhancement'
6+
assignees: ''
7+
8+
---
9+
10+
**Is your feature request related to a problem? Please describe.**
11+
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
12+
13+
**Describe the solution you'd like**
14+
A clear and concise description of what you want to happen.
15+
16+
**Describe alternatives you've considered**
17+
A clear and concise description of any alternative solutions or features you've considered.
18+
19+
**Additional context**
20+
Add any other context or screenshots about the feature request here.

‎docker/solaris/.github/ISSUE_TEMPLATE/maintenance.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
name: Maintenance
3+
about: Request for maintenance, refactoring, or related tasks
4+
title: "[MAINT]"
5+
labels: 'Status: Review Needed, Type: Maintenance'
6+
assignees: ''
7+
8+
---
9+
10+
## Maintenance request summary
11+
12+
A short summary of the maintenance to perform, including why it needs to be done.
13+
14+
## Task detail and notes
15+
16+
Any additional information:
17+
18+
- What other code will this impact?
19+
20+
- Do new tests need to be written or do existing tests need to be updated?
21+
22+
- Will this impact documentation?

0 commit comments

Comments
 (0)