Skip to content

Commit 60ececa

Browse files
author
rudolfix
committed
docker and docker composer deployment files, deployment documentation
1 parent a661699 commit 60ececa

File tree

8 files changed

+181
-17
lines changed

8 files changed

+181
-17
lines changed

‎deployment/README.md

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# Deployment information
2+
3+
Service is deployed using Docker. Dockerfiles and docker compose file are provided. Currently following services and resources are defined
4+
5+
1. api service containing external and internal apis done with Flask.
6+
2. mysql server
7+
3. volumes to persist database files and user profiles/cache
8+
4. r/o volume with cache and music graph my sql database dump
9+
10+
##Build process
11+
1. build intermediary python image (make subsequent deployments od flasks apps much quicker)
12+
```javascript
13+
docker build -f sleep_server\Dockerfile_Python -t sleep_python sleep_server
14+
```
15+
2. use docker-compose to build all images as defined in docker-compose.yml
16+
```javascript
17+
docker-compose --verbose build
18+
```
19+
3. we use standard mysql configuration (todo: use custom config to set sql-mode=TRADITIONAL,ANSI_QUOTES)
20+
4. docker-compose up -d to run all services. there is initialization process on first run, see later.
21+
22+
##Local deployment
23+
You may deploy on your local docker host to test the deployment. There is a callback defined in spotify app for that purpose
24+
http://dev.docker.wakeupapp.com/admin/login_completed
25+
host file:
26+
127.0.0.1 dev.wakeupapp.com wakeupapp.com
27+
192.168.99.100 dev.docker.wakeupapp.com
28+
29+
##Remote deployment
30+
31+
1. uses ssh connection to server
32+
2. port 2376 must be opened
33+
```javascript
34+
(aws) docker-machine create --driver generic --generic-ip-address=wakeupapp.dev.army --generic-ssh-key=sleep_server.pem --generic-ssh-user=ubuntu awssleepo
35+
(droplet)docker-machine create --driver generic --generic-ip-address=sleepapi.dev.army --generic-ssh-key=droplet_sleep.rsa --generic-ssh-user=root sleepdroplet
36+
```javascript
37+
38+
Minimum machine specs: 1GB ram, 10 GB SSD with reasonable speed (AWS t2 crap does not work, credits immediately expire)
39+
40+
##Monitoring
41+
api container produces logs in /var/log/sleep_server. get there by docker exec -i -t src_api_1 /bin/bash and
42+
tail -f /var/log/sleep_server/server.log
43+
##Procedures
44+
###rebuild cache and database dump
45+
there is read only volume "data" containing initial cache and music graph database.
46+
### docker run script in api container (run_all.sh)
47+
1. wait for database to go up
48+
2. if database is empty, restore a dump from "data" volume
49+
3. if cache is empty, restore from "data" volume
50+
4. run server api
51+
5. run frontend api
52+
53+
ACHTUNG: restoring db dump will take a lot of time (this is how mysql works)
54+
If the initial script fails, (disk full etc) there is a huge chance that it will not recover when container restarts. Kill all volumes and restart container
55+
56+
### Recreating cache
57+
### Updating db schema
58+
### Periodically getting similar artists and infering genres

‎deployment/mysql-prod.conf

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# For advice on how to change settings please see
2+
# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html
3+
4+
[mysqld]
5+
#
6+
# Remove leading # and set to the amount of RAM for the most important data
7+
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
8+
innodb_buffer_pool_size = 128M
9+
#
10+
# Remove leading # to turn on a very important data integrity option: logging
11+
# changes to the binary log between backups.
12+
# log_bin
13+
#
14+
# Remove leading # to set options mainly useful for reporting servers.
15+
# The server defaults are faster for transactions and fast SELECTs.
16+
# Adjust sizes as needed, experiment to find the optimal values.
17+
# join_buffer_size = 128M
18+
# sort_buffer_size = 2M
19+
# read_rnd_buffer_size = 2M
20+
skip-host-cache
21+
skip-name-resolve
22+
datadir=/var/lib/mysql
23+
socket=/var/lib/mysql/mysql.sock
24+
secure-file-priv=/var/lib/mysql-files
25+
user=mysql
26+
27+
# Disabling symbolic-links is recommended to prevent assorted security risks
28+
symbolic-links=0
29+
30+
#set strict mode
31+
sql-mode=TRADITIONAL,ANSI_QUOTES
32+
33+
log-error=/var/log/mysqld.log
34+
pid-file=/var/run/mysqld/mysqld.pid

‎deployment/mysql_prod.env

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
MYSQL_ROOT_PASSWORD=jksahd093-AAna
2+
MYSQL_DATABASE=music_graph
3+
MYSQL_USER=prod
4+
MYSQL_PASSWORD=prod

‎docker-compose.yml

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
version: '2'
2+
services:
3+
mysql:
4+
restart: always
5+
image: mysql/mysql-server:latest
6+
volumes:
7+
- dbdata:/var/lib/mysql
8+
expose:
9+
- "3306"
10+
env_file: ./deployment/mysql_prod.env
11+
12+
api:
13+
restart: always
14+
build: ./sleep_server
15+
# - "172.31.18.128:80:80"
16+
ports:
17+
- "80:80"
18+
links:
19+
- mysql
20+
volumes_from:
21+
- data
22+
volumes:
23+
- userdata:/var/local/sleep_server/user_storage/
24+
environment:
25+
- PYTHONIOENCODING=utf-8
26+
#networks:
27+
# - front-tier
28+
# - back-tier
29+
command: /usr/local/sleep_server/run_all.sh
30+
# command: sleep 1d
31+
32+
data:
33+
build: ./data
34+
volumes:
35+
- /var/local/sleep_server/data/
36+
command: "true"
37+
38+
volumes:
39+
dbdata:
40+
driver: local
41+
userdata:
42+
driver: local
43+
44+
#networks:
45+
# front-tier:
46+
# driver: bridge
47+
# back-tier:
48+
# driver: bridge

‎run_uwsgi_server.sh

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
#!/bin/bash
2+
uwsgi --http-socket :5001 --enable-threads --wsgi-file /vagrant/sleep_server/server/run.py --callable app --py-autoreload=1 --pythonpath /vagrant/sleep_server/server

‎sleep_server/.dockerignore

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
Dockerfile
2+
server/migrations
3+
server/.*
4+
server/old
5+
tests/test_accounts
6+
.ipynb_checkpoints/
7+
8+
# Byte-compiled / optimized / DLL files
9+
**/__pycache__/
10+
**/*.pyc
11+
**/*.pyo
12+
**/*.pyd
13+
**/*$py.class
14+
**/*.ipynb

‎sleep_server/Dockerfile

Lines changed: 11 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,12 @@
1-
#this will be a proper docker file later
2-
#instance DNS 52.28.90.131 wakeupapp.dev.army
3-
#ssh -i sleep_server.pem [email protected]
4-
#install uwsgi
5-
#set UWSGI_PROFILE_OVERRIDE=ssl=true;routing=true;pcre=true
6-
#pip3 install uwsgi
7-
#letsencrypt used for ssl cert https://letsencrypt.readthedocs.org/en/latest/using.html#installation
1+
FROM sleep_python
82

9-
#create dirs
10-
##mkdir /home/ubuntu/wakeupapp
11-
##chmod 777 /home/ubuntu/wakeupapp
12-
##mkdir /home/ubuntu/wakeupapp/user_storage
13-
##chmod 777 /home/ubuntu/wakeupapp/user_storage
14-
#copy files
15-
##scp -i sleep_server.pem ../src/run_uwsgi_prod.sh [email protected]:/home/ubuntu/wakeupapp
16-
##scp -r -i sleep_server.pem ../src/sleep_server/* [email protected]:/home/ubuntu/wakeupapp/sleep_server/
17-
##scp -i sleep_server.pem ../src/requirements.txt [email protected]:/home/ubuntu/wakeupapp
18-
#execute vagrant provision file
3+
LABEL description='image with sleep server /api and /server endpoints'
4+
5+
ENV WAKEUPP_APP_CONFIG_FILE config.production.py
6+
RUN mkdir /var/log/sleep_server/
7+
RUN chmod 757 /var/log/sleep_server/
8+
RUN mkdir /var/local/sleep_server/
9+
RUN mkdir /var/local/sleep_server/data/
10+
RUN mkdir /var/local/sleep_server/user_storage/
11+
ADD . /usr/local/sleep_server/
12+
RUN chmod +x /usr/local/sleep_server/*.sh

‎sleep_server/Dockerfile_Python

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
FROM ubuntu
2+
3+
LABEL description='image python modules and necessary libraries'
4+
5+
ENV UWSGI_PROFILE_OVERRIDE ssl=true;routing=true;pcre=true
6+
RUN mkdir /usr/local/sleep_server/
7+
ADD requirements.txt /usr/local/sleep_server/
8+
9+
RUN apt-get update && apt-get install -y build-essential python3-dev python3-pip python3-scipy git libpcre3 libpcre3-dev libmysqlclient-dev rabbitmq-server curl libssl-dev screen net-tools iputils-ping vim mysql-client
10+
RUN pip3 install -r /usr/local/sleep_server/requirements.txt

0 commit comments

Comments
 (0)