ten-lists

This project provides both a CLI (using click) and a webapp (using flask), which generate a playlist of 10 Bible Chapters (represented by 10 mp3 files) to be listened to on any given day x, according to Professor Grant Horner's Bible-Reading System. The audio Bible is as downloaded from the Faith Comes by Hearing® website.

You can see the webapp in action here

• ten-lists
  • Developer setup 💻
    • Requirements
      • Essential
      • Extra
    • Installation
      • You need the MP3 files
      • Concerning the CLI tool
    • Tests
  • Deployment
  • Notes
  • TODO
  • Credits

Developer setup 💻

Requirements

Essential

Start by ensuring that you have Docker and Docker Compose:

# check that you have docker on your machine
docker -v
# check that you have docker-compose on your machine
docker-compose -v

If you don't have Docker and Docker Compose, then click the respective links above for installation instructions for your platform.

Extra

Good to have if you'd like to hack on the project. Not required if you just wanna run it, in which case Docker and Docker Compose will suffice.

• A Python 3.8 virtual environment. You can use any tool of your choice to manage multiple Python versions on your machine.
  • Activate your python virtual environment and pip install --upgrade pip
  • Install dependencies: pip install -r requirements.txt.
  • Setup pre-commit by running pre-commit install followed by pre-commit install --hook-type commit-msg. Optionally run pre-commit run --all-files to make sure your pre-commit setup is okay.
• Node.js v14
  • Install the Node.js dependencies via npm install

Installation

Upon cloning this repository (or forking + cloning your fork), navigate to the cloned project directory.

Then create the required environment variables file (.env) by making a copy of the provided sample file .env.sample and renaming it to .dev.env:

cp -v .env.sample .env

NOTE: If you're not using a docker-based deployment approach, then, for production, the file should be .prod.env, for staging .stage.dev

You should be able to run the project without updating anything. If you wanna update the mail settings (which you probably don't really need immediately) -- you could use a service like mailtrap.io for development. If you choose Mailtrap, then the variables to update are TENLISTS_EMAIL_USER_DEV and TENLISTS_EMAIL_PWD_DEV. The other settings are for use in production (SERVER_NAME and SENTRY_DSN).

Build the images and spin up the containers:

COMPOSE_PROJECT_NAME=tenlists docker-compose up -d --build

When you run the above for the first time, it may take a while, depending on your internet connection speed.

Once the build is complete, you can check the logs for the web container via

docker-compose logs web

If everything is ok, you should see something like this:

tenlists-web-1  | tenlists :: ready
tenlists-web-1  |  * Serving Flask app "tenlists.webapp.ten_lists" (lazy loading)
tenlists-web-1  |  * Environment: development
tenlists-web-1  |  * Debug mode: on
tenlists-web-1  |  * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
tenlists-web-1  |  * Restarting with stat
tenlists-web-1  |  * Debugger is active!
tenlists-web-1  |  * Debugger PIN: 719-114-836

You can also check the logs for the node container via

docker-compose logs node

If all's well, you should see something like this:

tenlists-node-1  |
tenlists-node-1  | > [email protected] start /usr/src/app
tenlists-node-1  | > grunt
tenlists-node-1  |
tenlists-node-1  | Running "clean:dist" (clean) task
tenlists-node-1  | >> 4 paths cleaned.
tenlists-node-1  |
tenlists-node-1  | Running "newer:copy" (newer) task
tenlists-node-1  |
tenlists-node-1  | Running "newer:copy:fa" (newer) task
tenlists-node-1  |
tenlists-node-1  | Running "copy:fa" (copy) task
tenlists-node-1  | Copied 8 files
tenlists-node-1  |
tenlists-node-1  | Running "newer-postrun:copy:fa:1:/usr/src/app/node_modules/grunt-newer/.cache" (newer-postrun) task
tenlists-node-1  |
tenlists-node-1  | Running "newer:copy:bootstrap" (newer) task
tenlists-node-1  |
tenlists-node-1  | Running "copy:bootstrap" (copy) task
tenlists-node-1  | Copied 10 files
tenlists-node-1  |
tenlists-node-1  | Running "newer-postrun:copy:bootstrap:2:/usr/src/app/node_modules/grunt-newer/.cache" (newer-postrun) task
tenlists-node-1  |
tenlists-node-1  | Running "newer:copy:jquery" (newer) task
tenlists-node-1  |
tenlists-node-1  | Running "copy:jquery" (copy) task
tenlists-node-1  | Copied 4 files
tenlists-node-1  |
tenlists-node-1  | Running "newer-postrun:copy:jquery:3:/usr/src/app/node_modules/grunt-newer/.cache" (newer-postrun) task
tenlists-node-1  |
tenlists-node-1  | Running "newer:copy:moment" (newer) task
tenlists-node-1  |
tenlists-node-1  | Running "copy:moment" (copy) task
tenlists-node-1  | Copied 5 files
tenlists-node-1  |
tenlists-node-1  | Running "newer-postrun:copy:moment:4:/usr/src/app/node_modules/grunt-newer/.cache" (newer-postrun) task
tenlists-node-1  |
tenlists-node-1  | Running "newer:cssmin" (newer) task
tenlists-node-1  |
tenlists-node-1  | Running "newer:cssmin:dist" (newer) task
tenlists-node-1  | No newer files to process.
tenlists-node-1  |
tenlists-node-1  | Running "newer:uglify" (newer) task
tenlists-node-1  |
tenlists-node-1  | Running "newer:uglify:dist" (newer) task
tenlists-node-1  | No newer files to process.
tenlists-node-1  |
tenlists-node-1  | Running "browserSync:dev" (browserSync) task
tenlists-node-1  | [Browsersync] Proxying: http://web:5000
tenlists-node-1  | [Browsersync] Access URLs:
tenlists-node-1  |  -----------------------------------
tenlists-node-1  |        Local: http://localhost:3000
tenlists-node-1  |     External: http://151.26.0.3:3000
tenlists-node-1  |  -----------------------------------
tenlists-node-1  |           UI: http://localhost:3001
tenlists-node-1  |  UI External: http://localhost:3001
tenlists-node-1  |  -----------------------------------
tenlists-node-1  | [Browsersync] Watching files...
tenlists-node-1  |
tenlists-node-1  | Running "watch" task
tenlists-node-1  | Waiting...
tenlists-node-1  | [Browsersync] Reloading Browsers...
tenlists-node-1  | [Browsersync] Reloading Browsers...

You can access the dev server at http://127.0.0.1:5000.

You need the MP3 files

Now, at this stage, you probably won't get much from the now running app, because you actually need the MP3 files to work with! So, especially for the CLI tool, you have to download the Audio Bible from the Faith Comes by Hearing® website. The 2001 ESV dramatized Bible (size is over 2Gb) formed the basis for this project's code, including the expected filenames. If the file naming convention has changed, then there's a chance everything will break. Anyway, I hope that won't be the case ...

For the web app, we previously would create a directory ENGESVC2DA in tenlists/webapp/ten_lists/static/ and place your downloaded MP3 files in there. See this line in tenlists/webapp/ten_lists/main/routes.py.

However, the code has been rewritten in such a way as to use a cloud service (s3, MinIO, Backblaze, Cloudinary, etc.) for the files. This should hopefuly simplify both on-boarding and deployment, especially that at the time of writing this the webapp is deployed using Dokku. You'll therefore have to upload these files to your preferred cloud provider, and set the environment variable TENLISTS_MP3_CLOUD_STORAGE_BASE_URL (see that .env.sample file for details).

Concerning the CLI tool

This project started off initially as a CLI tool to generate these mp3 files on my computer and play them on a USB stick or copy them over to my phone. However, the scope later changed to have a web application that can allow me to listen from anywhere. So lately the focus has been more on the webapp than the CLI.

Notwithstanding, if you wanna use the CLI, you need to firstly ensure that you have a folder containing the required MP3 files, as described above. If you don't specify the folder when running the CLI tool, it'll assumes that there's a directory ENGESVC2DA in the project root. Once you've sorted this out, then you can run the CLI tool with the --help option so that you see how to use it:

If you have a setup a python virtual environment and installed the python dependencies as described earlier in this README

python tenlists/cli/__main__.py --help

So, for example 1, if I have my Bible folder ENGESVC2DA in the project root, and I run python tenlists/cli/__main__.py -d 87, I'll see the following in the console

❯ python tenlists/cli/__main__.py -d 87
 _           _                __
|_)o|_ | _  |_)| _.  |o __|_ /__ _ ._  _ .__._|_ _ ._
|_)||_)|(/_ |  |(_|\/||_> |_ \_|(/_| |(/_|(_| |_(_)|
                   /

Welcome to the Bible Playlist Generator

Creating a playlist for day 87 ...
✓ playlist for day 87 successfully created.
✓ Copying of the Bible Chapters into the day087 directory was successful.
✓ ID3 tag info for the generated files in this directory has been updated.
✓ The files have been renamed in a sequential order.

-----Soli Deo Gloria-----

What has happended here? Well,

• I ran a command to generate a playlist for day 87 of the Bible Reading Plan
• The CLI tool has generated an M3U Playlist, day087.m3u in the project root
• The CLI tool has created a folder, day087, in the project root -- containing the 10 MP3 files in the above playlist

Example 2; if I want to use tenlists/webapp/ten_lists/static/ENGESVC2DA/ as my folder, then I'll go ahead and

python tenlists/cli/__main__.py -d 87 -f tenlists/webapp/ten_lists/static/ENGESVC2DA/

And I'll see the same results!

If you're using Docker

docker-compose exec web python tenlists/cli/__main__.py --help

Everything remains the same as the case where you're not using Docker.

Tests

docker-compose exec web pytest

WARNING: Running this test will delete the directory `tenlists/webapp/ten_lists/static/ENGESVC2DA/, so please ensure that you have a copy of your MP3 files somewhere. This shouldn't be the case though, and I intend to fix it soon, as you can see it's top on the TODO list. Feel free to submit a PR if this hasn't been fixed!

Deployment

You can deploy this project using your preferred choice of deployment (Docker, Heroku, Linux Server, etc.). However, I have set it up to be ready "out of the box" for deployment to a VPS using Dokku.

If you're gonna use Dokku, feel free to use @engineervix/pre-dokku-server-setup to setup an Ubuntu Server on your VPS prior to installation of Dokku. You can have a look at this gist for more details of how to deploy an application to Dokku. Other excellent resources:

• How to deploy Django project to Dokku
• Setting up Dokku with DigitalOcean and Namecheap (GitHub gist)
• Deploying an app with Dokku
• Dokku Docs: Process Management
• Dokku Docs: Zero Downtime Deploys
• Dokku with Let's Encrypt behind Cloudflare
• Cloudflare certificates + Dokku
• Securing Dokku with Let's Encrypt TLS Certificates

Notes

• This project will be most useful to you if you use Professor Grant Horner's Bible-Reading System as the basis for your Bible Reading Plan.
• The Audio Bible version used is the 2001 ESV dramatized Bible (complete), as freely downloaded from http://www.bible.is/audiodownloader. (The size is over 2Gb)

TODO

• Fix test to avoid overwriting tenlists/webapp/ten_lists/static/ENGESVC2DA/ and deleting its contents
• Use a CDN for the media files
• write a Dokku deploy script
• Use Invoke to encapsulate some tasks. For instance, docker-compose exec web python tenlists/cli/__main__.py --help is too long to type!
• Address #1. configparser might come in handy here.
• Package this project. This is a must read.
• Improve Code Quality
• Rather than using plain text files, find a better way of storing the Bible Chapters (JSON file, SQLite database, TinyDB, etc)
• Create a GUI frontend or web service to cater for non-tech users in order to not only cater for non-tech users but also to listen on-the-go. See the webapp directory for the source code. Also see the package.json file.

Credits

• og:image: https://unsplash.com/photos/TNlHf4m4gpI
• Bible Icon: https://www.pngrepo.com/svg/235374/bible