Consolidate dockerfiles for testing (#519)

* temp remove user cmd

* run tests for this branch

* move workdir

* add a chown

* add debug statements to setup-parallel

* move useradd

* test moving bins to opt/

* more debug msg

* absolute paths

* don't add volumes

* fix docker-entrypoint.sh with DEPLOY_ENV set to test

* print env

* explicitly pass deploy env in ci

* rm env

* back to detached mode

* return to local deploy env

* use tmp dir elsewhere

* delete migration creating default directories

* set default log path to use /var/log

* copy cookies.txt in ci-test-pr workflow

* use new log paths config

* different logpath syntax

* change more paths

* update paths

* attempt to copy schema.rb to /opt/db

* forcibly create /opt/db?

* fix local filepath

* fix mkdir

* fix mkdir

* actually do this in the script

* check schema.rb in

* explicit mkdir /opt/db

* maybe just use /opt?

* mkdir as root??

* add schema.rb

* try new docker test

* comment out path mods

* add chmod on schema.rb

* add debug msg

* chmod a+w

* ok the other file

* only use docker-test.yml

* go back to lograge logger

* use deploy_env override

* log debug mode

* try with rake routes

* fix workflow syntax

* fix workflow syntax

* run docker logs

* run docker logs again

* run docker logs again

* run test no parallel

* add docker inspect

* inspect pender-pender-1

* update syntax

* reset puma dirpath?

* change storage path??

* modify chmod of all files

* try again with parallel tests

* mkdir coverage first

* chmod coverage folder

* add empty coverage dir

* rm otel-collector from docker-test.yml

* sort .gitignore

* remove debug messages from ci test

* run workflow with continue-on-error for tests

* write test ssm to separate file

* use pg_isready and update readme

* create a .env.test in the CI environment

* add build:. as backup for now

* do not use an internal docker network

* add chown flags to copy commands in dockerfile

* try to fix codeclimate ci step

* retry container cp

* docker cp with absolute paths

* change relative path of cc-test-reporter

* run go mod init

* actually cd to go module dir

* run cc-test-reporter without go modules

* check go env

* set proper path of resultset

* put flags at end

* Fixing issues with code coverage report generation and upload to Code Climate (#489)

Fixes the steps that generate and upload the code coverage report to Code Climate.

The main issue was that previously, we ran the tests, prepared the coverage report, and uploaded it all within the Docker container. Now, however, the tests run inside the Docker container, while the coverage report is prepared outside of it, leading to a mismatch in file paths. The solution I implemented was to use sed to replace the paths, changing /app/pender to /home/runner/work/pender. I also needed to copy codeclimate.json to the coverage directory and set the $CC_TEST_REPORTER_ID environment variable.

Reference: CV2-5020.

* remove .env before recreating

* remove references to .env.test

* (1) Using latest Code Climate binary instead of pulling random unexistent branch from GitHub & (2) Fixing error `Error: strconv.Atoi: parsing "2025-02-13T09:46:37-05:00": invalid syntax`.

* address PR comments

---------

Co-authored-by: Caio Almeida <[email protected]>

Author: 2 people

.github/workflows/ci-test-pr.yml

@@ -48,6 +48,16 @@ jobs:
         restore-keys: |
           ${{ runner.os }}-buildx-
 
+    - name: Set up configuration
+      run: |
+        touch .env
+        cp config/database.yml.example config/database.yml
+        cp config/sidekiq.yml.example config/sidekiq.yml
+        cp config/config.yml.example config/config.yml
+        cp config/cookies.txt.example config/cookies.txt
+        bin/get_env_vars.sh
+        chmod -R a+w .
+
     - name: Docker Buildx (build)
       id: build-image
       env:
@@ -63,43 +73,49 @@ jobs:
           --file ./Dockerfile ./
         echo "image=$ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG" >> $GITHUB_OUTPUT
 
-    - name: Set up configuration
-      run: |
-        cp config/database.yml.example config/database.yml
-        cp config/sidekiq.yml.example config/sidekiq.yml
-        cp config/config.yml.example config/config.yml
-        bin/get_env_vars.sh
-
     - name: Run container
       id: run-container
       run: |
-        docker compose -f docker-compose.yml -f docker-test.yml up -d pender
+        docker compose -f docker-test.yml up -d pender
 
     - name: Set up PR Tests
       id: setup-tests
       run: |
-        docker compose exec -T pender test/setup-parallel
+        docker compose -f docker-test.yml exec -T -e DEPLOY_ENV=ci pender test/setup-parallel
 
     - name: Run PR Tests
       id: run-tests
       env:
         TEST_RETRY_COUNT: 5
+      continue-on-error: true
       run: |
-        docker compose exec -e TEST_RETRY_COUNT=$TEST_RETRY_COUNT -T pender bundle exec rake "parallel:test[3]"
-        docker compose exec -e TEST_RETRY_COUNT=$TEST_RETRY_COUNT -T pender bundle exec rake parallel:spec
+        docker compose -f docker-test.yml exec -e TEST_RETRY_COUNT=$TEST_RETRY_COUNT -e DEPLOY_ENV=ci -T pender bundle exec rake "parallel:test[3]"
+        docker compose -f docker-test.yml exec -e TEST_RETRY_COUNT=$TEST_RETRY_COUNT -e DEPLOY_ENV=ci -T pender bundle exec rake parallel:spec
 
-    - name: After PR Tests
-      id: after-tests
+    - name: Upload coverage to CodeClimate
+      id: upload-coverage
       env:
         GIT_SHA: ${{ github.sha }}
         GIT_COMMITED_AT: ${{ github.event.head_commit.timestamp }}
+        CC_TEST_REPORTER_ID: ${{ secrets.CC_TEST_REPORTER_ID }}
       run: |
-        docker compose exec -T pender cat tmp/performance.csv
-        docker compose exec -e GIT_COMMIT_SHA=$GIT_SHA -e GIT_COMMITTED_AT=$GIT_COMMITTED_AT -T pender test/test-coverage
+        docker compose -f docker-test.yml exec -T pender cat tmp/performance.csv
+        docker compose -f docker-test.yml exec -T pender ls -l coverage/
+        docker cp pender-pender-1:/app/pender/tmp/performance.csv performance.csv
+        docker cp pender-pender-1:/app/pender/coverage/.resultset.json test/.resultset.json
+
+        # pulled from test/test-coverage
+        cd test
+        curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 -o cc-test-reporter
+        chmod +x cc-test-reporter
+        sed -i 's/\/app\/pender/\/home\/runner\/work\/pender\/pender/g' .resultset.json # convert container-paths to local-paths
+        GIT_COMMIT_SHA=$(git rev-parse HEAD) GIT_COMMITTED_AT=$(git log -1 --format=%ct) ./cc-test-reporter format-coverage .resultset.json -t simplecov -o codeclimate.json
+        sed -i 's/\/home\/runner\/work\/pender\///g' codeclimate.json
+        cp codeclimate.json ../coverage/
+        ./cc-test-reporter upload-coverage -r $CC_TEST_REPORTER_ID -i codeclimate.json
 
     - name: Reset cache
       id: reset-cache
-      if: ${{ failure() || success() }}
       run: |
         rm -rf /tmp/.buildx-cache
         mv /tmp/.buildx-cache-new /tmp/.buildx-cache

.gitignore

@@ -1,30 +1,29 @@
+*.env
+*.sw*
+*~
+.byebug_history
+.idea
 /.bundle
+/.project
 /db/*.sqlite3
 /db/*.sqlite3-journal
-db/schema.rb
 /log/*.log
 /tmp/cache*
 /tmp/pids
 /tmp/sessions
 /tmp/sockets
-config/database.yml
 config/config.yml
-*.sw*
-*~
+config/cookies.txt
+config/database.yml
+config/sidekiq.yml
+coverage/*
+dump.rdb
+otel-collector-config.yaml
 public/assets
-tmp/restart.txt
-/.project
 public/cache
 public/screenshots
-tmp/restart.txt
-tmp/puma-*
-tmp/profile
 tmp/performance*
-config/sidekiq.yml
-dump.rdb
-config/cookies.txt
-coverage
-.idea
-.byebug_history
-*.env
-otel-collector-config.yaml
+tmp/profile
+tmp/puma-*
+tmp/restart.txt
+tmp/restart.txt

Dockerfile

@@ -1,47 +1,38 @@
 FROM ruby:3.3.3-slim
 LABEL [email protected]
 
-# PROD
-# ENV RAILS_ENV=production \
-#     BUNDLE_DEPLOYMENT=true \
-#     BUNDLE_WITHOUT=development:test
-
-# DEV
-ENV RAILS_ENV=development \
-    SERVER_PORT=3200
-
+ENV APP=pender
 # Set a UTF-8 capabable locale
-ENV LC_ALL=C.UTF-8 \
-    LANG=C.UTF-8 \
-    LANGUAGE=C.UTF-8
+ENV LANG=C.UTF-8
 
-ENV APP=pender
+ENV RAILS_ENV=development \
+    SERVER_PORT=3200 \
+    BUNDLE_DEPLOYMENT="" \
+    BUNDLE_WITHOUT=""
 
+# Build-time variables
 ARG DIRPATH=/app/pender
-ARG BUNDLER_VERSION="2.3.5" 
+ARG BUNDLER_VERSION="2.3.5"
 
 RUN apt-get update && apt-get install -y curl \
     build-essential \
     git \
     libpq-dev --no-install-recommends
 
-# pender user
-RUN mkdir -p ${DIRPATH}
-RUN useradd ${APP} -s /bin/bash -m
-WORKDIR ${DIRPATH}
+RUN useradd "${APP}" --shell /bin/bash --create-home
+WORKDIR "${DIRPATH}"
 
-# install our app
-COPY Gemfile Gemfile.lock ./
-RUN gem install bundler -v ${BUNDLER_VERSION} --no-document \
+COPY Gemfile Gemfile.lock .
+RUN gem install bundler -v "${BUNDLER_VERSION}" --no-document \
     && bundle install --jobs 20 --retry 5
-COPY . ./
+# FIXME: chown flags required for local macos (and likely windows) builds
+COPY --chown=${APP} ./ .
+COPY --chown=${APP} bin/ /opt/bin/
+COPY --chown=${APP} db/schema.rb /opt/db/
+RUN chmod a+w /opt/db/schema.rb
 
 USER ${APP}
 
-# DEV
-# RUN chmod +x ./bin/docker-entrypoint.sh
 EXPOSE 3200
-ENTRYPOINT ["./bin/docker-entrypoint.sh"]
-
-# PROD
 # EXPOSE 8000
+ENTRYPOINT ["/opt/bin/docker-entrypoint.sh"]

README.md

@@ -34,17 +34,33 @@ Besides the specific parsers Pender can parse any link with an oEmbed endpoint o
 
 ## Setup
 
-To run Pender, follow these steps:
+To set Pender up locally:
+
+```
+git clone https://github.com/meedan/pender.git
+cd pender
+find -name '*.example' | while read f; do cp "$f" "${f%%.example}"; done
+```
+
+To run Pender in development mode, follow these steps:
 
 ```
-$ git clone https://github.com/meedan/pender.git
-$ cd pender
-$ find -name '*.example' | while read f; do cp "$f" "${f%%.example}"; done
 $ docker-compose build
 $ docker-compose up --abort-on-container-exit
 ```
 Open http://localhost:3200/api-docs/index.html to access Pender API directly.
 
+To run the full test suite of Pender tests locally the way CI runs them:
+
+```
+bin/get_env_vars.sh
+docker build . -t pender
+docker compose -f docker-test.yml up pender
+docker compose -f docker-test.yml exec pender test/setup-parallel
+docker compose -f docker-test.yml exec pender bundle exec rake "parallel:test[3]"
+docker compose -f docker-test.yml exec pender bundle exec rake "parallel:spec"
+```
+
 ### Setting Cookies for Requests
 
 We send cookies with certain requests that require logged-in users (e.g. Instagram, TikTok).

bin/docker-entrypoint.sh

@@ -3,6 +3,7 @@ if [[ -z ${DEPLOY_ENV+x} || -z ${APP+x} ]]; then
   echo "DEPLOY_ENV and APP must be in the environment. Exiting."
   exit 1
 fi
+echo "running ${APP} in ${DEPLOY_ENV} environment..."
 
 # pender
 if [ "${APP}" = 'pender' ] ; then
@@ -17,7 +18,7 @@ if [ "${APP}" = 'pender' ] ; then
     echo "--- STARTUP COMPLETE ---"
   fi
 
-  DIRPATH=${PWD}/tmp
+  DIRPATH=/tmp
   PUMA="${DIRPATH}/puma-${DEPLOY_ENV}.rb"
   mkdir -p "${DIRPATH}/pids"
   cp config/puma.rb "${PUMA}"
@@ -27,7 +28,7 @@ if [ "${APP}" = 'pender' ] ; then
   port ${SERVER_PORT}
 EOF
 
-  if [ "${DEPLOY_ENV}" = 'local' ] ; then
+  if [ "${DEPLOY_ENV}" = 'local' ] || [ "${RAILS_ENV}" = 'test' ] ; then
     rm -f "${DIRPATH}/pids/server-${DEPLOY_ENV}.pid"
     bundle exec puma -C "${PUMA}"
   else # qa, live etc

bin/get_env_vars.sh

@@ -1,16 +1,21 @@
 #!/bin/bash
 export REGION=eu-west-1
 aws sts get-caller-identity >/dev/null 2>&1
-if (( $? != 0 )); then
+
+if [[ $? != 0 ]]; then
   echo "Error calling AWS get-caller-identity. Do you have valid credentials?"
-else 
+else
   SSM_NAMES=$(aws ssm get-parameters-by-path --region $REGION --path /test/pender/ --recursive --with-decryption --output text --query "Parameters[].[Name]")
-  echo "Getting variables"
+
+  if [ -f ".env" ]; then
+    echo "File .env exists, removing and overwriting."
+    rm .env
+  fi
+  echo "Getting SSM variables"
   for NAME in $SSM_NAMES; do
     echo "."
     VALUE=$(aws ssm get-parameters --region $REGION --with-decryption --name "$NAME" | jq .Parameters[].Value)
     VARNAME=$(basename "$NAME")
-
     echo "$VARNAME=$VALUE" >> .env
   done
 fi

bin/setup

@@ -20,9 +20,8 @@ Dir.chdir APP_ROOT do
   puts "\n== Preparing database =="
   system "bin/rake db:setup"
 
-  puts "\n== Removing old logs and tempfiles =="
+  puts "\n== Removing old logs =="
   system "rm -f log/*"
-  system "rm -rf tmp/cache"
 
   puts "\n== Restarting application server =="
   system "touch tmp/restart.txt"

config/environments/test.rb

@@ -42,10 +42,11 @@
 
   config.allow_concurrency = true
 
-  config.cache_store = :file_store, "#{Rails.root}/tmp/cache#{ENV['TEST_ENV_NUMBER']}"
+  config.cache_store = :file_store, "/tmp/cache#{ENV['TEST_ENV_NUMBER']}"
 
   config.lograge.enabled = true
 
+  config.logger = ActiveSupport::Logger.new(STDOUT)
   config.lograge.logger = ActiveSupport::Logger.new(STDOUT)
   config.lograge.custom_options = lambda do |event|
     options = event.payload.slice(:request_id, :user_id)
@@ -55,4 +56,7 @@
   end
   config.lograge.formatter = Lograge::Formatters::Json.new
   config.log_level = :warn
+  config.paths['log'] = "/var/log/#{ENV['DEPLOY_ENV']}.log"
+  config.paths['tmp'] = "/tmp"
+  config.paths['db'] = "/opt/db"
 end

config/storage.yml

@@ -1,6 +1,6 @@
 test:
   service: Disk
-  root: <%= Rails.root.join("tmp/storage") %>
+  root: /tmp
 
 local:
   service: Disk