Skip to content

OS Preparation - Packages Pre Installation (CentOS Stream)

Notifications You must be signed in to change notification settings

charlietag/os_preparation

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Table of Contents

Try Ubuntu now

Try Ubuntu World ===> https://github.com/charlietag/ubuntu_preparation

CentOS Linux Server OS Preparation

You want initialize your linux server by your own script. But you DO NOT want to use PUPPET , CHEF , Ansible. You can just leverage my initialization project here.

This is a small light bash project. Suit small companies which have only few servers to maintain. GIVE IT A TRY!!

centos stream release 9 server environment settings

  • This is useful when

    • You have less than 5 CentOS-Stream-9 servers to maintain.
    • You are deploying monolithic architecture app.
  • This repo is TOTALLY transfer from passenger to puma for rails.

    • NGINX + PUMA + PHP-FPM + MariaDB + Rails + Laravel + Redmine

Environment

  • CentOS Stream release 9

    • os_preparation
      • release : master v3.x.x
  • CentOS Stream release 8

    • os_preparation
      • release : v2.x.x
  • CentOS 8 (8.x)

    • os_preparation
      • release : v1.x.x
  • CentOS 7 (7.x) (deprecated)

    • os_preparation
      • release : v0.x.x
  • CentOS 7 (7.x) - passenger + nginx version (deprecated)

    • os_preparation
      • release : before_passenger_to_puma

      • If you prefer passenger + nginx (passenger-install-nginx-module)

        please switch to git tag named "before_passenger_to_puma"

        git clone --depth 1 --branch before_passenger_to_puma https://github.com/charlietag/os_preparation.git

Notice

  • Due to CentOS 8 - EPEL-modular repo always failed everytime updating metadata cache. This will disable repo cache expiration permanently, instead, using dnf-automatic / yum-cron to makecache
    • [Optional] Make cache before doing DNF / YUM installation
      • /root/bin/dnf.sh (alias dnf) will help you with this
      • dnf makecache / yum makecache
    • Disabled
      • dnf repo cache expiration
        • dnf config-manager --setopt metadata_expire=-1 --save
    • Enabled
      • dnf-automatic
        • For dnf makecache by daily
  • Before os_security
  • Systemd target
    • Default target (os_preparation will force to use this target)
      • multi-user.target
    • Ref. /etc/inittab
      • multi-user.target: analogous to runlevel 3
      • graphical.target: analogous to runlevel 5
    • WARNING If you are under graphical.target NOT under multi-user.target.
      • It is highly recommended that you do the following:
        • Reinstall whole CentOS using "Minimal Install" / "Server"
      • Or at least try the following:
        • systemctl set-default multi-user
        • reboot
        • dnf groupinstall 'Minimal Install' (If this is not working, try also dnf groupinstall 'Server')
        • dnf groupremove 'Server with GUI'
        • reboot
        • you can start with os_prepation now
      • Reference description here
    • Check method
      • systemctl get-default
  • Environment Groups
    • Based on (os_preparation will make sure this environment group is installed)
      • "Minimal Install"
    • Removing Environment Groups (os_preparation will remove these environment groups by default)
      • "Server with GUI"
      • "Workstation"
      • "KDE Plasma Workspaces"
      • "Virtualization Host"
      • "Custom Operating System"
    • Check method
      • dnf grouplist

Warning

  • Please do this in fresh install OS
  • What does this not cover, DO the following manually
    • Login user
      • Change password of root
      • Add GENERAL USER and setup password of GENERAL USER
    • /etc/ssh/sshd_config
      • PermitRootLogin no
      • PasswordAuthentication yes
    • RAM
      • mkswap if RAM is insufficient to start MariaDB

        mkdir /swap
        dd if=/dev/zero of=/swap/swapfile bs=1M count=4096
        mkswap /swap/swapfile
        chmod 0600 /swap/swapfile
        /sbin/swapon /swap/swapfile
        chmod 755 /etc/rc.d/rc.local
        echo "/sbin/swapon /swap/swapfile" >> /etc/rc.d/rc.local

Configuration

  • ssh without SendEnv

    • command ssh

      • ssh_to

      • config /etc/ssh/ssh_config

        # SendEnv LANG LC_*
    • iTerm2 setting iterm2_disable_setting_LC_ALL.png

  • Before installation

    dnf clean all
    dnf install -y git
    git clone https://github.com/charlietag/os_preparation.git
  • Make sure config files exists , you can copy from sample to modify.

    cd databag
    ls |xargs -I{} bash -c "cp {} \$(echo {}|sed 's/\.sample//g')"
  • Mostly used configuration :

    • DEV use (server in Local / server in Cloud) && Production use (server in Local / server in Cloud)

      databag/
      ├── F_01_ENV_02_os_01_env.cfg
      ├── F_01_ENV_04_ssh_config.cfg
      └── _gitconfig.cfg
  • Verify config files (with syntax color).

    cd databag
    
    echo ; \
    ls *.cfg | xargs -I{} bash -c " \
    echo -e '\e[0;33m'; \
    echo ---------------------------; \
    echo {}; \
    echo ---------------------------; \
    echo -n -e '\033[00m' ; \
    echo -n -e '\e[0;32m'; \
    cat {} | grep -vE '^\s*#' |sed '/^\s*$/d'; \
    echo -e '\033[00m' ; \
    echo "
  • Verify ONLY modified config files (with syntax color).

    cd databag
    
    echo ; \
    ls *.cfg | xargs -I{} bash -c " \
    echo -e '\e[0;33m'; \
    echo ---------------------------; \
    echo {}; \
    echo ---------------------------; \
    echo -n -e '\033[00m' ; \
    echo -n -e '\e[0;32m'; \
    cat {} | grep -v 'plugin_load_databag.sh' | grep -vE '^\s*#' |sed '/^\s*$/d'; \
    echo -e '\033[00m' ; \
    echo "

Easy Installation

I'm a lazy person. I want to install ALL and give me default configurations running Nginx , MariaDB, php-fpm, puma 5 (rails). And help me to create default projects about "Rails" and "Laravel"

  • Command

    ./start -a
    reboot
  • Default project path

    • DEFAULT user for rails/laravel developer is not ssh allowed

      • /etc/ssh/sshd

        DenyGroups no-ssh-group
    • group "no-ssh-group" add to default dev user

      • phpuser (this name can be modified)
      • rubyuser (this name can be modified)
    • rails

      • default user: rubyuser (can be changed)
    /home/${current_user}/rails_sites/myrails/
    --->
    /home/rubyuser/rails_sites/myrails/
    • Redmine
      • default user: rubyuser (can be changed)
    /home/${current_user}/rails_sites/redmine/
    --->
    /home/rubyuser/rails_sites/redmine/
    • laravel
      • default user: phpuser (can be changed)
    /home/${current_user}/laravel_sites/myrails/
    --->
    /home/phpuser/laravel_sites/myrails/
  • Config your own hosts file (/etc/hosts)

    <192.168.x.x> myrails.centos8.localdomain
    <192.168.x.x> redmine.centos8.localdomain
    <192.168.x.x> mylaravel.centos8.localdomain
  • Browse URL

    http://myrails.centos8.localdomain
    http://redmine.centos8.localdomain (default account: admin/admin)
    http://mylaravel.centos8.localdomain

Advanced Installation

I want to choose specific part to install.

  • Command

    ./start.sh -h
    usage: start.sh
      -a                   ,  run all functions
      -i func1 func2 func3 ,  run specified functions

Customize your own function

Folder

  • functions/

    • Write your own script here, file named start with F_[0-9][0-9]_YourOwnFuntionName.sh

    • Run command

      ./start.sh -i YourOwnFuntionName
  • templates/

    • Put your own templates here, folder named the same as YourOwnFuntionName
  • databag/

    • Put your special config variables here, file named the same as YourOwnFuntionName
    • How to use
      • In databag/YourOwnFunctionName

        • local your_vars_here
      • In templates/YourOwnFunctionName/yourowntemplate_file

        • You can use ${your_vars_here}
      • In YourOwnFuntionName , you can call

        # Method : eval "echo \"$variable\""
        # Might have escape issue, if template is complicated
        RENDER_CP ${$CONFIG_FOLDER}/yourowntemplate_file /SomeWhere/somewhere
        # Method : cat template | sed 's/\{\{var\}\}/$var/g'
        # BETTER method for rendering template
        RENDER_CP_SED ${$CONFIG_FOLDER}/yourowntemplate_file /SomeWhere/somewhere

        instead of

        cp ${$CONFIG_FOLDER}/yourowntemplate_file /SomeWhere/somewhere
      • In YourOwnFuntionName , you just want to LOAD VARIABLES ONLY from databag, try add a comment into your function script

        # For Load Variables Only Usage, add the following single comment line with keyword DATABAG_CFG:enable
        # DATABAG_CFG:enable
  • helpers/

    • Write your own script here, file named start with helper_YourOwnHelperName.sh
    • Works with helpers_views
  • helpers_views/

    • Put your own templates for ONLY helper USE here, folder named the same as YourOwnHelperName
  • tasks/

    • Write your own script here, file named start with task_YourOwnTaskName.sh , _task_YourOwnTaskName.sh
    • Scripts here will automatically transfer to function, just like scripts under "functions/"
    • But this is for global use for os_preparation , os_security. So it's been moved to os_preparation_lib
  • plugins/

    • Only scripts which can be called everywhere like, ${HELPERS}/plugins_scripts.sh
    • Use this as a script, not function

Predefined variables

(root)# ./start.sh -i F_00_debug
#############################################
         Preparing required lib
#############################################
Updating required lib to lastest version...
Already up to date.

#############################################
            Running start.sh
#############################################

---------------------------------------------------
NTP(chrony) ---> pool.ntp.org
---------------------------------------------------
RUN: chronyd -q 'pool pool.ntp.org iburst'
2020-09-08T01:47:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS +SECHASH +IPV6 +DEBUG)
2020-09-08T01:47:38Z System clock wrong by -0.002320 seconds (step)
2020-09-08T01:47:38Z chronyd exiting
RUN: hwclock -w
---------------------------------------------------


==========================================================================================
        F_00_debug
==========================================================================================
-----------lib use only--------
CURRENT_SCRIPT : /root/os_preparation/start.sh
CURRENT_FOLDER : /root/os_preparation
FUNCTIONS      : /root/os_preparation/functions
LIB            : /root/os_preparation/../os_preparation_lib/lib
TEMPLATES      : /root/os_preparation/templates
TASKS          : /root/os_preparation/../os_preparation_lib/tasks
HELPERS        : /root/os_preparation/helpers
HELPERS_VIEWS  : /root/os_preparation/helpers_views

-----------lib use only - predefined vars--------
FIRST_ARGV     : -i
ALL_ARGVS      : F_00_debug

-----------function use only--------
PLUGINS            : /root/os_preparation/plugins
TMP                : /root/os_preparation/tmp
CONFIG_FOLDER      : /root/os_preparation/templates/F_00_debug
DATABAG            : /root/os_preparation/databag
DATABAG_FILE       : /root/os_preparation/databag/F_00_debug.cfg

-----------function extended use only--------
IF_IS_SOURCED_SCRIPT  : True: use 'return 0' to skip script
IF_IS_FUNCTION        : True: use 'return 0' to skip script
IF_IS_SOURCED_OR_FUNCTION  : True: use 'return 0' to skip script

${BASH_SOURCE[0]}    : /root/os_preparation/functions/F_00_debug.sh
${0}                 : ./start.sh
${FUNCNAME[@]}          : source F_00_debug L_RUN L_RUN_SPECIFIED_FUNC source source main
Skip script sample    : [[ -n "$(eval "${IF_IS_SOURCED_OR_FUNCTION}")" ]] && return 0 || exit 0
Skip script sample short : eval "${SKIP_SCRIPT}"

================= Testing ===============
----------Helper Debug Use-------->>>

-------------------------------------------------------------------
        helper_debug
-------------------------------------------------------------------
HELPER_VIEW_FOLDER : /root/os_preparation/helpers_views/helper_debug


----------Task Debug Use-------->>>

-----------------------------------------------
        task_debug
-----------------------------------------------

Note

Installed Packages

Folder privilege

After this installation repo, the server will setup with "Nginx + Puma (socket)" , "Nginx + PHP-FPM (socket)" , so your Rails, Laravel, can run on the same server. The following is something you have to keep an eye on it.

  1. folder privilege
  • Rails Project

    rails new <rails_project> -d mysql --skip-spring
    cd <rails_project>
    chown -R ${current_user}.${current_user} log tmp
  • Laravel Project

    composer create-project --prefer-dist laravel/laravel <laravel_project>
    cd <laravel_project>
    chown -R ${current_user}.${current_user} storage
    chown -R ${current_user}.${current_user} bootstrap/cache
  1. Command
  • Rails

    rails new <rails_project> -d mysql --skip-spring
  • Rails 5.1 has dropped dependency on jQuery, you might want it back via yarn

    1. Add npm of jquery using Yarn

      cd <rails_project>
      yarn add jquery
    2. Setup jquery npm for asset pipeline

      vi <rails_project>/app/assets/javascripts/application.js
      //= require rails-ujs
      //= require turbolinks
      //= require jquery/dist/jquery
      //= require bootstrap/dist/js/bootstrap
      //= require_tree .
    3. Yarn works with rails 5.1 asset pipeline as below

    • Usage for default path: <rails_project>/node_modules/{pkg_name}/dist/{pkgname}.{js,css}

      //= require jquery
    • If package is different from this rule, ex: bootstrap. You might specify explicitly (better)

      //= require jquery
      //= require jquery/dist/jquery
      //= require bootstrap/dist/js/bootstrap
  • Laravel

    composer create-project --prefer-dist laravel/laravel <laravel_project>
  • Useful script snippet

    • If you are always get disconnected, and you want to kill last failed connection of SSH

      netstat -palunt |grep -i est | awk '{print $7}'| cut -d'/' -f1 |xargs -I{} bash -c "ps aux |grep sshd |grep {}|grep -v grep" | head -n -1 | awk '{print $2}' |xargs -I{} kill {}
    • If you want to restart network for new config, instead of using systemctl restart network, which is deprecated in CentOS 8

      • Reload network config (mostly, this would work)

        nmcli c reload
      • Stop networking and start networking in NetworkManager (NM)

        nmcli n off; nmcli n on
    • Since RHEL 9 , no more configs under /etc/sysconfig/network-scripts/, instead, keyfile under /etc/NetworkManager/system-connections only. So config network using nmcli will be a better method

    • Modify network static IP using nmcli

      • Setup static ip

        nmcli connection modify eth0 \
          ipv4.addresses 192.168.122.7/24 \
          ipv4.gateway 192.168.122.1 \
          ipv4.dns 192.168.122.1 \
          ipv4.method manual
      • Disable IPv6, and peerDNS

        nmcli connection modify eth0 \
          ipv4.ignore-auto-dns "true"
        
        nmcli connection modify eth0 \
          ipv6.method "disabled" \
          ipv6.addr-gen-mode "stable-privacy" \
          ipv6.ignore-auto-dns "true" \
          ipv6.ignore-auto-routes "true" \
          ipv6.never-default "true"
      • List only device name except loop 0 using nmcli

        nmcli -g name connection show

Ruby gem config

  • gem install without making document
    • Deprecated

      no-ri, no-rdoc

    • Config

      echo "gem: --no-document" > ~/.gemrc

Database configuration for production

  • Remove test database and setup root password

    After doing this, still need some tweak, try to manage database with https://www.adminer.org/

    $ mysql_secure_installation

    Just keep hitting <ENTER>, to USE ALL DEFAULT SETTING

  • After mysql_secure_installation

    • MariaDB 10.5 auth method will just like MariaDB 10.3
  • Database tools - Adminer

Extra functions

  • RENDER_CP

    • Render template using eval (Might have escape issue, if template is complicated)

      # Method : eval "echo \"$variable\""
    • Sample

      • databag
      local var="Hello World"
      • template (${$CONFIG_FOLDER}/yourowntemplate_file)
      This is $var
      • function
      RENDER_CP ${$CONFIG_FOLDER}/yourowntemplate_file /SomeWhere/somewhere
      • result (/SomeWhere/somewhere)
      This is Hello World
  • RENDER_CP_SED

    • Render template using sed (BETTER method for rendering template)

      # Method : cat template | sed 's/\{\{var\}\}/$var/g'
    • Sample

      • databag
      local var="Hello World"
      • template (${$CONFIG_FOLDER}/yourowntemplate_file)
      This is {{var}}
      • function
      RENDER_CP_SED ${$CONFIG_FOLDER}/yourowntemplate_file /SomeWhere/somewhere
      • result (/SomeWhere/somewhere)
      This is Hello World
  • SAFE_DELETE

    • Check file names and path before rm any dangerous files, preventing from destoying whole server

      • check for the following dangerous key words

        .
        ..
        *
        /
        .*
        *.*
        "$(echo "$(find / -maxdepth 1 ;  readlink -m /* )" | sort -n | uniq)"
    • Sample

      # --- Should be failed ---
      DELETE_FILE="/root/delete_me/.*"
      # --- safe delete command usage ---
      SAFE_DELETE "${DELETE_FILE}"

(Git) Stash details

  • Ref. https://git-scm.com/docs/git-stash

  • (Git) stash list

    $ git stash list
    stash@{0}: WIP on redmine_4.0.7: a853fc0 Fix sort projects table by custom field (#32769).
    stash@{1}: WIP on redmine_4.0.6: 22ebc68 tagged version 4.0.6
    • redmine_4.0.6 / redmine_4.0.7, these mean branch name
    • if you want to restore data, you'd better checkout the the related branch
  • Display all stash contents

    git stash list | cut -d':' -f1 | xargs -I{} bash -c "\
      echo; \
      echo ----------------------------------------------- {} -----------------------------------------------;\
      git stash show -p {}; echo\
    "

(Git) Push and Pull

  • Push git commits to remote

    git push

  • Push git tags to remote

    git push --tags

  • Fetch git commits to local

    git fetch

  • Fetch git tags to local

    git fetch --tags

  • Fetch git commits to local and then MERGE to Working Directory

    git pull

Upgrading Redmine

Reference

Backup current redmine

  • Database
    • mysqldump -u {db_user} -p --lock-all-tables --skip-tz-utc redmine > redmine_$(date +"%Y%m%d")_skip-tz-utc.sql
  • Application & files
    • cp -a redmine redmine_bak

Customized files

  • plugins
    • /home/rubyuser/rails_sites/redmine/plugins/redmine_*
  • themes
    • /home/rubyuser/rails_sites/redmine/public/themes/{a1,circle,PurpleMine2}
  • session token
    • /home/rubyuser/rails_sites/redmine/config/initializers/secret_token.rb
  • uploaded files
    • /home/rubyuser/rails_sites/redmine/files/

(Method 1) Upgrading from a git checkout

  • Stop puma server

    • puma-systemd-mgr -p -i redmine
  • Go to the Redmine root directory and run the following command:

    cd redmine
    git stash
    git checkout master
    git fetch
    git fetch --tags
    git pull

    sometimes git pull will not fetch tags, instead, we need to fetch tags by git fetch --tags

    especially when tags name or tags <-> commit , has been changed

    git co 4.0.7 -b redmine_4.0.7
    git stash pop
    git status |grep 'both modified:' |awk '{print $3}' |xargs -I{} bash -c "echo --- git reset HEAD {} ---; git reset HEAD {}"
  • Fix conflicts

  • Perform the upgrade

    # gemset name using redmine version
    echo "gemset_redmine_4.1.0" > .ruby-gemset
    
    # switch to the new gemset
    cd
    cd -
    
    # Update gem / bundler for this gemset
    gem update --system
    gem install bundler
    
    # Install the required gems by running the following command
    bundle update
    
    # Update the database
    bundle exec rake db:migrate RAILS_ENV=production
    bundle exec rake redmine:plugins RAILS_ENV=production
    
    # Clean up
    bundle exec rake tmp:cache:clear RAILS_ENV=production
  • Start puma server

    • puma-systemd-mgr -s -i redmine
  • Go to "Admin -> Roles & permissions" to check/set permissions for the new features, if any.

  • Finally, clear browser's cached data (To avoid strange CSS error)

    • Chrome -> History -> Clear History -> Choose ONLY "Cached images and files"

(Method 2) Upgrading from a fresh installation

  • Stop puma server

    • puma-systemd-mgr -p -i redmine
  • Backup current redmine

  • Remove the following lines from script functions/F_02_PKG_06_ruby_09_redmine_create.sh (F_02_PKG_06_ruby_09_redmine_create_diff.png)

    if [[ -z "${redmine_db_pass}" ]]; then
      mysql -u root -e "CREATE DATABASE ${redmine_db_name} CHARACTER SET utf8;"
    else
      mysql -u root -p${redmine_db_pass} -e "CREATE DATABASE ${redmine_db_name} CHARACTER SET utf8;"
    fi
    su -l $current_user -c "cd ${redmine_web_root} && bundle _${this_redmine_bundler_version}_ exec rake generate_secret_token"
    if [[ -n "${redmine_default_lang}" ]]; then
      su -l $current_user -c "cd ${redmine_web_root} && bundle _${this_redmine_bundler_version}_ exec rake redmine:load_default_data RAILS_ENV=production REDMINE_LANG=${redmine_default_lang}"
    fi
  • Perform the fresh installation

    • ./start -i F_02_PKG_06_ruby_09_redmine_create
  • Restore files from backup

    • redmine/config/initializers/secret_token.rb
    • redmine/files/
  • Start puma server

    • puma-systemd-mgr -s -i redmine
  • Go to "Admin -> Roles & permissions" to check/set permissions for the new features, if any.

  • Finally, clear browser's cached data (To avoid strange CSS error)

    • Chrome -> History -> Clear History -> Choose ONLY "Cached images and files"

Upgrading MariaDB

For some cases, we need to upgrade MariaDB without data lost. Here is my note about this.

Reference (mariadb.com)

How to Upgrade

  • Backup current database

    # mysqldump -u root -p --lock-all-tables --skip-tz-utc -A > all_`date +"%Y%m%d"`_skip-tz-utc.sql
  • Stop MariaDB

    # systemctl stop mariadb
  • Uninstall the old version of MariaDB

    dnf remove -y MariaDB-common MariaDB-client MariaDB-shared MariaDB-server MariaDB-devel
  • Modify the repository configuration to newer version

  • Install the new version of MariaDB

    dnf install -y MariaDB-common MariaDB-client MariaDB-shared MariaDB-server MariaDB-devel
  • Make any desired changes to configuration options in option files, such as my.cnf. This includes removing any options that are no longer supported.

    # cat /etc/my.cnf.d/server.cnf | grep -B1 '127.0.0'
    [mysqld]
    bind-address = 127.0.0.1
  • Start MariaDB

    # systemctl start mariadb
  • Run mysql_upgrade

    • mysql_upgrade does two things:
      • Ensures that the system tables in the#mysql database are fully compatible with the new version.
      • Does a very quick check of all tables and marks them as compatible with the new version of MariaDB .
    • mysql_upgrade -u root -p
      • After this command, there would be a file generated for letting you know this database has already been upgraded. (owner of the file is root)

        $ ls /var/lib/mysql | grep upgrade
        -rw-r--r--  1 root  root    15 Sep  9 14:24 mysql_upgrade_info
        $ cat mysql_upgrade_info
        10.5.5-MariaDB
  • Restart MariaDB - Done

    • It would be better to restart MariaDB, if it's allowed.

      # systemctl restart mariadb

CHANGELOG