Skip to content

nnthanh101/analytics

Repository files navigation

πŸ”₯ CloudOps Automation at Scale πŸ¦…

🌟 You can use Enterprise-grade CloudOps Automation Runbooks, integrating AWS Cloud Foundations & FinOps best practices for DevOps and SRE teams 🌐

πŸ† The CloudOps Runbooks πŸ‘οΈ

The CloudOps Runbooks project shows solid foundation but needs optimization in several key areas: dependency management, CI/CD pipeline efficiency, code organization following cloud-foundations patterns, and integration of battle-tested AWS automation tools.

  • πŸ† Mission: Our mission is to simplify CloudOps Automation for DevOps and SRE teams by providing an extensive, community-driven repository of actions and runbooks that streamline day-to-day operations.
  • πŸ‘οΈ Vision: Our vision is to be the πŸ₯‡ One-Stop Multi-Cloud Platform Engineering & Best Practices Solution for all CloudOps Automation needs, allowing DevOps and SRE teams to automate their workflows with ease, improve efficiency, and minimize toil.
Nhat-Thanh Nguyen Thanh Nguyen Nhat-Thanh Nguyen

PyPI Version Docker Hub CI/CD Status Coverage License


🎯 Key Features

Core Capabilities

  • πŸ—οΈ AWS Cloud Foundations Integration - Battle-tested templates and automation from AWS best practices
  • πŸ“Š Cloud Foundations Assessment Tool (CFAT) - Automated AWS account assessment and compliance checking
  • πŸ” Comprehensive Inventory Scripts - Full AWS resource discovery and management
  • πŸŽ›οΈ Control Tower Automation - Streamlined AWS Control Tower deployment and management
  • πŸ” Identity & Access Management - AWS IAM and SSO/IDC automation
  • πŸ“ Centralized Logging - CloudTrail and Config aggregation patterns
  • 🏷️ Tagging Governance - Organizational tagging policies and enforcement

Technical Excellence

  • ⚑ Lightning-Fast Toolchain - Powered by uv for dependency management, ruff, ty
  • 🐳 Production-Ready PyPi and Containers - Multi-arch Docker images with Wolfi base
  • πŸ§ͺ Comprehensive Testing - Automated testing with pytest and coverage reporting
  • πŸ“š Rich Documentation - MkDocs-powered documentation site
  • πŸ€– AI-Ready - Integration points for OpenAI/Anthropic and local LLMs

πŸš€ Quick Start

1. Installation via pip (Recommended) or uv (Fastest)

pip install runbooks

uv pip install runbooks

2. Installation via Docker

docker run -it nnthanh101/runbooks:latest runbooks --help

3. Run in VSCode's DevContainer 🐳

  1. Open the project in VSCode.
  2. Install the Remote - Containers extension.
  3. Reopen in Container:
    Command Palette Ctrl+Shift+P β†’ Reopen in Container.

Python Template

πŸ› οΈ Features
  • πŸ₯‰ βœ…Lightning-Fast Toolchain: Powered by πŸ“¦ uv - Next-generation Python dependency and build management, πŸ’… ruff - Linting and formatting at blazing speed, and πŸ§ͺ pytest - Robust testing framework with coverage reports.

  • πŸ₯ˆ βœ…Effortless CI/CD Pipelines: πŸ› οΈ Taskfile Automation - Say goodbye to manual SDLC repetitive tasks, 🐳 Containerized Workflows – πŸ›‘οΈ Security-first practices and Multi-stage Wolfi-based Docker builds for scalable production-ready environments, and βš™οΈ Auto-publish to PyPI and GitHub Container Registry (GHCR) with GitHub Actions.

  • πŸ₯‡ β˜‘οΈCloudOps Automation and FinOps Toolkit – Pre-configured hybrid-cloud workflows and seamlessly integrations (jupyterlab, mkdocs, boto3, moto) for managing cloud infrastructure 🌐.

    Feature Toolchain Purpose
    πŸ› οΈ Configuration pyproject.toml Centralized configuration for dependencies, testing, and linting.
    🧹 Task Automation Taskfile Automates repetitive tasks like linting, testing, and publishing.
    πŸ“¦ Python Dependencies uv Lightning-fast dependency resolution, caching, and builds.
    πŸ’… Linting & Formatting ruff Enforces code quality standards, auto-formatting, and import sorting.
    πŸ§ͺ Testing Framework pytest Comprehensive unit tests, integration tests with coverage reporting.
    🐳 Docker Integration Dockerfile + DevContainer Optimized wolfi-based multi-stage builds for CI/CD and local development environments.
    🦾 CI/CD Pipelines GitHub Actions Automated builds, tests, and deployments to PyPI and GHCR.
    πŸ“ Security Compliance chainguard/wolfi-base + SBOM + Attestations Ensures compliance, vulnerability scanning, and security transparency.
Project Structure

πŸ›  End-to-end Production-grade project structure for successful πŸ’Ž CloudOps Automation and Visual Analytics FinOps projects πŸš€

cloudops-automation/
β”œβ”€β”€ .devcontainer/     ## Dev Container configurations
β”‚   └── Dockerfile     ## Container image build file
β”œβ”€β”€ .github/           ## CI/CD workflows
β”‚   β”œβ”€β”€ workflows/     ## GitHub Actions workflows
β”‚   └── templates/     ## Workflow templates
β”œβ”€β”€ .vscode/           ## IDE-specific configurations
β”œβ”€β”€ config/            ## Configuration files (YAML, JSON)
β”œβ”€β”€ data               πŸ” Where all your raw and processed data files are stored.
β”‚   β”œβ”€β”€ external       <- Data from third-party sources.
β”‚   β”œβ”€β”€ interim        <- Intermediate data that has been transformed.
β”‚   β”œβ”€β”€ processed      <- The final, canonical data sets for modeling.
β”‚   └── raw            <- The original, unprocessed, immutable data dump.
β”‚
β”œβ”€β”€ docs               πŸ““ A default mkdocs project; see mkdocs.org for details
β”‚   β”œβ”€β”€ api/                 ## API documentation
β”‚   β”œβ”€β”€ architecture/        ## Architecture diagrams
β”‚   β”œβ”€β”€ tutorials/           ## Tutorials and guides
β”‚   β”œβ”€β”€ getting-started.md   ## Quickstart guide
β”‚   └── index.md             ## Overview documentation
β”‚
β”œβ”€β”€ logs/                    ## Log files for debugging
|
β”œβ”€β”€ models             🧠 Store your trained and serialized models for easy access and versioning.
β”‚
β”œβ”€β”€ notebooks          πŸ’» Jupyter notebooks for experiments and visualization.
β”‚   β”œβ”€β”€ data_exploration.ipynb
β”‚   β”œβ”€β”€ data_preprocessing.ipynb
β”‚   β”œβ”€β”€ model_training.ipynb
β”‚   └── model_evaluation.ipynb
β”‚
β”œβ”€β”€ pyproject.toml     <- Project configuration file with package metadata for 
β”‚                         runbooks and configuration for tools like black
β”‚
β”œβ”€β”€ src/                            ## 🧩 Source code for use in this project.
β”‚   β”œβ”€β”€ runbooks/                   ## Main module for CloudOps Runbooks automation
β”‚   β”‚   β”œβ”€β”€ __init__.py             ## Package initializer
β”‚   β”‚   β”œβ”€β”€ calculator.py           ## [Python101] Calculator
β”‚   β”‚   β”œβ”€β”€ config.py
β”‚   β”‚   β”œβ”€β”€ exceptions.py
β”‚   β”‚   β”œβ”€β”€ utils.py                ## Utility scripts (logging, configs)
β”‚   β”‚   └── cleanup.py              ## Cleanup automation runbook
β”‚   β”œβ”€β”€ main.py     
β”œβ”€β”€ test/                           ## Unit and integration tests
β”‚   β”œβ”€β”€ conftest.py
β”‚   β”œβ”€β”€ __init__.py
β”‚   β”œβ”€β”€ test_calculator.py          ## [Python101] Test cases for calculator
β”‚   β”œβ”€β”€ test_utils.py               ## Test cases for utils
β”‚   └── test_exceptions.py         
β”œβ”€β”€ templates/                      ## Terraform and CloudFormation templates
β”œβ”€β”€ tools/                          ## Developer tools and scripts
β”œβ”€β”€ .dockerignore                   ## Docker ignore file
β”œβ”€β”€ .env                            ## Environment variables
β”œβ”€β”€ .gitignore                      ## Git ignore file
β”œβ”€β”€ .python-version                 ## Python version management
β”œβ”€β”€ .gitignore
β”œβ”€β”€ mkdocs.yml                      # Documentation generator configuration
β”œβ”€β”€ README.md          🀝 Explain your project and its structure for better collaboration.
β”œβ”€β”€ references         <- Data dictionaries, manuals, and all other explanatory materials.
β”‚
β”œβ”€β”€ reports            πŸ“Š Generated analysis (reports, charts, and plots) as HTML, PDF, LaTeX.
β”‚   └── figures        <- Generated graphics and figures to be used in reporting
β”‚
β”œβ”€β”€ requirements.txt   πŸ›  The requirements file for reproducing the analysis environment, for easy environment setup.
└── Taskfile           <- Taskfile with convenience commands like `task data` or `task train`

About

🌟 An end-to-end full-stack data science project, including modelling, MLOps, and data storytelling. ✨

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published