Welcome to Crato 👋

Crato is an open source framework for small applications to easily deploy centralized logging. Crato is built with Node.js, Rsyslog, Apache Kafka, InfluxDB, and AWS S3. This repo contains the core Crato project that receives, transforms, and stores log data for further analysis.

🏠 Homepage

Table of Contents

• Crato Usage
• Environment Variables
• Install
• Deploying Crato
• Networking
• Show Your Support

Crato Usage

Crato provides a CLI to make using the core system and tracking logs much easier. It requires installing both NPM and Node.js.

After installing the above dependencies, run npm install && npm link which will make the crato command available.

Here are a list of Crato commands:

Commands	Description
services	Provides a listing and description of all of Crato's services
deploy	Starts up Crato system
shutdown	Stops Crato system and all services
start <service>	Starts a specific Crato service
stop <service>	Stops a specific Crato service
install-kafka	Installs Kafka cluster and textlogs & jsonlogs topics
container-logs | cl <service>	Displays Docker container logs for a specific service
live-tail | lt	See all external logs streaming into Crato. Press Ctrl-C to exit
shell | sh <service>	Attaches a shell for a specific Crato service
status	Displays the status of all of Crato's services

Environment Variables

Crato uses some environment variables for its Node.js consumer app mainly to archive logs to Amazon Web Services (AWS) Simple Storage Service (S3).

Crato uses AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_S3_BUCKET_NAME to authenticate for AWS S3. We also require the QUEUE_MAX_SIZE to determine the number of logs to queue, before being uploaded to AWS S3.

These environment variables should be written on your local machine and Docker will automatically read them if they are. See this great guide by Twilio for how to set environment variables on different operating systems.

Install

The Crato system is deployed and orchestrated through Docker and Docker-Compose with the docker-compose.yml file. Both of these must be installed.

To install Crato, clone this repo. Then run docker-compose pull to download the necessary Docker images to build the service containers. The install can be finished via Crato CLI commands or Docker or Docker-Compose CLI commands.

Deploying Crato

Using Crato CLI

1. crato install-kafka
2. crato deploy

Using Docker-Compose CLI

1. docker-compose up -d zookeeper
2. docker-compose up -d kafka
3. docker-compose exec kafka kafka-topics --create --zookeeper zookeeper:8092 --replication-factor 3 --partitions 6 --topic textlogs
4. docker-compose exec kafka kafka-topics --create --zookeeper zookeeper:8092 --replication-factor 3 --partitions 6 --topic jsonlogs
5. docker-compose up -d

Use crato status or docker ps -a to check the status of the service containers that have been created.

Networking

Crato exposes ports 514 and 10514 for receiving log data into the system. On your own host machine where Crato is deployed, make sure these are available.

Show your support

Give a ⭐️ if you liked this project!

---