Calculator Microservices

A microservices-based calculator system built using Go (gRPC) and Node.js (REST). It uses Kafka for message communication with outbox pattern. also Observability and Moitoring using Prometheus and Grafana.

---

🧩 Components

1. gRPC Producer (Go)

• Exposes calculator operations (Add, Subtract, Multiply, Divide) via gRPC.
• Stores requests in a PostgreSQL-backed outbox table.
• Polls the outbox table and publishes events to Kafka.

2. Kafka Poller (Go)

• Periodically reads events from the outbox table.
• Publishes events to a Kafka topic (calculations).

3. Kafka Consumer & Metrics Exporter (Node.js)

• Consumes messages from the calculations topic.
• Logs results to a file.
• Exposes Prometheus metrics:
  • messages_consumed_total
  • kafka_message_latency_seconds (histogram)

4. Prometheus

• Scrapes and stores time-series metrics.
• Accessed via http://localhost:9090.

5. Grafana

• Visualizes Prometheus metrics with custom dashboards.
• Accessed via http://localhost:3000.

6. K6 Load Testing

• Performance and stress testing on the gRPC API.
• Located in k6_testing/test.js.

7. Docker

• Services are containerized using Docker.
• docker-compose.yml runs Kafka, PostgreSQL, Prometheus, and Grafana.
• Dockerfile provided for both gRPC and REST applications.

---

🚀 Quick Start

⚙️ Prerequisites

• Docker + Docker Compose
• PostgreSQL, Kafka, Prometheus, and Grafana run in Docker Compose
• Node.js
• Go

---

📁 Project Structure

├── calculator_grpc
│   ├── calculator.proto
│   ├── client
│   │   └── main.go
│   ├── create_outbox_table.sql
│   ├── go.mod
│   ├── go.sum
│   ├── pb
│   │   ├── calculator.pb.go
│   │   └── calculator_grpc.pb.go
│   ├── poller
│   │   └── poller.go
│   └── server.go
├── calculator_rest
│   ├── calculations.log
│   ├── index.js
│   ├── package-lock.json
│   └── package.json
├── docker-compose.yml
├── k6_testing
│   └── test.js
├── prometheus.yml
└── readme

---

🛠️ Setup Guide

Step 1: Start Dependencies (Locally)

docker-compose up -d

Verify services:

• Kafka: localhost:9092
• PostgreSQL: localhost:5431 (user: postgres, password: mysecretpassword)
• Prometheus: http://localhost:9090
• Grafana: http://localhost:3001 (login: admin / admin)

---

🧮 Running the Go gRPC App

Step 2.1: Install Go Modules

cd calculator_grpc
go mod tidy

Step 2.2: Create the outbox_events Table

Default password: mysecretpassword (used in server.go)

Step 2.3: Run the gRPC Server

go run server.go

Step 2.4: Run the Kafka Poller

In a separate terminal:

go run poller/poller.go

---

🌐 Running the Node.js Kafka Consumer

Step 3.1: Install Dependencies

cd calculator_rest
npm install

Step 3.2: Run the Consumer and REST API

node index.js

• Exposes GET /last-record
• Exposes Prometheus metrics at: http://localhost:3000/metrics

---

📡 Calling the gRPC API

You can use a gRPC client, or run the CLI client in calculator_grpc/client/main.go.

Example:

grpcurl -plaintext -d '{"num1": 3, "num2": 4}' localhost:50051 CalculatorService/Add

---

Step 4: View Metrics

🔍 Prometheus (http://localhost:9090)

Try queries like:

• messages_consumed_total
• rate(messages_consumed_total[1m])
• histogram_quantile(0.95, rate(kafka_message_latency_seconds_bucket[1m]))

📊 Grafana (http://localhost:3001)

• Add Prometheus data source (URL: http://host.docker.internal:9090)

• Create dashboards with:

  • messages_consumed_total
  • kafka_message_latency_seconds

---

🧪 K6 Testing

To run load tests on the gRPC API:

cd k6_testing
k6 run test.js

Ensure gRPC server is running locally or exposed on the target port.

---

📌 Notes

• Outbox pattern ensures reliable delivery from DB to Kafka.
• Latency is calculated using production timestamp inside Kafka message and time of consumption.
• Prometheus scrapes metrics from REST container's /metrics endpoint.

---

📦 Future Improvements

• Add retry mechanism and dead-letter queue
• Add persistent volume claims for stateful services
• Visualize latency per operation (add, subtract, etc.)
• full Kubernetes deployment

---

👨‍💻 Author

Abdelrahman Hedia

---

📄 License

MIT License