[EPIC] Implement Core Microservice Infrastructure

[maxzaikin]

Description

Current State & Problem

The TGB-MicroSuite services (a-rag, tg-gateway) currently communicate in a simplistic, non-resilient manner (e.g., direct HTTP calls defined in configuration). This architecture lacks key production-grade capabilities such as service discovery, fault tolerance, and observability. As the system grows, this direct coupling will lead to brittleness, difficulty in scaling, and an inability to monitor the system's health effectively.

The primary architectural challenge is to evolve the project from a set of loosely-coupled services into a robust, observable, and resilient microservice ecosystem.

Goals & Objectives

This epic focuses on integrating a foundational set of infrastructure components to enable a more professional microservice architecture. This will be achieved by containerizing all components and orchestrating them with a focus on dynamic discovery, asynchronous communication, and comprehensive monitoring.

---

1. Implement Service Discovery

• Description: Enable services to dynamically find and communicate with each other without hardcoded addresses.
• Tasks:
  • Integrate Consul into the docker-compose.infra.yml setup.
  • Modify a-rag and tg-gateway to register themselves with Consul on startup.
  • Implement health check endpoints (/health) in all services for Consul to monitor.
  • Refactor inter-service communication to query Consul for target service addresses.

2. Introduce an API Gateway

• Description: Establish a single, managed entry point for all external traffic, enhancing security and simplifying client-side configuration.
• Tasks:
  • Integrate Ocelot as an API Gateway service (apigateway) in docker-compose.infra.yml.
  • Configure Ocelot to use Consul for dynamic routing to backend services (a-rag, etc.).
  • Remove externally exposed ports from individual service configurations in docker-compose.yml, routing all traffic through the gateway.

3. Implement Asynchronous Communication

• Description: Decouple services by introducing a message broker for asynchronous tasks, improving resilience and scalability.
• Tasks:
  • Integrate RabbitMQ into the docker-compose.infra.yml setup.
  • Refactor the tg-gateway to a-rag communication flow: instead of a direct HTTP call, tg-gateway will publish a job to a RabbitMQ queue.
  • Implement a worker process within a-rag to consume jobs from the RabbitMQ queue.

4. Establish Core Observability Stack

• Description: Implement a telemetry pipeline to collect, store, and visualize metrics and logs from all services.
• Tasks:
  • Integrate OpenTelemetry Collector and VictoriaMetrics into docker-compose.infra.yml.
  • Instrument Python services (a-rag, tg-gateway) with the OpenTelemetry SDK to emit key metrics (e.g., request latency, error rates).
  • Configure the collector to receive telemetry data and export it to VictoriaMetrics.

Definition of Done

• All core infrastructure components (Consul, Ocelot, RabbitMQ, VictoriaMetrics) are running via Docker Compose.
• Services a-rag and tg-gateway successfully register with Consul.
• External API access to a-rag is routed exclusively through the Ocelot API Gateway.
• Inter-service communication for core tasks is handled asynchronously via RabbitMQ.
• Basic application metrics are being collected and are viewable.