Production-Grade eBay Market Monitor

This project is a modular, production-ready system for monitoring eBay market trends. It separates concerns into distinct modules for scraping, analysis, and visualization, orchestrated by a central Celery task. It includes a real-time event monitor to track task progress.

Architecture

• scraper.py: Handles robust web scraping with proxy support (Bright Data) and automatic fallbacks.
• analizer.py: Manages the AI analysis workflow using LangGraph and Google Gemini.
• visualizer.py: Generates professional interactive dashboards using Bokeh.
• main.py: The entry point that defines the Celery task and orchestrates the pipeline.
• monitor.py: A real-time event listener that tracks the status of Celery tasks (Received, Started, Succeeded, Failed).

Setup

1. Install Dependencies:

   pip install requests beautifulsoup4 celery redis bokeh pandas langchain langgraph langchain-google-genai python-dotenv

2. Environment Variables: Create a .env file in the project3 directory:

   GOOGLE_API_KEY=your_google_api_key
   BRD_USERNAME=your_proxy_username
   BRD_PASSWORD=your_proxy_password

3. Redis: Ensure Redis is running: redis-server

Usage

To run the system in production mode with full monitoring, you will need three terminal windows.

Terminal 1: The Monitor

Start the real-time event monitor. This will listen for updates from the worker.

cd project3
python monitor.py

Terminal 2: The Worker

Start the Celery worker. Crucial: You must use the -E flag to enable event sending so the monitor can see what's happening.

cd project3
celery -A main worker -E --loglevel=info

Terminal 3: The Dispatcher

Run the main script to dispatch a task to the worker.

cd project3
python main.py

• This will send a "gaming laptop" scraping task to the worker.
• Watch Terminal 1 to see the task progress in real-time.
• Watch Terminal 2 to see the worker logs.

Local Testing (Optional)

If you want to run everything in a single script without a worker (for debugging), use the --local flag:

python main.py --local

Note: The real-time monitor (monitor.py) will not capture events in this mode.

Output

• CSV Data: ebay_data_<search_term>.csv
• Dashboard: ebay_dashboard_<search_term>.html
• Logs: market_monitor.log