Support for multi server configuration?

[squigglefest]

This looks like a great project. In our app we have a web server, sql server, elastic search server and so on. Calls are made directly to these servers, in some cases. Would there be anyway to set this up on multiple servers then replay that traffic? Thanks!

[dosubot[bot]]

🤖

Hey @ryanrockwood, great to see you back! Ready to dive into some more GoReplay magic?

Sure, you can definitely set up GoReplay on multiple servers and replay the traffic. Here's how you can achieve this:

1. First, you need to install GoReplay on each of the servers where you want to capture traffic. This could be your web server, SQL server, and Elastic Search server.

2. Once installed, you can run GoReplay on each server with the --input-raw option to specify the application port you want to capture traffic from, and the --output-http option to specify where you want to replay the traffic. For example, on your web server, you might run:

sudo gor --input-raw :80 --output-http http://staging.com

3. If you want to forward traffic to multiple endpoints, you can use the --output-http option multiple times. For example:

gor --input-tcp :28020 --output-http "http://staging.com"  --output-http "http://dev.com"

4. If you want to split the traffic equally between multiple outputs, you can use the --split-output option:

gor --input-raw :80 --output-http "http://staging.com"  --output-http "http://dev.com" --split-output true

5. If you want to track responses as well as requests, you can use the --input-raw-track-response option:

gor --input-raw :80 --input-raw-track-response --output-http "http://staging.com"

6. If you want to track the original IP addresses of the requests, you can use the --input-raw-realip-header option:

gor --input-raw :80 --input-raw-realip-header "X-Real-IP" --output-http "http://staging.com"

Remember that GoReplay requires root permissions to analyze network traffic, so you may need to run it with sudo. If you want to run it as a non-root user, you can configure it accordingly.

For handling traffic capturing and replaying on multiple servers simultaneously, you can run the GoReplay command on each server where you want to capture traffic. For example, you can run the following command on each web machine:

sudo gor --input-raw :80 --output-http http://staging.com

This command will record and replay traffic from the same machine. If you want to forward the captured traffic to a separate server, you can use the Aggregator-forwarder setup. In this setup, GoReplay on your web machines forward traffic to a GoReplay aggregator instance running on a separate server.

You can also forward traffic to multiple endpoints by specifying multiple --output-http options in the command:

gor --input-tcp :28020 --output-http "http://staging.com"  --output-http "http://dev.com"

By default, GoReplay will send the same traffic to all outputs. However, you can equally split the traffic (round-robin) using the --split-output option:

gor --input-raw :80 --output-http "http://staging.com"  --output-http "http://dev.com" --split-output true

Please note that to analyze network traffic, GoReplay needs permissions which are available only to root users. However, it is possible to configure GoReplay to be run for non-root users.

For more details, you can refer to the Capturing And Replaying Traffic section in the GoReplay documentation.

Sources

• Capturing And Replaying Traffic

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[squigglefest]

Thanks so much for this. OK, so I get that ever server could record their traffic independently but that those individual replay's wouldn't replicate the original traffic but merely one component of the original traffic, right? Based on your response I did notice something about an "aggregator" - which sounds like it might be perfect. Could individual servers all send to an aggregator which would accept all inputs simultaneously and log for later replay? Or would you still be in the same situation of trying to restart replay on 6 different files simultaneously?

[buger]

@squigglefest sorry for the late reply!

Essentially when you point GoReplay to the folder with your files, it reads all files in parallel, to ensure that requests recorded from multiple servers will be replayed in the same original order.

Also, dealing with plain files not that easy when you have multiple instances, so in GoReplay PRO there is built-in support for S3 for both recording and replaying https://github.com/buger/goreplay/wiki/%5BPRO%5D-Using-S3-for-storing-and-replaying-traffic

And if you have a very high traffic environemnt and you need to replay it at speed of like 100kRPS or similar, you can have one GoReplay instance deployed as coordination process, and it will control fleet of GoReplay worker instances, actually doing replay.

Hope it makes sense!