Define tasks in yaml files and chain them asynchronously. This is a powerfull way to glue your tools together in a simple and efficient way. By using unique identifiers for each record, it is possible to monitor for new records or changes.
Each task is called a pipe because you can pipe your tasks together to create whole workflows.
- each pipe is defined in a
yml
file, takes an (filtered) input and returns an output (string or json) - each input to a pipe is passed to a system command, each output is then returned and persisted
- each output specifies an
ident
, so new changes can be detected
Example: The pipe with the name http_detect takes domains as input, passes them to httpx and persists the results:
name: http_detect
input:
table: domains
cmd: echo ${.input.asset} | httpx -silent -json -response-in-json
output:
table: services
ident: ${.outputJson.url}|${index .outputJson "status-code"}
data:
service: http
webserver: ${.outputJson.webserver}
url: ${.outputJson.url}
title: ${.outputJson.title}
status: ${index .outputJson "status-code"}
interval: 24h
timeout: 10m
worker: 10
- Scalable, all tasks are queued (over redis), each pipe can define how many workers are started
- Perodic/timed scanning (how often should each row be re-queued)
- Task uniqueness guaranteed (same task/pipe + data cant be executed at the same time twice)
- IP blacklist checks (private IPs and special networks)
- Pipe filtering through javascript
- Exclude assets by setting the
exclude
field
- Install redis and postgres
go build
A .env
file is used for basic config stuff:
DATABASE_URL="database=pipers"
SLACK_WEBHOOK="https://hooks.slack.com/services/XXX/YYY"
The database schema is created automatically.
There are three modes which can be run:
This will load all pipes from ./resources/pipes/*.yml
and periodically schedule
all due tasks.
./pipers
Using the -worker
flag will load all pipes from ./resources/pipes/*.yml
and run
multiple workers for each pipe. The amount of each workers for each pipe is defined
in the pipe itself via the worker
field.
./pipers -worker
Using this mode no database will be used and data (asset) is loaded from stdin. This allows to debug pipes and print the result.
./pipers -noDb
-single pipe/to/load.yml
will load only a single pipe.- To test a pipe in production,
debug: true
can be set in the yaml file so results are not saved in the database.
Let's install pipers, setup a simple workflow and add some initial data. Our example workflow should:
- enumerate domains
- detect http services
- discover content
This workflow will then run continuously and monitor for new assets/services/paths.
Let's begin with the first step in our chain, enumerating assets. We want to get all
"scope" domains from the domains table and pass it to a simple API retrieving subdomains.
For this we will define a simple filter, pass it to curl
and put the parsed result in
a new table output
.
The interval (how often this pipe should be run for each row) is set to 12h, while the task will be cancelled after 1m. Additionally, an alert message can be defined which is saved with the alert and used for notification hooks.
./resources/pipes/domains_crobat.yml
name: domains_crobat
input:
table: assets
filter:
scope: true
cmd: curl https://sonar.omnisint.io/subdomains/${.input.asset} -s | jq -r -c '.[]'
output:
table: assets
ident: ${.output}
asset: ${.output}
interval: 12h
timeout: 1m
worker: 1
alert_msg: New domain '${.output}'
Our second step in our chain is to detect http services on all domains found in the previous step(s).
Our ident
is defined with the url and the status code. This results in new entries
as soon as a new service is found or the status code changes.
./resources/pipes/http_detect.yml
name: http_detect
# load data from assets table
input:
table: assets
# pipe asset into httpx
cmd: echo ${.input.asset} | httpx -json -response-in-json -ports=80,81,300,443,3128,8080,8081
filter:
# example filter to remove cloudflare blocked entries
waf: | var data = JSON.parse(output)
data.response.includes('Sorry, you have been blocked')
output:
table: services
ident: ${.outputJson.url}|${index .outputJson "status-code"}
data:
service: http
webserver: ${.outputJson.webserver}
url: ${.outputJson.url}
title: ${.outputJson.title}
response: ${.outputJson.serverResponse}
status: ${index .outputJson "status-code"}
# alert message contains the url, title and server (from headers)
alert_msg: ${.outputJson.url} - TITLE '${.outputJson.title}' - SERVER ${.outputJson.webserver}
# run this pipe every 12h on every row from input table
interval: 12h
# cancel task after httpx runs for more than 10m
timeout: 10m
# run 10 workers for this pipe
worker: 10
./resources/pipes/http_content.yml
name: http_content
input:
table: services
filter:
service: http
# pipe asset into httpx
cmd: ./discover_content ${.input.asset}
output:
table: content
ident: ${.outputJson.url}
data:
url: ${.outputJson.url}
title: ${.outputJson.title}
response: ${.outputJson.serverResponse}
status: ${index .outputJson "status-code"}
interval: 12h
timeout: 10m
worker: 10
First start the scheduler:
./pipers
and then also start all workers:
./pipers -worker
.
The recommended way to do this is to use a DB browser. The following SQL queries will add three scope domains for a target with the name example:
INSERT INTO domains (ident, asset, target, data) values ('example.com','example.com', 'example', '{"scope": true}');
INSERT INTO domains (ident, asset, target, data) values ('example2.com','example2.com', 'example', '{"scope": true}');
INSERT INTO domains (ident, asset, target, data) values ('example3.com','example3.com', 'example', '{"scope": true}');
The previously started scheduler will pick this entries up automatically and queue them for processing.
It is possible to exclude assets (and subdomains) by setting the exclude flag in the domains
table to true
.
Example to exclude foo.bar.com
(and all subdomains of this domain):
INSERT INTO domains (ident, asset, target, exclude) values ('example.com','example.com', 'example', true);