Monitoring APIs for realtime and historical data

[gsanchietti]

Local reports

Issue #756

Local reports are used to visualize data in real time.
All data are stored in memory, so when the machine is rebooted all historical data get lost.
Also, to minimize disk space usage, some data are aggregated or removed at regular interval.

The following APIs can be used to draw charts and graphs directly inside NethSecurity Web UI.

1. DPI report

APIs are inspired by lightsquid report, see demp

APIs are inside ns.dpireport.

• List of available days: list-days method
• Total daily traffic, divided by client, sorted in descending order: summary method, clients field
• Total daily traffic, divided by destination, sorted in descending order: summary method, host field
• Total daily traffic, divided by protocol (pie chart): summary method, protocol field
• Total daily traffic, divided by application (pie chart): summary method, application field
• Chart of total traffic by hour: summary method, hours field
• For each client/day:
  • Total daily traffic: summary-by-client method, total field
  • Total daily traffic by application: summary-by-client method, application field
  • Total daily traffic by protocol: summary-by-client method, protocol field
  • List of visited hosts: summary-by-client method, host field
  • Traffic by hour: summary-by-client method, hour field

2. Malware: threat shield IP

API named tsip-malware-report inside ns.report.

Only current day data (if available):

• Total number of blocked packets for the day: malware_count field
• Number of blocked packets per hour: malware_by_hour field
• Pie chart: malware by direction (in/out): malware_by_chain field
• Pie chart: malware by category: malware_by_category field

3. Threats: threat shield IP (fail2ban behavior)

API available inside ns.report, named tsip-attack-report

Only current day data, if available

• Number of blocked IPs per day: attack_count field
• List of most blocked IPs: attack_by_ip field
• Graph of blocked IPs per hour: attack_by_hour field

4. Multiwan

API available inside ns.report, named mwan-report

Only current day data, if available:

• Number of online/offline events today: total_online and total_offline fields
• Scatter plot with online/offline events per WAN: events_by_wan field

APIs available inside ns.report, the API is named mwan-report.

5. OpenVPN Road Warrior

APIs available inside ns.report.

Data are read from a sqlite database that is reset only after a reboot.

• List of available days: ovpnrw-list-days method
• List of connected clients per day: ovpnrw-clients-by-day method
• Number of connected clients per hour: ovpnrw-count-clients-by-hour method
• Total client traffic per hour: ovpnrw-bytes-by-hour method
• Traffic per client per hour: ovpnrw-bytes-by-hour-and-user method

Remote reports

Issue #754

Remote reports are implemented inside a NethSecurity Controller.
Data are permanently stored inside a non-volatile database.

See server implementation NethServer/nethsecurity-controller#38

A unit connected to a controller, will send all data to the controller every 15 minutes.

The following APIs have been designed to be called from a cron job, the result must be sent to the controller.
New APIs are implemented inside ns.controller:

• dump-ts-malware
• dump-ts-attacks
• dump-mwan-events
• dump-dpi-stats
• dump-openvpn-connections