Logging/Alert System

[rwinstanley1]

📜 Description

Linked to issue alan-turing-institute/data-safe-haven-production#40 as part of requirements to have a system

• 6.1.1
  • system to identify data security and protection breaches
  • to work with the DSP Toolkit incident reporting tool?
• 6.1.4
  • potentially an alert to those responsible for data security built into the system?

requirements don't specify that for either of these it has to be a technical solution, but, was discussed previously that there was a desire for a technical logging solution by the team so think it's worth looking into. could be a combination of technical and manual processes for March (especially for the 'alert' part of requirement).

🍓 Desired behaviour

To have an (ideally) technical solution for logging and alerts.

📄 Tasks

Include specific tasks (if any) in the order in which they need to be done.

• Identify potential logging options that could be deployed in existing infrastructure.
• Find out if there is a way to automate system alerts at least partly, if not what would manual process look like?

[JimMadge]

Identify potential logging options that could be deployed in existing infrastructure.

Currently we use Azure Log Analytics to query logs in Azure Monitor(?). Feedback is that querying data is not very straightforward. How hard is it to integrate non-Azure things into this system, for example generic Linux logs in /var/log or logs in a container?

Alternatives worthconsidering,

• Graylog
• Logstash
• journalcheck
• logwatch

Find out if there is a way to automate system alerts at least partly, if not what would manual process look like?

A fully automated system seems unlikely, how can we automatically determine what should raise concern?

With one of the dashboard type solutions we could configure searches/dashboards to help filter useful information (e.g. authentication logs, antivirus logs). These should also let us create alerts for events which obviously merit investigation (e.g. antivirus positive results).

[JimMadge]

There is a question here of what needs to be logged. For example, we could (presumably) extract text and journald logs from the containers running internal webapps (GitLab, HackMD). I expect this would take more effort than getting similar data from the host OS on a VM and it feels like this information is less important. Due to the isolation model of the safe haven, if there is malicious activity on one of these webapps then the safe haven has already be compromised. (But maybe we should be more worried about people breaking out?)

Similarly, I expect we don't want to keep logs of users command history.

[JimMadge]

Graylog

• Ingest logs from journald, syslog, Windows eventlog, text, JSON via HTTP and others
• Uses elastic search
• Ansible role! and docker image options
• Alerts
• It looks like we might need to use Azure Monitor to interface with Azure logs (via Log Analytics)