Better logging

[jemrobinson]

✅ Checklist

• I have searched open and closed issues for duplicates.
• This is a request for a new feature in the Data Safe Haven or an upgrade to an existing feature.
• The feature is still missing in the latest version.
• I have read through the documentation.
• This isn't an open-ended question (open a discussion if it is).

🍓 Suggested change

We should ensure that our logging is improved by:

• logging more things
• alerting when things are not as expected

🚂 How could this be done?

Could consolidate in LogAnalyticsWorkspace and send to e.g. GreyLog? Not sure on the best way to alert

Task list

• VMs not connecting to Log Analytics Workspace #1931
• Add logging for container instances #2295
• Add firewall logs #2307
• Add blob storage logs #2309
• File share logging #2318

[JimMadge]

Related to #1931

[JimMadge]

Azure monitor data source

Log sources are:

• Azure resources (most azure resources, I think this is basic data like SKU, state, ...)
• Entra ID (audit data, authentication events etc.)
• App insights (data from app services like webapps)
• Virtual machines (e.g. syslog)
• Kubernetes Clusters
• Custom sources, using the REST API

I think our working assumption was that using the Azure logging resources would mean we are easily able to ingest logs from Azure resources, whereas using an alternative log manager would involve lots of work to ingest logs.

The number of things which Azure monitor can ingest natively is fairly limited though so there might not be such a big advantage to using it.

[craddm]

Additional logs to investigate:

• Network events
  • Firewall logs
  • NSG logs
• Service logs
  • Containers
    • apricot
    • guac user sync
    • Nexus allowlist
    • Gitea
    • Hedgedoc
• Event logs
  • Data ingress
  • Data egress
• Authentication logs
  • From VM (PAM, ldap)
  • Entra (would need to work cross tenant)

[craddm]

Logging from container instances https://learn.microsoft.com/en-us/azure/container-instances/monitor-azure-container-instances

Log Analytics workspaces provide a centralized location for storing and querying log data not only from Azure resources, but also on-premises resources and resources in other clouds. Azure Container Instances includes built-in support for sending logs and event data to Azure Monitor logs.

To send container group log and event data to Azure Monitor logs, specify an existing Log Analytics workspace ID and workspace key when configuring a container group.

Seems like this must be configured when deploying a container group.

In pulumi-azure-native https://www.pulumi.com/registry/packages/azure-native/api-docs/containerinstance/containergroup/#loganalytics

[craddm]

• Can't enable LogAnalytics/Diagnostic settings directly on the Entra tenants we use, since these features require a subscription on the tenant
• Simplest solution probably just to direct people to the right place, rather than try to find a way to ingest these logs into a central Log Analytics Workspace
• Logs available on Entra
  • User sign-in
  • Audit
  • Provisioning

[JimMadge]

firewall monitoring

[craddm]

One complexity is that network flow logs require a Network Watcher.
There is only one network watcher per region per subscription, which by default gets automatically created when any virtual network gets created within the subscription.

https://learn.microsoft.com/en-us/azure/network-watcher/network-watcher-overview

This means we need to import the existing network watcher resource rather than create it and manage it ourselves (and assume, if there is not one already, that it'll be created during our deployment). May be an issue if users have already created their own network watcher with a non-default name etc

[jemrobinson]

It's fine to postpone network flow logs for a future release if it's not a quick fix. Better to have some things working (and know which ones need to be added later) than have nothing ready until it all is.

[JimMadge]

Looks like there isn't anything useful we can get out of the virtual networks/NSGs.

Metrics are not supported by log analytics, and @craddm noted an issue with the flow logs (not actually sure what those are).

[jemrobinson]

Looks like virtual network flow logs are the new thing (NSG flow logs deprecated from 2027). Not sure if they are any easier to use though.

[JimMadge]

It seems like most 'Azure native' resources collect logs and metrics, which you can collect in a log analytics workspace using a Diagnostic setting (in pulumi).

[craddm]

Looks like virtual network flow logs are the new thing (NSG flow logs deprecated from 2027). Not sure if they are any easier to use though.

Also won't be able to create new NSG flow logs from June 30 2025

Assume the pulumi flow logs are virtual network flow logs rather than NSG flow logs, but it's not explicitly stated https://www.pulumi.com/registry/packages/azure-native/api-docs/network/flowlog/

[craddm]

Looks like there isn't anything useful we can get out of the virtual networks/NSGs.

Metrics are not supported by log analytics, and @craddm noted an issue with the flow logs (not actually sure what those are).

flow logs should keep track of things like network traffic (source and destinations) and how the NSG rules are being applied

[JimMadge]

flow logs should keep track of things like network traffic (source and destinations) and how the NSG rules are being applied

I think that is what we would want. But it sounds difficult because it has to use the subscription wide network watcher?

If so, I think it would be OK to leave it for now.

[craddm]

flow logs should keep track of things like network traffic (source and destinations) and how the NSG rules are being applied

I think that is what we would want. But it sounds difficult because it has to use the subscription wide network watcher?

If so, I think it would be OK to leave it for now.

Yes, I'm not sure if there's a way to get Pulumi to import it if it exists or create it if it doesn't - I think it's a problem like this Pulumi issue