Audit Log System

An audit log system that collects details about any action that was performed on a resource. For ex, user X deleted database Y at some timestamp, using api token X (masked) some agent made a request to the platform.

SOC2 and compliance’s push companies to have controls in place to approve changes, and log the changes. One might never need the logs but when the day comes where you need the logs, not having the logs is never good.

Most audit logging records the actions done to a thing.

• User A created Object B.
• User C modified Object D.
• User E deleted Object F.

Usually the audit log will record before/after or just the specific change/field that was modified.

Let us consider developing audit log system using sql triggers. Every time a table was updated, a record was added to the audit log table who, what, when, where (system, Api, version etc..), change made. This approach ensures the data cannot be updated by mistake without creating an audit record, while it bring forth multiple challenges. Yes, it is very granular. Still it is harder to record recognised context ie user was doing X. it does not record query on the database, may be important depending on the system

We considered above to explain that many application do have audit logs. Does application capture what the company needs to know? May be these logs do not help. When one tells team to capture logs without any context, the log system may not be suitable for the purpose required in future.

Here are some questions to enable developers to capture based on the broad general context that applies to audit log system. (check with their stakeholders).

“Who did the change?” For any kind of shared resource that multiple users can modify (e.g. dashboards), there is possibility that the resource gets changed in unexpected ways and users won’t know who to talk to resolve the issue. Especially in larger organisations, once this happens a few times, there is a new standard requirement for all tools.

“What happened?” Often security breaches involve multiple hops across multiple systems, services, accounts. The ability to retrospectively inspect the activity on the platform may provide ability for customers to stitch together how a breach happened and what they need to do in response. This information is useful from a reliability perspective. A customer takes some action in your platform ultimately that triggers an outage for the customer. When that customer does a detailed post-mortem of the incident, timestamped history of the actions taken is super helpful.

• Effectively capture answers to these questions related to action. What they touched, Where, i.e. which system was touched, when the change was effective.
• Effectively capture these details about events , who (someone) created / updated / deleted something, whose permissions were changed (RBAC only, much harder for ABAC)

Here are few things that can be captured by log for actions and events?

• Date time of the modification
• Object modified
• Before values
• After values
• User
• Reason for the action (generally not that useful, could be required nonetheless)
• Authorization (maybe optional for systems but for actions that require 4 eyes, may be needed)
• hash value (if necessary record a hash value that cannot be regenerated by an outside source and can ensure the log record was not modified)

Compliance Compliance regimes vary on exact details. Still all compliance requires a common requirement in regulated environments, the ability to assert that controls are working as expected. Detailed audit logs help to meet that requirement. One example can be the customer has a compliance requirement for least privilege access to a certain dataset, a full audit logs of all access attempts to that dataset helps to support the assertion that those access controls are working properly.

Filling in gaps Audit logs can help fill feature gaps or address more needs that may not make sense to implement as dedicated features. One common example is cost attribution. Larger organisations want the ability to break up a single large vendor bill across teams/orgs/budget/etc. This is complicated exercise, involving business logic specific to the customer. Audit logs can provide the raw data to customer who can process internally, whatever they need to do to attribute costs as they want.

Enterprise customers make it a standard requirement for all vendors. while there is nothing special about how your platform triggering this, this can be enterprise way to try to encode knowledge and improve the odds of not repeating past mistakes withing their organisation.

Where to log? Some regulation may require that logs are recorded outside of the system data storage. Some regulators may require the audit logs to be stored off premise and also some assurance is provided that logs cannot be modified and cannot be lost (some form of redundancy)

How long to store audit logs? Recommend to make the storage duration configurable. There can be regulations that demand logs can never be deleted. Generally most regulation require the audit logs be stored from 1-7 years.

How to view audit logs? Likely a simple reporting mechanism that suits audience is required. While developers may be satisfied with grep command on large file, auditors may require simple view mechanism to access/view logs

Where to write to audit logs? The blog earlier discussed first approach to write logs based on db trigger and explained that idea may be good for database and can lead to a performance hit and cost hit. Some application perform logging on flat files, which would require additional work. A second approach is to send audit data to audit logging service, working in the background and write to a system with latency or interruptions to communication. A third approach may be an event driven architecture. Is this expensive one? Here it is possible to have an audit service, listening for data modification events.

Some pointers While display your logs internally can be viewed as good, it is better to assume that the real goal is a valid auditing system(s)

• Should you attempt to build logging system yourself? No. No. Don’t even attempt to build your own auditing system. if you are a regulated entity, possibly you already have this.
• Better to feed out logs to a configurable api or syslog service – anyone who really needs logs will feed the logs to this system, that will deal with retention, security, etc.
• Keep data in logging for what’s “reasonable” for your support/billing teams