Audit logging

A payroll service runs pay for a few thousand small businesses. On payday, Dana, a barista at one café, gets nothing: the pay went to a bank account Dana has never heard of. The café owner says nobody there changed it. Support opens the logs and finds one line from three weeks ago, PUT /employees/88 200 41ms, with no user, no old value and no new value.

That line was written to debug the service, not to answer a question about responsibility. An audit log (or audit trail) is a separate record of security-relevant actions, kept so that someone can later reconstruct who did what, in what order, and prove it. The difference shows in every property:

Audit trails are collected for different purposes than other logs and belong apart from them, and plain statement logging isn't one either: a list of every operation is not enough, because an auditor has to be able to find the statements that matter. Metrics, traces and debug logs live in Observability: metrics, logs & traces; this page is about the audit trail.

An audit record must establish what type of event occurred, when, where, its source, its outcome, and the identity of every person or system involved. For the payroll service, the bank change would have looked like this:

With that record, support answers in a minute: a support account, from an unfamiliar address, changed the account ending 3317 to one ending 9054. The request_id ties the record to the request's other logs and traces. The event is a named business action, not an HTTP route, so a search for every bank change across all customers is one query.

Which events deserve a record is a decision made at design time, in proportion to the risk. Some events should always be logged:

Just as important is what stays out. Some data should never go into logs as it is, but be removed, masked, hashed or encrypted first: passwords, session identifiers, access tokens, connection strings, encryption keys, bank account and card data, and sensitive personal data. That's why the record above keeps only the last four digits. An audit log full of secrets becomes the next thing to steal (Secrets management).

The input has to be cleaned too. Input containing a line break can add what looks like a second, false record to a text log, covering an attacker's tracks; this is log injection, and the fix is to encode carriage returns, line feeds and delimiters, or to write structured records like the one above. Clocks on every server must be synchronized, or the order of events across machines can't be trusted (Time, clocks & ordering).

The simple design saves the new bank account, then sends an audit event to a logging service. If the process crashes between the two steps, or the logging service is down, the change happened and no record of it exists. Reverse the order and a crash leaves a record of a change that never happened. Either way the trail lies, at the moment it matters most.

The fix is the same one used for any event that must match a database write: insert the audit record into a table in the same database transaction as the change, and let a separate relay copy it to the audit store afterwards. This is the transactional outbox, explained in Distributed transactions: 2PC & sagas:

Some teams capture the same information from the database's change stream instead (Change data capture). That records every row change, even ones made by hand outside the application, but a row change knows less than the application: which person, from which address, through which support ticket. The two are often combined.

Then decide what happens when audit logging fails. The usual control is an alert to named people within a set time, plus an action chosen in advance: overwrite the oldest records, stop generating records, or shut the system down. The right answer differs by action. In general a logging failure shouldn't stop the application; the payroll service makes high-risk changes the exception: with the record in the same transaction, a bank change that can't be audited doesn't happen at all.

Suppose sam.lee's account belonged to an insider who also had write access to the audit table. Deleting one row would erase the evidence. The standard controls address this in layers:

• A separate system: store audit records on a different system from the one being audited, so compromising one doesn't compromise both.
• Separate privileges: only a small set of people may manage audit logging, distinct from those who administer the system being audited.
• Cryptographic integrity: signed hashes, so anyone with the public key can check the records.

A hash chain makes edits detectable. Each record stores a hash of the previous record's hash together with its own content, so every record depends on all the ones before it. Changing an old record changes its hash, and the next record's stored value no longer matches:

A chain alone has one gap: cutting off the newest records leaves a shorter chain that still checks out. So the latest hash is signed and published somewhere the log's writer can't rewrite. AWS CloudTrail does exactly this: it hashes every log file with SHA-256, and every hour delivers a digest file listing the last hour's files and their hashes, signed with a private key. Each digest also contains the previous digest's signature, so the digests form a chain of their own.

Validation then shows whether a file was changed or deleted after delivery, and can also confirm that no log files were delivered during a period, which is how a missing hour is told apart from a quiet one. Chaining detects tampering; it doesn't prevent it. Prevention comes from WORM (write once, read many) storage, such as Amazon S3 Object Lock:

Café 412's pay was diverted three weeks after the change. If the payroll service had kept its audit records for 14 days, the trail would already be gone. Investigations start late: this change was found on payday, three weeks after it was made. Cloud providers' own defaults show how different the answer can be by log type:

The cloud defaults are a floor, not a policy. Google writes admin activity logs always, and they can't be disabled; data access logs, which record who read what, are off by default for most services because of their volume. AWS keeps 90 days of management events automatically; anything longer needs a trail you create, and data events, such as reads of individual stored objects, are not logged by default even then. The data you most need in a breach is often the data you had to switch on.

The 2023 intrusion into Microsoft's hosted email showed why that matters. After learning of the breach, the US Commerce Department, one of the victims, asked Microsoft for logs, and Microsoft switched on the premium logging. Commerce still could not see past activity: the logs captured data only from the moment the advanced logging was enabled. A log turned on after the incident starts recording after the evidence.

Retention cuts both ways. Logs must not be destroyed before the required retention period, and must not be kept beyond it. Audit records name people and their actions, so they carry the same privacy duties as other personal data (Privacy, residency & retention). Old records can move to cheaper, slower storage tiers (Data lifecycle: tiering, TTL & compression), as long as they can still be searched within the time an investigation needs.

Reading the audit log is itself an action worth auditing. All access to logs should be recorded and monitored, and the privilege to read them restricted and reviewed periodically. Support staff who can search every customer's bank changes hold a lot of personal data in one query box.

The same 2023 intrusion was first noticed by the US State Department. Its security team had written a custom alert rule over Microsoft's mailbox-access log, a log it could read only because it had bought the premium license that includes enhanced logging. On June 16, 2023, the rule fired; the team investigated, judged the activity malicious, and contacted Microsoft that day.

Few, if any, other victims had the same logging, and buying it would not have been enough on its own: few organizations analyzed that very large log in detail. The lesson drawn afterward was that cloud providers should give customers audit logs of all access to their data by default, at no extra charge, kept for at least six months.

For the payroll service, detection means turning the audit store into a feed for rules that a person responds to:

OWASP's 2025 Top 10 ranks security logging and alerting failures ninth and lists the usual gaps: auditable events such as failed sign-ins or high-value transactions not logged, logs not protected from tampering, logs only stored locally, logs not monitored, and alert thresholds that nobody responds to in time. Every one of them would have left the café's question unanswered.

For the payroll service, and most systems that act on other people's money or data:

Audit logging and event sourcing overlap but aren't the same: an event-sourced system stores its state as a history of events, which gives a natural record of what changed (Event sourcing & CQRS). It still needs the who, from where and with what authority, and the history still needs protecting from the people it describes.