An audit trail (or audit log) is something both intuitive and misleading at the same time. There are many definitions of an audit trail, and all of them give you an idea of what it is about:
A system that traces the detailed transactions relating to any item in an accounting record.
A record of the changes that have been made to a database or file.
An audit trail (also called audit log) is a security-relevant chronological record, set of records, and/or destination and source of records that provide documentary evidence of the sequence of activities that have affected at any time a specific operation, procedure, or event.
An audit trail is a step-by-step record by which accounting or trade data can be traced to its source
The definitions are clear, but they rarely give enough detail on how they apply to a particular IT setup. The Wikipedia article is pretty informative, but I prefer referring to the NIST document about audit trail. This relatively short document from more than 20 years ago covers many the necessary details.
I won’t repeat the NIST document, but I’d like to focus on the practical aspects of an audit trail in order to answer the question in the title – what is an audit trail? In the context of a typical IT setup, the audit trail includes all or some of the following:
- Application-specific audit trail – ideally, each application records business-relevant events. They may be logged in text files or in separate database tables. They allow reconstructing the history much better than the arbitrary noisy logging that is usually in place
- Application logs – this is a broader category as it includes logs that are not necessarily part of the audit trail (e.g. debug messages, exception stacktraces). Nevertheless, they may be useful, especially in case there is no dedicated application-specific audit trail functionality
- Database logs – whether it is logged queries, change data capture or change tracking functionality, or some native audit trail functionality
- Operating system logs – for Linux that would include the /var/log/audit/audit.log (or similar files), /var/log/auth.log. For Windows it would include the Windows Event logs for the Security and System groups.
- Access logs – access logs for web servers can be part of the audit trail especially for internal systems where a source IP address can more easily be mapped to particular users.
- Network logs – network equipment (routers, firewalls) generate a lot of data that may be seen as part of the audit trail (although it may be very noisy)
All of these events can (and should) be collected in a central place where they can be searched, correlated and analyzed.
Ideally, an audit trail event has an actor, an action, details/payload and optionally an entity (type + id) which is being accessed or modified. Having this information in a structured form allows for better analysis and forensics later.
Once the audit trails is collected, it has two other very important properties:
- Availability – is the audit trail available at all times, and how far back in time can it be accessed (also referred to as “retention”). Typically application, system and network logs are kept for shorter periods of time (1 to 3 months), which is far from ideal for an audit trail. Many standards and regulation require higher retention periods of up to 2 years
- Integrity – data integrity is too often ignored. But if you can’t prove the integrity of your logs, both internally and to third parties (auditors, courts), then they are of no use.
Many organizations understand that the integrity of their audit trail is important only after a security incident takes place and they realize they cannot rely on their audit logs. It is better to protect the audit trail before something bad happens. And not just for some abstract reasons like “best practices” or “improved security”. A secure audit trail allows organizations to:
- Identify that something wrong has happened. If the audit trail is not protected, a malicious actor can make it seem like nothing anomalous has taken place
- Prove what happened and who did it. Digital forensics is important for large organizations. Especially in cases where lawsuits are involved.
- Reconstruct the original data. The audit trail is not a replacement for a full backup, but it can allow you to reconstruct the data that was modified and the time it was modified, so that either you know which backup to use, or to avoid restoring form the backup altogether by just relying on the data provided in the audit trail.
- Be compliant. Because of all the reasons stated above, many standards and regulations require a secure audit trail. Simple logs are usually not compliant, even though auditors may turn a blind eye.
The audit trail is an important concept, and has very practical implications. It is a widely researched topic, but in practice many IT setups lack sufficient audit trail capabilities. Partly because the risks are not immediately obvious, and partly because it is a technically challenging task.
(The article is originally published on our corporate blog, but I think it can be interesting here as well, so I copied most of it, leaving out the corporate references and calls to action)