What are Logging and Monitoring Best Practices?
Logging refers to the practice of logging errors and changes or to the application logs that are collected. The purpose of logging is to create an ongoing record of application events. Log files can be used to review any event within a system, including failures and state transformations. Logging helps developers to optimize application performance, quickly diagnose and troubleshoot issues, and enhance a system’s overall security. Monitoring is an umbrella term referring to application performance monitoring (APM). APM is the process of instrumenting an application to collect, aggregate, and analyze metrics to better evaluate the use of the system by gauging availability, response time, memory usage, bandwidth, and CPU time consumption.1 Monitoring is the live review of application and security logs using various forms of automation.
The following is a list of security logging and security best practices:
Define your goals. Determine why logging and monitoring solutions are needed and what the logging and monitoring goals are. Some reasons for needing logging solutions may include compliance, regulations, laws, and incident response requirements.
Identify what needs to be logged and monitored. Based on your goals, determine what metadata needs to be captured and what events need to be logged. Some examples of metadata and events to be logged (and why) may include:
- PII/PHI transactions to be HIPAA compliant
- Financial transactions to be PCI DSS compliant
- Authentication attempts to a server (successful and failed logins, password changes)
- Commands executed on a server
- Queries (especially DML queries) executed on a database2
Define attributes of logs which should include, at a minimum:
- An actor (who: username, IP address)
- An action (what: read/write on which resource)
- A time (when: timestamp)
- A location (where: geolocation, browser, code script name)
Identify assets and events that need to be monitored. Identify which systems/applications should be monitored and what level of monitoring is required. Classify your data and systems according to the organization’s statutory, regulatory, or contractual requirements.
Design secure logging and monitoring systems. Examples of security processes include redaction, role-based access controls, log integrity checks, encryption, least-privilege log source configuration, and log sanitation.
Establish policies. Define organization-wide policies, protocols, and procedures for logging.
Optimize log retention strategy. Application requirements will determine the log retention period. When developing a plan for log storage, consider cost-effectiveness for both frequently and infrequently accessed logs, balancing needs for historical data against costs, and optimal log rotation strategies.
1 AppDynamics, 2023, “What’s the Difference? Logging vs Monitoring”
2 Rana, 2021, “Top seven logging and monitoring best practices”