Log Rotatation in Linux

housekeeping logs

logrotate is a Linux utility that is used to automatically manage and rotate log files. It is used to archive, compress and delete old log files, to keep them from consuming too much disk space.

The logrotate utility works by reading a configuration file that specifies which log files to rotate, how often to rotate them, and how many rotated versions to keep. The configuration file also specifies options such as whether to compress the rotated log files, when to rotate them, and what to do with log files that are removed or missing.

When logrotate runs, it reads the configuration file and performs the specified actions on the log files.

To create a logrotate configuration for Cloudera services, you can follow these steps:

1. Create a new file in the /etc/logrotate.d/ directory with a name that corresponds to the Cloudera service you want to rotate the logs for. For example, /etc/logrotate.d/hbase for HBase logs.

2. Add the following configuration in the newly created file:

/var/log/hbase/hbase*.log.out {
    size 100M
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
}

In a logrotate configuration file, the following options are used to specify how the log files should be rotated:

size 100M: This option tells logrotate to rotate the log files when they reach a size of 100MB. Once a log file reaches this size, logrotate will rotate it, creating a new file with the same name and continuing to write new log entries to the new file.

rotate 7: This option tells logrotate to keep 7 rotated log files. Once the eighth log file is created, the first one will be deleted.

compress: This option tells logrotate to compress the rotated log files using gzip compression. This will save disk space but it will increase the CPU usage when it compresses the files.

delaycompress: This option tells logrotate to delay the compression of the most recent rotated log file. This allows you to view the most recent rotated log file without having to decompress it first.

missingok: This option tells logrotate to not produce an error if the log file is missing. This is useful if the log file is removed by another process.

notifempty: This option tells logrotate to not rotate the log file if it is empty.

In summary, this logrotate configuration will rotate the log files when they reach 100MB, keep 7 rotated log files, compress the rotated log files, delay the compression of the most recent rotated log file, delete old rotated log files to free up disk space and not produce an error if the log file is missing, and not rotate the log file if it is empty.

logrotate can be scheduled to run automatically at specific intervals, typically by using the cron daemon via crontab -e

0 0 * * * logrotate /etc/logrotate.d/cloudera-hadoop

This will run the logrotate command daily at midnight.

Test the configuration by running logrotate -d /etc/logrotate.d/hbase to check the results without actually performing the rotation.

Logrotate is a powerful tool that is widely used in Linux servers and it can be used to keep track of various services and applications log files.