Automating Log Cleanup for GBase 8a MPP Clusters: A Step-by-Step Guide Using Shell Script and Crontab

Efficient log management is crucial for maintaining the stability and performance of GBase 8a MPP Clusters. Over time, system logs can accumulate, consuming significant storage space and potentially impacting system operations. Implementing an automated log cleanup process ensures optimal storage utilization and simplifies maintenance tasks. This article gives a method to set up automatic log cleanup using a shell script and crontab.

1. Developing the Cleanup Script

The first step involves creating a shell script, cleanlogs.sh, which defines the maximum allowable log file size (ceiling) and specifies the log files to monitor. The script checks each log file’s size and clears its contents if it exceeds the defined threshold. Additionally, it removes loader log directories older than a day.

Below is a sample cleanlogs.sh script:

#!/bin/bash

# Load environment variables
if [ -f /home/gbase/.gbase_profile ]; then
    . /home/gbase/.gbase_profile
else
    echo "Error: Environment profile not found."
    exit 1
fi

# Default maximum log file size (10GB)
default_ceiling=10000000000

# Set ceiling value from command-line argument or use default
ceiling=${1:-$default_ceiling}

# Define log file for script actions
script_log="/var/log/cleanlogs_script.log"

# Function to log messages
log_message() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$script_log"
}

# Function to clear log if it exceeds the ceiling
clear_log_if_exceeds() {
    local log_file=$1
    if [ -f "$log_file" ]; then
        local size=$(stat -c %s "$log_file")
        if [ "$size" -gt "$ceiling" ]; then
            : > "$log_file"
            if [ $? -eq 0 ]; then
                log_message "Cleared $log_file (size: $size bytes)"
            else
                log_message "Failed to clear $log_file"
            fi
        fi
    else
        log_message "Log file $log_file does not exist."
    fi
}

# Define log file paths
gc_log_home="$GCLUSTER_BASE/log/gcluster"
gn_log_home="$GBASE_BASE/log/gbase"
loader_log_home="$GCLUSTER_BASE/log/gcluster/loader_logs"
gcware_home="$GCWARE_BASE/log/"

# Clear specific logs
clear_log_if_exceeds "$gc_log_home/express.log"
clear_log_if_exceeds "$gn_log_home/express.log"
clear_log_if_exceeds "$gc_log_home/system.log"
clear_log_if_exceeds "$gn_log_home/system.log"
clear_log_if_exceeds "$gcware_home/gcware.log"

# Remove loader log directories older than a day
if [ -d "$loader_log_home" ]; then
    find "$loader_log_home" -type d -mtime +0 -exec rm -rf {} + 2>/dev/null
    if [ $? -eq 0 ]; then
        log_message "Removed loader log directories older than a day in $loader_log_home."
    else
        log_message "Failed to remove some loader log directories in $loader_log_home."
    fi
else
    log_message "Loader log directory $loader_log_home does not exist."
fi

2. Setting Script Permissions

Ensure the script is executable by the gbase user:

chmod +x /opt/gbasetools/cleanlogs.sh

3. Scheduling the Script with crontab

To automate the execution of the cleanup script, configure crontab to run it at a desired interval. For instance, to execute it at midnight on the first day of each month:

1. Edit the crontab for the gbase user:

   crontab -e

2. Add the following line to schedule the script:

   0 0 1 * * /opt/gbasetools/cleanlogs.sh

3. Save and exit the editor.

4. Verifying the crontab Configuration

To confirm the scheduled task:

crontab -l

This setup will help maintain log sizes within acceptable limits, ensuring smoother operations for your GBase 8a MPP Cluster.

Note: Adjust the ceiling value and script paths as needed to fit your specific environment.

Learn More:

How to Set Up a Self-Hosted PostgreSQL Database with SSL Support

7 Best Open-Source AI Coding Tools Every Developer Should Know

Build Your Own AI-Powered Search Engine with SWIRL

Top 8 Open Source MLOps Tools for Production