Elizabeth
Engineer
Engineer
  • UID625
  • Fans4
  • Follows1
  • Posts68
Reads:1909Replies:0

[Share]Timed deletion of online logs

Created#
More Posted time:Oct 31, 2016 16:38 PM
Application scenarios
I recently took over a project with such a huge amount of logs that the server disk issued alarms at a very high frequency. After checking, I found that not all logs were that huge in size, but only several of the log files using DailyRollingFileAppender. The DailyRollingFileAppender generated a large log file every day and the file format was basically the file name plus the date, such as cndata.log.2016-09-27. These several large log files were not that important, so I considered deleting them automatically once the disk usage reached a certain level through a timed job. This is a temporary solution, saving you some time for deleting the logs. But optimization is still necessary given so many logs.
Usage
Run the sudo crontab -u root -e command on the server and add the following in the edit window:
*/10 * * * * /bin/sh /home/lq/delete_log.sh -c 80 -f hbase.log,cndata.log -p /home/admin/logs -b 1 -l 3

The /home/lq/delete_log.sh is the script path. The -c indicates the disk usage rate to which the log deletion will start; -f indicates the name of the log file to be deleted and multiple log files are separated by commas; -p indicates the path of the log folder; -b indicates the date from which the deletion starts, namely 1 indicates yesterday, 2 indicates the day before yesterday, and the like, while -l indicates the number of days of logs to be deleted.
My configuration is to execute log deletion once every 10 minutes. If the disk usage rate reaches 80%, delete the hbase.log and cndata.log files of yesterday, the day before yesterday and two days before yesterday.
With the sudo crontab -u root -l command, you can check whether the job has been added successfully.
With the sudo /sbin/service crond reload command, you can manually re-load the timed job.
Script code (delete_log.sh)
#!/bin/sh
while getopts "p:c:f:b:l:" arg #The colon following the option indicates this option requires parameters.
do
    case $arg in
        p)
                        #Log path
                        logPath=$OPTARG
                        ;;
        c)
                        #The disk usage rate to which the log deletion will be triggered.
                        alarmCapacity=$OPTARG
                        ;;
        f)
                        #Name of the files to be deleted. Multiple names are separated by commas.
                        deleteLogFileNames=$OPTARG
                        ;;
        b)
                        #Date from when the deletion starts. 1 indicates yesterday, 2 indicates the day before yesterday, and the like.
                        dayBegin=$OPTARG
                        ;;
        l)
                        #Number of days the logs of which will be deleted. If the dayBegin is 1, and the dayLength is 3, it indicates to delete the logs of yesterday, the day before yesterday and two days before yesterday.
                        dayLength=$OPTARG
                        ;;
        ?)             #When there is an unrecognized option, the arg is ?.
                        echo "unkonw argument"
                        exit 1
                ;;
    esac
done
#By default, logs of three days, namely yesterday, the day before yesterday and two days before yesterday, are deleted.
if [ -z "$dayBegin" ]&&[ -z "$dayLength" ];then
        dayBegin=1
        dayLength=3
fi


if [ -z "$logPath" ]||[ -z "$alarmCapacity" ]||[ -z "$deleteLogFileNames" ]||[ -z "$dayBegin" ]||[ -z "$dayLength" ]
then
echo "arg is not valid"
exit 10
fi

#If there is no slash after the input path, add it.
if ! [[ "$logPath" =~ '/$' ]];then
logPath=${logPath}/
fi


dayLength=`expr $dayBegin + $dayLength`

#Separate the name of the file to be deleted.
OLD_IFS="$IFS"
IFS=","
arr=($deleteLogFileNames)
IFS="$OLD_IFS"

#Obtain the disk usage rate
diskCapacity=`df -h / |awk '{print$5}'|grep '^[0-9][0-9]*%$'`
diskCapacity=${diskCapacity%?}

#A temporary directory for storing the log files to be deleted.
tmpDir=${logPath}logbak_`date +"%F"`

#Define the job log for the convenience of checking which files have been deleted.
resultLog=${logPath}/crontab.log
if [ ! -f "$resultLog" ];then
        touch "$resultLog"
fi

#Start to delete logs when the disk usage rate exceeds the user-input alarm value.
if [[ $(echo "$diskCapacity>$alarmCapacity"|bc) -eq 1 ]];then
for((serialNumber=$dayBegin;serialNumber<$dayLength;serialNumber++))
do
serialNumberDayTime=`date -d"$serialNumber day ago" +"%F"`
        for fileName in ${arr[@]}
           do
               preDeleteFile=${logPath}${fileName}.${serialNumberDayTime}
               if [ -f "$preDeleteFile" ];then
                  if [ ! -d "$tmpDir" ];then
                        mkdir $tmpDir
                  fi
                  #Put the logs to be deleted in the temporary directory.
                  mv $preDeleteFile $tmpDir
               fi
           done
done
#Delete logs.
date "+%G-%m-%d %H:%M:%S">>"$resultLog"
ls -l $tmpDir|grep -v total|awk '{print$9}'>>"$resultLog"
rm -rf $tmpDir
fi
Guest