Container Compute Service:Rotate ACS container logs with logrotate sidecar

How it works

The following figure show how this solution works.

• When only a few workloads are deployed in the cluster (① in the preceding figure), you can configure logrotate sidecar for the application to implement log rotation.

• When large numbers of workloads are deployed in the cluster (② in the preceding figure), you can create a SidecarSet and inject logrotate sidecar into workloads within the specified cluster or namespace range.

Configure logrotate sidecar for applications

In the following example, the native sidecar mechanism is used to deploy a logrotate container (an init container configured with restartPolicy: Always). For ACS clusters that run Kubernetes 1.28 or earlier, you can use the ACS enhanced environment variable __IS_SIDECAR=true to mark regular containers in order to manage their lifecycle in the same way as sidecar containers. For more information, see Configure the startup and exit priorities of sidecar containers.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-logrotate
  labels:
    app: test
spec:
  replicas: 1 
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      initContainers:
        - name: logrotate
          env:
            - name: CRON_EXPR
              value: "5 * * * *"
            - name: LOGROTATE_LOGFILES
              value: "/var/log/*/*.log"
            - name: LOGROTATE_FILENUM
              value: "5"
            - name: LOGROTATE_FILESIZE
              value: "10M"
            - name: __IGNORE_RESOURCE__
              value: "true"
            - name: __IGNORE_READY__
              value: "true"
          command: [ "sh", "/start.sh" ]
          image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/logrotate:v1.1
          volumeMounts:
            - mountPath: /var/log
              name: shared-log
            - mountPath: /var/lib
              name: logrotate-state
          restartPolicy: Always
          resources:
            limits:
               cpu: 0.25
               memory: 0.5Gi
      containers:
        - name: busybox
          image: mirrors-ssl.aliyuncs.com/busybox:latest
          command: [ "sh", "-c" ]
          args:
            - |
              mkdir /var/log/busybox
              while true; do
                TIMESTAMP=$(date "+%Y-%m-%d %H:%M:%S")
                LOG_LEVEL=$(shuf -n1 -e INFO WARNING ERROR)
                APP_NAME="busybox"
                HOSTNAME=$(hostname)
                MESSAGE=$(shuf -n1 -e "Success" "Timeout" "Database error")
                echo "$TIMESTAMP $HOSTNAME [$LOG_LEVEL] $APP_NAME: $MESSAGE" >> /var/log/busybox/busybox.log
                sleep 1
              done
          volumeMounts:
            - mountPath: /var/log
              name: shared-log
      volumes:
        ## Collect stdout logs.
        - emptyDir:
          name: shared-log
        - emptyDir:
          name: logrotate-state

The following table describes the key environment variables.

Configure logrotate sidecar for application containers in batches

OpenKruise SidecarSets allow you to manage sidecar containers by automatically injecting and separately upgrading sidecar containers. The SidecarSet in this example injects logrotate sidecar into all pods with the kruise.io/inject-logrotate: "true" label. The SidecarSet also enables the shareVolume policy shareVolumePolicy.type=enabled. This way, all volumeMounts of the pods are automatically mounted to the logrotate sidecar. If all pods to be injected have an agreed volume name, you can also claim the volume in the SidecarSet.

apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: logrotate-sidecarset
spec:
  selector:
    matchLabels:
      kruise.io/inject-logrotate: "true"
  shareVolumePolicy:
    type: enabled
  updateStrategy: 
    type: NotUpdate
  initContainers:
    - name: logrotate
      env:
        - name: CRON_EXPR
          value: "5 * * * *"
        - name: LOGROTATE_LOGFILES
          value: "/var/log/*/*.log"
        - name: LOGROTATE_FILENUM
          value: "5"
        - name: LOGROTATE_FILESIZE
          value: "10M"
        - name: __IGNORE_RESOURCE__
          value: "true"
      command: [ "sh", "/start.sh" ]
      image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/logrotate:v1.1
      volumeMounts:
        - mountPath: /var/lib/
          name: logrotate-state
      restartPolicy: Always
      resources:
        limits:
           cpu: 0.25
           memory: 0.5Gi
  volumes:
    - name: logrotate-state
      emptyDir: {}

References

The following example shows the dockerfile and script used to build the logrotate image in this topic. You can modify the dockerfile or script accordingly.

• The dockerfile:

  FROM registry-cn-hangzhou.ack.aliyuncs.com/dev/alpine:3.20-update
  
  RUN apk --update add bash logrotate
  
  ADD start.sh /start.sh
  
  CMD ["/start.sh"]

• The start.sh script:

  #!/bin/sh
  
  LOGROTATE_LOGFILES="${LOGROTATE_LOGFILES:?Files for rotating must be given}"
  LOGROTATE_FILESIZE="${LOGROTATE_FILESIZE:-10M}"
  LOGROTATE_FILENUM="${LOGROTATE_FILENUM:-5}"
  
  cat > /etc/logrotate.conf << EOF
  ${LOGROTATE_LOGFILES}
  {
    size ${LOGROTATE_FILESIZE}
    missingok
    notifempty
    copytruncate
    rotate ${LOGROTATE_FILENUM}
  }
  EOF
  
  if [ -z "$CRON_EXPR" ]; then
    CRON_EXPR="0 6 * * *"
    echo "CRON_EXPR environment variable is not set. Set to default: $CRON_EXPR"
  else
    echo "CRON_EXPR environment variable set to $CRON_EXPR"
  fi
  
  echo "$CRON_EXPR	/usr/sbin/logrotate -v /etc/logrotate.conf" >> /etc/crontabs/root
  
  (crond -f) & CRONPID=$!
  trap "kill $CRONPID; wait $CRONPID" SIGINT SIGTERM
  wait $CRONPID