Container Compute Service:Rotate general ACS application logs with a sidecar

How it works

The solution in this topic is designed as follows:

• For a small number of workloads (Part ① in the figure), you can configure a logrotate sidecar for each application to rotate logs.

• For many workloads (Part ② in the figure), you can use a SidecarSet to inject a logrotate sidecar into workloads in batches. You can specify a scope at the cluster or namespace level.

Configure a logrotate sidecar for an application

The following example shows how to use the native sidecar mechanism to deploy a logrotate container. This mechanism uses an init container that is configured with restartPolicy: Always. For ACS clusters that run Kubernetes 1.28 or earlier, you can use the ACS-enhanced environment variable __IS_SIDECAR=true to mark a regular container as a sidecar. This lets you manage its lifecycle in the same way as a native sidecar container. For more information, see Configure the startup and shutdown order of sidecar containers.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-logrotate
  labels:
    app: test
spec:
  replicas: 1 
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      initContainers:
        - name: logrotate
          env:
            - name: CRON_EXPR
              value: "5 * * * *"
            - name: LOGROTATE_LOGFILES
              value: "/var/log/*/*.log"
            - name: LOGROTATE_FILENUM
              value: "5"
            - name: LOGROTATE_FILESIZE
              value: "10M"
            - name: __IGNORE_RESOURCE__
              value: "true"
            - name: __IGNORE_READY__
              value: "true"
          command: [ "sh", "/start.sh" ]
          image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/logrotate:v1.1
          volumeMounts:
            - mountPath: /var/log
              name: shared-log
            - mountPath: /var/lib
              name: logrotate-state
          restartPolicy: Always
          resources:
            limits:
               cpu: 0.25
               memory: 0.5Gi
      containers:
        - name: busybox
          image: mirrors-ssl.aliyuncs.com/busybox:latest
          command: [ "sh", "-c" ]
          args:
            - |
              mkdir /var/log/busybox
              while true; do
                TIMESTAMP=$(date "+%Y-%m-%d %H:%M:%S")
                LOG_LEVEL=$(shuf -n1 -e INFO WARNING ERROR)
                APP_NAME="busybox"
                HOSTNAME=$(hostname)
                MESSAGE=$(shuf -n1 -e "Success" "Timeout" "Database error")
                echo "$TIMESTAMP $HOSTNAME [$LOG_LEVEL] $APP_NAME: $MESSAGE" >> /var/log/busybox/busybox.log
                sleep 1
              done
          volumeMounts:
            - mountPath: /var/log
              name: shared-log
      volumes:
        ## Collect stdout logs.
        - emptyDir:
          name: shared-log
        - emptyDir:
          name: logrotate-state

The following table describes the main environment variables.

Configure logrotate sidecars for applications in batches

OpenKruise's SidecarSet lets you manage sidecar containers by automatically injecting and upgrading them independently. In this example, a SidecarSet injects a logrotate sidecar into all pods that have the kruise.io/inject-logrotate: "true" label. This SidecarSet also enables the shareVolumePolicy (shareVolumePolicy.type=enabled). When the shareVolumePolicy is enabled, all the pod's volumeMounts are automatically mounted into the sidecar. If all injected pods share a common volume name, you can also declare that name in the SidecarSet.

apiVersion: apps.kruise.io/v1alpha1
kind: SidecarSet
metadata:
  name: logrotate-sidecarset
spec:
  selector:
    matchLabels:
      kruise.io/inject-logrotate: "true"
  shareVolumePolicy:
    type: enabled
  updateStrategy: 
    type: NotUpdate
  initContainers:
    - name: logrotate
      env:
        - name: CRON_EXPR
          value: "5 * * * *"
        - name: LOGROTATE_LOGFILES
          value: "/var/log/*/*.log"
        - name: LOGROTATE_FILENUM
          value: "5"
        - name: LOGROTATE_FILESIZE
          value: "10M"
        - name: __IGNORE_RESOURCE__
          value: "true"
      command: [ "sh", "/start.sh" ]
      image: registry-cn-hangzhou.ack.aliyuncs.com/ack-demo/logrotate:v1.1
      volumeMounts:
        - mountPath: /var/lib/
          name: logrotate-state
      restartPolicy: Always
      resources:
        limits:
           cpu: 0.25
           memory: 0.5Gi
  volumes:
    - name: logrotate-state
      emptyDir: {}

Related information

The following Dockerfile and script are used to build the logrotate image that is described in this topic. You can modify the Dockerfile or script as needed.

• Dockerfile:

  FROM registry-cn-hangzhou.ack.aliyuncs.com/dev/alpine:3.20-update
  
  RUN apk --update add bash logrotate
  
  ADD start.sh /start.sh
  
  CMD ["/start.sh"]

• start.sh file:

  #!/bin/sh
  
  LOGROTATE_LOGFILES="${LOGROTATE_LOGFILES:?Files for rotating must be given}"
  LOGROTATE_FILESIZE="${LOGROTATE_FILESIZE:-10M}"
  LOGROTATE_FILENUM="${LOGROTATE_FILENUM:-5}"
  
  cat > /etc/logrotate.conf << EOF
  ${LOGROTATE_LOGFILES}
  {
    size ${LOGROTATE_FILESIZE}
    missingok
    notifempty
    copytruncate
    rotate ${LOGROTATE_FILENUM}
  }
  EOF
  
  if [ -z "$CRON_EXPR" ]; then
    CRON_EXPR="0 6 * * *"
    echo "CRON_EXPR environment variable is not set. Set to default: $CRON_EXPR"
  else
    echo "CRON_EXPR environment variable set to $CRON_EXPR"
  fi
  
  echo "$CRON_EXPR	/usr/sbin/logrotate -v /etc/logrotate.conf" >> /etc/crontabs/root
  
  (crond -f) & CRONPID=$!
  trap "kill $CRONPID; wait $CRONPID" SIGINT SIGTERM
  wait $CRONPID