在Kubernetes叢集中,apiserver的審計日誌可以協助叢集管理員記錄或追溯不同使用者的日常操作,是叢集安全營運中重要的環節。本文旨在協助您瞭解阿里雲Kubernetes叢集apiserver審計日誌的相關配置,以及如何通過SLSLog Service收集和搜尋指定的日誌內容。

配置介紹

當前建立Kubernetes叢集會預設開啟apiserver審計功能,相關的參數配置功能如下:
说明 登入到Master節點,apiserver設定檔的目錄是 /etc/kubernetes/manifests/kube-apiserver.yaml
配置 說明
audit-log-maxbackup 審計日誌最大分區儲存10個記錄檔
audit-log-maxsize 單個審計日誌最大size為100MB
audit-log-path 審計日誌輸出路徑為/var/log/kubernetes/kubernetes.audit
audit-log-maxage 審計日誌最多儲存期為7天
audit-policy-file 審計日誌配置策略檔案,檔案路徑為:/etc/kubernetes/audit-policy.yml
登入Master節點機器,審計配置策略檔案的目錄是 /etc/kubernetes/audit-policy.yml,內容如下:
apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # The following requests were manually identified as high-volume and low-risk,
  # so drop them.
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
      - group: "" # core
        resources: ["endpoints", "services"]
  - level: None
    users: ["system:unsecured"]
    namespaces: ["kube-system"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["configmaps"]
  - level: None
    users: ["kubelet"] # legacy kubelet identity
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes"]
  - level: None
    userGroups: ["system:nodes"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes"]
  - level: None
    users:
      - system:kube-controller-manager
      - system:kube-scheduler
      - system:serviceaccount:kube-system:endpoint-controller
    verbs: ["get", "update"]
    namespaces: ["kube-system"]
    resources:
      - group: "" # core
        resources: ["endpoints"]
  - level: None
    users: ["system:apiserver"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["namespaces"]
  # Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
      - /healthz*
      - /version
      - /swagger*
  # Don't log events requests.
  - level: None
    resources:
      - group: "" # core
        resources: ["events"]
  # Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
  # so only log at the Metadata level.
  - level: Metadata
    resources:
      - group: "" # core
        resources: ["secrets", "configmaps"]
      - group: authentication.k8s.io
        resources: ["tokenreviews"]
  # Get repsonses can be large; skip them.
  - level: Request
    verbs: ["get", "list", "watch"]
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # Default level for known APIs
  - level: RequestResponse
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # Default level for all other requests.
  - level: Metadatak8s.io"
      - group: "storage.k8s.io"
  # Default level for all other requests.
  - level: Metadata
说明
  • 在收到請求後不立即記錄日誌,當返回體header發送後才開始記錄。
  • 對於大量冗餘的kube-proxy watch請求,kubelet和system:nodes對於node的get請求,kube組件在kube-system下對於endpoint的操作,以及apiserver對於namespaces的get請求等不作審計。
  • 對於/healthz*/version*, /swagger*等唯讀url不作審計。
  • 對於可能包含敏感資訊或二進位檔案的secrets, configmaps, tokenreviews介面的日誌等級設為metadata,該level只記錄請求事件的使用者、時間戳記、請求資源和動作,而不包含請求體和返回體。
  • 對於一些如authenticatioin、rbac、certificates、autoscaling、storage等敏感介面,根據讀寫記錄相應的請求體和返回體。

採集和檢索

在使用Kube-apiserver審計日誌之前,請確保在建立叢集的配置中開啟了SLSLog Service,並成功建立對應的日誌Project和Logstore。

  1. 登入 Log Service管理主控台
  2. 單擊左側導覽列中Project管理,選擇建立叢集時設定的日誌Project,單擊名稱進入日誌Project頁面。


  3. 在Project詳情頁中,預設進入日誌庫頁面,查看名為audit-${clustered}的日誌庫(logstore),單擊右側的查詢,叢集對應的審計日誌會收集在該日誌庫中。
    说明 在您建立過程中,指定的日誌Project中會自動添加一個名稱為audit-${clustered}的日誌庫。


  4. 當叢集管理員需要關注某一子帳號的行為時,可以輸入相應子帳號id,追溯其操作。


  5. 當叢集管理員關注某一具體資來源物件時,可以輸入相應的資源名稱,檢索時間段內的指定操作。


支援第三方日誌解決

您可以在叢集Master各節點,在/var/log/kubernetes/kubernetes.audit 路徑下找到審計日誌的源檔案。該檔案是標準的json格式,您可以在部署叢集時選擇不使用阿里雲的Log Service,根據需要對接其他的日誌解決方案,完成相關審計日誌的採集和檢索。