Container Service for Kubernetes:Amankan operasi kluster dengan auditing server API

Ikhtisar

Menampilkan semua event kluster dan detail event prioritas tinggi, termasuk operasi Pengguna RAM, akses Internet, eksekusi perintah, penghapusan resource, akses Secret, dan kerentanan CVE.

Ikhtisar operasi

Memberikan statistik operasi create, update, delete, dan read pada:

• Resource komputasi: Deployment, StatefulSet, CronJob, DaemonSet, Job, dan Pod

• Resource jaringan: Service dan Ingress

• Resource penyimpanan: ConfigMap, Secret, dan PersistentVolumeClaim (PVC)

• Resource kontrol akses: Role, ClusterRole, RoleBinding, dan ClusterRoleBinding

Detail operasi

Menampilkan operasi untuk tipe resource tertentu. Pilih tipe resource untuk dikueri. Laporan ini menampilkan jumlah total, distribusi namespace, laju keberhasilan, dan tren dari waktu ke waktu.

Untuk mengkueri resource CRD atau resource yang tidak tercantum, masukkan nama resource jamaknya. Misalnya, untuk mengkueri AliyunLogConfig, masukkan AliyunLogConfigs.

Kerentanan CVE

Menampilkan kerentanan CVE Kubernetes. Filter berdasarkan ID Pengguna RAM. Lihat \[CVE Securities\] Perbaikan kerentanan CVE untuk remediasi.

Contoh peringatan 1: Perintah dieksekusi di kontainer

Memberikan peringatan ketika pengguna mengeksekusi perintah di kontainer. Setiap peringatan mencakup kontainer, perintah, pengguna, ID event, waktu, dan alamat IP sumber.

Pernyataan kueri contoh:

verb : create and objectRef.subresource:exec and stage:  ResponseStarted | SELECT auditID as "Event ID", date_format(from_unixtime(__time__), '%Y-%m-%d %T' ) as "Time",  regexp_extract("requestURI", '([^\?]*)/exec\?.*', 1)as "Resource",  regexp_extract("requestURI", '\?(.*)', 1)as "Command" ,"responseStatus.code" as "Status code",
 CASE
 WHEN "user.username" != 'kubernetes-admin' then "user.username"
 WHEN "user.username" = 'kubernetes-admin' and regexp_like("annotations.authorization.k8s.io/reason", 'RoleBinding') then regexp_extract("annotations.authorization.k8s.io/reason", ' to User "(\w+)"', 1)
 ELSE 'kubernetes-admin' END
 as "User account",
CASE WHEN json_array_length(sourceIPs) = 1 then json_format(json_array_get(sourceIPs, 0)) ELSE  sourceIPs END
as "Source IP address" order by "Time" desc  limit 10000

Ekspresi kondisi: Event =~ ".*"

Contoh peringatan 2: Gagal akses Internet dari server API

Memantau permintaan Internet arah keluar dari kluster Anda. Memicu peringatan ketika jumlah permintaan mencapai 10 dan laju kegagalan melebihi 50%.

Pernyataan kueri contoh:

* | select ip as "Source IP address", total as "Number of times of Internet access", round(rate * 100, 2) as "Failure rate in percentage", failCount as "Number of times of illegal access", CASE when security_check_ip(ip) = 1 then 'yes' else 'no' end  as "Whether the IP address is risky",  ip_to_country(ip) as "Country", ip_to_province(ip) as "Province", ip_to_city(ip) as "City", ip_to_provider(ip) as "ISP" from (select CASE WHEN json_array_length(sourceIPs) = 1 then json_format(json_array_get(sourceIPs, 0)) ELSE  sourceIPs END
as ip, count(1) as total,
sum(CASE WHEN "responseStatus.code" < 400 then 0
ELSE 1 END) * 1.0 / count(1) as rate,
count_if("responseStatus.code" = 403) as failCount
from log  group by ip limit 10000) where ip_to_domain(ip) != 'intranet' and ip not LIKE '%,%' and not try(is_subnet_of('7.0.0.0/8', ip)) ORDER by "Number of times of Internet access" desc limit 10000

Ekspresi kondisi: Source IP address =~ ".*"

Ubah proyek SLS

Untuk memigrasi log audit ke proyek SLS berbeda:

1. Login ke Konsol ACK. Di panel navigasi kiri, klik Clusters.

2. Di halaman Clusters, klik nama kluster. Di panel kiri, pilih Security > Cluster Auditing.

3. Klik Change Log Service Project dan ikuti petunjuknya.

Nonaktifkan auditing kluster

1. Login ke Konsol ACK. Di panel navigasi kiri, klik Clusters.

2. Di halaman Clusters, klik nama kluster. Di panel kiri, pilih Security > Cluster Auditing.

3. Klik Disable Cluster Auditing.

Gunakan layanan log pihak ketiga (hanya untuk kluster khusus ACK)

SLS direkomendasikan sebagai penyimpanan log audit. Untuk menggunakan layanan log pihak ketiga, lewati SLS saat pembuatan kluster dan integrasikan layanan pihak ketiga tersebut. Log audit mentah tersedia di node master pada /var/log/kubernetes/kubernetes.audit dalam format JSON.

Kebijakan audit

Kebijakan audit menentukan event mana yang dikumpulkan dan pada tingkat detail apa. Terdapat empat tingkat audit:

Kebijakan audit dimuat dari /etc/kubernetes/audit-policy.yml di node master (melalui flag --audit-policy-file). Kebijakan ACK default:

apiVersion: audit.k8s.io/v1 # Required. Set to audit.k8s.io/v1 if the Kubernetes version of the cluster is 1.24 or later and set to audit.k8s.io/v1beta1 if the Kubernetes version of the cluster is earlier than 1.24.
kind: Policy
# No need to generate audit events at the RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # The following types of requests are frequent and the risk of these requests is low. We recommend that you set the rule to None to skip these requests.
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
      - group: "" # core
        resources: ["endpoints", "services"]
  - level: None
    users: ["system:unsecured"]
    namespaces: ["kube-system"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["configmaps"]
  - level: None
    users: ["kubelet"] # legacy kubelet identity
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes"]
  - level: None
    userGroups: ["system:nodes"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["nodes"]
  - level: None
    users:
      - system:kube-controller-manager
      - system:kube-scheduler
      - system:serviceaccount:kube-system:endpoint-controller
    verbs: ["get", "update"]
    namespaces: ["kube-system"]
    resources:
      - group: "" # core
        resources: ["endpoints"]
  - level: None
    users: ["system:apiserver"]
    verbs: ["get"]
    resources:
      - group: "" # core
        resources: ["namespaces"]
  # Set the rule to None for read-only URLs, such as /healthz*, /version*, and /swagger*.
  - level: None
    nonResourceURLs:
      - /healthz*
      - /version
      - /swagger*
  # Set the rule to None for events.
  - level: None
    resources:
      - group: "" # core
        resources: ["events"]
  # Set the rule to Metadata for Secrets, ConfigMaps, and TokenReview API requests that may contain sensitive information or binary files.
  - level: Metadata
    resources:
      - group: "" # core
        resources: ["secrets", "configmaps"]
      - group: authentication.k8s.io
        resources: ["tokenreviews"]
  # Responses may contain large amounts of data. Set the rule to Request so that the response body is not collected.
  - level: Request
    verbs: ["get", "list", "watch"]
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # The rule is set to RequestResponse by default for known Kubernetes API requests to collect the request and response bodies.
  - level: RequestResponse
    resources:
      - group: "" # core
      - group: "admissionregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
  # The rule is set to Metadata by default for other requests.
  - level: Metadata

Perilaku utama:

• Log dihasilkan setelah header respons dikirim, bukan saat permintaan diterima.

• Permintaan watch kube-proxy, permintaan GET kubelet dan system:nodes ke node, operasi endpoint kube-system, dan permintaan GET server API ke namespace dikecualikan.

• Operasi baca (get, list, watch) untuk API authentication, rbac, certificates, autoscaling, dan storage dicatat pada tingkat Request. Operasi tulis pada API-API ini dicatat pada tingkat RequestResponse.

• Secret dan ConfigMap hanya dicatat pada tingkat Metadata, sehingga kontennya tidak pernah ditulis ke log audit.

Backend audit

Event audit ditulis sebagai file JSON ke sistem file lokal di node master. Konfigurasi server API di /etc/kubernetes/manifests/kube-apiserver.yaml mengontrol perilaku backend dengan flag-flag berikut: