Elasticsearch:Use Elastic Stack to implement integrated monitoring for containers in Kubernetes

2a. Set environment variables

In both kind: DaemonSet and kind: Deployment, set the following variables to match your cluster:

env:
- name: ELASTICSEARCH_HOST
  value: es-cn-nif23p3mo0065****.elasticsearch.aliyuncs.com
- name: ELASTICSEARCH_PORT
  value: "9200"
- name: ELASTICSEARCH_USERNAME
  value: elastic
- name: ELASTICSEARCH_PASSWORD
  value: ****
- name: KIBANA_HOST
  value: es-cn-nif23p3mo0065****-kibana.internal.elasticsearch.aliyuncs.com
- name: KIBANA_PORT
  value: "5601"

2b. Add spec.selector to DaemonSet and Deployment

## kind: DaemonSet
spec:
  selector:
    matchLabels:
       k8s-app: metricbeat
  template:
    metadata:
      labels:
        k8s-app: metricbeat

## kind: Deployment
spec:
  selector:
    matchLabels:
       k8s-app: metricbeat
  template:
    metadata:
      labels:
        k8s-app: metricbeat

2c. Configure Elasticsearch and Kibana output

In both metricbeat-daemonset-config and metricbeat-deployment-config:

output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
setup.kibana:
      host: "https://${KIBANA_HOST}:${KIBANA_PORT}"
setup.dashboards.enabled: true

2d. Configure DaemonSet modules

The metricbeat-daemonset-modules ConfigMap defines which metrics each node collects. The System module collects CPU, load, memory, network, and filesystem metrics. The Kubernetes module collects node, pod, container, and volume data.

For available metricsets, see System module and Kubernetes module.

apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  system.yml: |-
    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        - core
        - diskio
        - socket
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5      # include top 5 processes by CPU
        by_memory: 5   # include top 5 processes by memory

    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - node
        - system
        - pod
        - container
        - volume
      period: 10s
      host: ${NODE_NAME}
      hosts: ["localhost:10255"]

2e. Configure Deployment modules

The metricbeat-deployment-modules ConfigMap collects kube-state-metrics and Kubernetes events.

apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  # This module requires `kube-state-metrics` up and running under `kube-system` namespace
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_pod
        - state_container
      period: 10s
      host: ${NODE_NAME}
      hosts: ["kube-state-metrics.arms-prom:8080"]
      # Uncomment this to get k8s events:
    - module: kubernetes
      metricsets:
        - event

2f. Configure role-based access control (RBAC)

Create a ClusterRole and bind it to Metricbeat's ServiceAccount so Metricbeat can read ACK cluster resources:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: metricbeat
subjects:
- kind: ServiceAccount
  name: metricbeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: metricbeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: metricbeat
  labels:
    k8s-app: metricbeat
rules:
- apiGroups: [""]
  resources:
  - nodes
  - namespaces
  - events
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
  resources:
  - replicasets
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources:
  - statefulsets
  - deployments
  verbs: ["get", "list", "watch"]
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metricbeat
  namespace: kube-system
  labels:
    k8s-app: metricbeat
---

2a. Set environment variables in kind: DaemonSet

env:
- name: ELASTICSEARCH_HOST
  value: es-cn-nif23p3mo0065****.elasticsearch.aliyuncs.com
- name: ELASTICSEARCH_PORT
  value: "9200"
- name: ELASTICSEARCH_USERNAME
  value: elastic
- name: ELASTICSEARCH_PASSWORD
  value: ****
- name: KIBANA_HOST
  value: es-cn-nif23p3mo0065****-kibana.internal.elasticsearch.aliyuncs.com
- name: KIBANA_PORT
  value: "5601"
- name: NODE_NAME
  valueFrom:
     fieldRef:
        fieldPath: spec.nodeName

2b. Configure Elasticsearch and Kibana output

In the filebeat-config ConfigMap, the environment variables defined in the DaemonSet are referenced here. Update the output section:

output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
setup.kibana:
      host: "https://${KIBANA_HOST}:${KIBANA_PORT}"

2c. Configure log collection

The filebeat-inputs ConfigMap tells Filebeat to collect logs from all Docker containers and enrich them with Kubernetes metadata:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-inputs
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  kubernetes.yml: |-
    - type: docker
      containers.ids:
      - "*"
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            in_cluster: true
---

1a. Create the APM Server configuration

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: apm-deployment-config
  namespace: kube-system
  labels:
    k8s-app: apmserver
data:
  apm-server.yml: |-
      apm-server.host: "0.0.0.0:8200"
      output.elasticsearch:
         hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
         username: ${ELASTICSEARCH_USERNAME}
         password: ${ELASTICSEARCH_PASSWORD}
      setup.kibana:
         host: "https://${KIBANA_HOST}:${KIBANA_PORT}"

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: apmserver
  namespace: kube-system
  labels:
    k8s-app: apmserver
spec:
  selector:
    matchLabels:
       k8s-app: apmserver
  template:
    metadata:
      labels:
        k8s-app: apmserver
    spec:
      serviceAccountName: apmserver
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: apmserver
        image: docker.elastic.co/apm/apm-server:6.8.14
        args: [
          "-c", "/etc/apm-server.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: es-cn-oew20i5h90006****.elasticsearch.aliyuncs.com
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          value: ****
        - name: KIBANA_HOST
          value: es-cn-oew20i5h90006****-kibana.internal.elasticsearch.aliyuncs.com
        - name: KIBANA_PORT
          value: "5601"
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 50Mi
          requests:
            cpu: 20m
            memory: 30Mi
        volumeMounts:
        - name: config
          mountPath: /etc/apm-server.yml
          readOnly: true
          subPath: apm-server.yml
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: apm-deployment-config
---
apiVersion: v1
kind: Service
metadata:
  name: apmserver
  namespace: kube-system
  labels:
    k8s-app: apmserver
spec:
  clusterIP: None
  ports:
  - name: http-metrics
    port: 8200
    targetPort: 8200
  selector:
    k8s-app: apmserver
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: apmserver
  namespace: kube-system
  labels:
    k8s-app: apmserver
---

1b. Deploy APM Server and verify

kubectl apply -f apm-server.yml
kubectl get pods -n kube-system

2a. Create a Dockerfile named myapply

FROM frolvlad/alpine-oraclejdk8
MAINTAINER peterwanghao.com
VOLUME /tmp
ADD spring-boot-0.0.1-SNAPSHOT.jar spring-boot-0.0.1-SNAPSHOT.jar
ADD elastic-apm-agent-1.21.0.jar elastic-apm-agent-1.21.0.jar
EXPOSE 8080
ENTRYPOINT ["java","-javaagent:/elastic-apm-agent-1.21.0.jar","-Delastic.apm.service_name=my-application","-Delastic.apm.server_url=http://apmserver:8200","-Delastic.apm.application_packages=com.example","-jar","/spring-boot-0.0.1-SNAPSHOT.jar"]

The ENTRYPOINT passes the following Java agent parameters:

2b. Build the Docker image

docker build -t myapply .

2c. Load the image onto other nodes

Copy the built image to the other nodes in your ACK cluster.

2d. Create the pod deployment file my-application.yaml

---
apiVersion: v1
kind: Pod
metadata:
  name: my-apply
  namespace: kube-system
  labels:
    app: my-apply
spec:
  containers:
    - name: my-apply
      image: myapply:latest
      ports:
        - containerPort: 8080
      imagePullPolicy: Never
---
apiVersion: v1
kind: Service
metadata:
  name: my-apply
  namespace: kube-system
  labels:
    app: my-apply
spec:
  type: NodePort
  ports:
  - name: http-metrics
    port: 8080
    nodePort: 30000
  selector:
    app: my-apply

---

2e. Deploy the application

kubectl apply -f my-application.yaml

2f. Verify the APM agent

After all pod resources are running, send a test request over port 30000:

curl http://10.7.XX.XX:30000

Replace 10.7.XX.XX with the IP address of a node in your ACK cluster. A successful response confirms the APM agent is running and sending data to APM Server.