Use the trial mode for ASM authorization policies to reduce risks in production environments - Alibaba Cloud Service Mesh

A misconfigured authorization policy can unintentionally deny valid traffic or allow unauthorized access. In trial mode, an ASM authorization policy evaluates requests and logs the outcome without enforcing rules. This lets you verify that a policy is correct and reliable without affecting your production environment.

Prerequisites

A cluster has been added to an ASM instance that is version 1.14 or later. For instructions on how to upgrade an instance, see Upgrade an ASM instance.
A sample namespace named foo is created, and automatic injection is enabled. For more information, see Manage global namespaces.

Background information

Authorization policies in Service Mesh (ASM) provide access control for workloads at the namespace and workload levels. Because authorization policies are a traffic management feature, a misconfiguration can unexpectedly block normal business traffic or allow traffic that should be blocked. This poses a significant challenge for mesh administrators. To address this, ASM provides a trial mode (called dry-run mode in Istio) for authorization policies. When you enable trial mode, the policy only logs its evaluation outcome instead of blocking or allowing traffic. Mesh administrators can use these logs to verify whether the policy behaves as expected. After you tune the policy so that it performs as expected, disable trial mode to enforce it.

This example deploys two test applications, sleep and httpbin. The overall workflow is as follows: From the sleep application, use curl to access the httpbin application and verify connectivity. Then, configure an authorization policy in ASM to deny specific requests and enable its trial mode. Next, send requests that match the policy's deny conditions. Because trial mode is enabled, the requests are not denied, but the sidecar logs the trial run. After you confirm from the logs that the policy works as intended, disable trial mode to enforce the policy.

Step 1: Deploy test applications and verify connectivity

Create a file named sleep.yaml with the following content:

Expand to view sleep.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
  namespace: foo
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  namespace: foo
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: curlimages/curl
        command: ["/bin/sleep", "3650d"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true
---

Connect to the cluster by using kubectl and deploy the sleep application in the foo namespace.
```
kubectl apply -f sleep.yaml -n foo
```

Create a file named httpbin.yaml with the following content:

Expand to view httpbin.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
  namespace: foo
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  namespace: foo
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  namespace: foo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80

Deploy the httpbin application in the foo namespace.
```
kubectl apply -f httpbin.yaml -n foo
```

Test the connectivity between the sleep and httpbin applications.

for i in {1..20}; do kubectl exec "$(kubectl get pod -l app=sleep -n foo -o jsonpath={.items..metadata.name})" -c sleep -n foo -- curl http://httpbin.foo:8000/headers -s -o /dev/null -w "%{http_code}\n"; done

Expected output:

The 200 status code returned for all 20 requests indicates that the sleep pod can connect to the httpbin application.

Step 2: Create a policy and enable trial mode

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Mesh Security Center > AuthorizationPolicy, and then click Create.
On the Create page, configure the following parameters, and then click Create.
- Name: test
- Policy Type: Deny
- Select Enable Trial Mode.
- Click the Workload tab. Set Namespace to foo, set the scope to Service, set Workload to httpbin, and set the matching label to app:httpbin.
- In the Request Matching Rules section, turn on the HTTP Paths switch, and enter /headers.

Step 3: Observe the policy effect

Send requests from the sleep application to the httpbin application again.

for i in {1..20}; do kubectl exec "$(kubectl get pod -l app=sleep -n foo -o jsonpath={.items..metadata.name})" -c sleep -n foo -- curl http://httpbin.foo:8000/headers -s -o /dev/null -w "%{http_code}\n"; done

Expected output:

The requests are still successful because the authorization policy is in trial mode.

Set the Role-Based Access Control (RBAC) logging level of the httpbin application's sidecar to debug.

kubectl exec "$(kubectl get pod -l app=httpbin -n foo -o jsonpath={.items..metadata.name})" -c istio-proxy -n foo -- curl -X POST 127.0.0.1:15000/logging?rbac=debug

Expected output:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0active loggers:
  ...
  rbac: debug
  ...
100  1028    0  1028    0     0  1003k      0 --:--:-- --:--:-- --:--:-- 1003k

Filter the sidecar logs of the httpbin application for the authorization policy's dry-run output.

kubectl logs "$(kubectl -n foo -l app=httpbin get pods -o jsonpath={.items..metadata.name})" -c istio-proxy -n foo | grep "shadow denied"

The output contains dry-run logs similar to the following:

2023-12-20T03:58:47.107915Z     debug   envoy rbac external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:130        shadow denied, matched policy ns[foo]-policy[test]-rule[0]     thread=32
2023-12-20T03:58:48.800098Z     debug   envoy rbac external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:130        shadow denied, matched policy ns[foo]-policy[test]-rule[0]     thread=33
2023-12-20T03:58:50.420179Z     debug   envoy rbac external/envoy/source/extensions/filters/http/rbac/rbac_filter.cc:130        shadow denied, matched policy ns[foo]-policy[test]-rule[0]     thread=32

Step 4: Disable trial mode

After verifying from the logs that the authorization policy behaves as expected, disable trial mode to enforce the policy.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Mesh Security Center > AuthorizationPolicy.
On the Authorization Policy page, find the authorization policy that you created in Step 2, turn off the switch in the Trial Mode column. Then click OK in the Confirm dialog.

Send the requests again.

for i in {1..20}; do kubectl exec "$(kubectl get pod -l app=sleep -n foo -o jsonpath={.items..metadata.name})" -c sleep -n foo -- curl http://httpbin.foo:8000/headers -s -o /dev/null -w "%{http_code}\n"; done

Expected output:

The output shows that the requests are denied with a 403 status code, which indicates that the authorization policy is enforced.

After the test is complete, restore the sidecar's logging level to warning.

kubectl exec "$(kubectl get pod -l app=httpbin -n foo -o jsonpath={.items..metadata.name})" -c istio-proxy -n foo -- curl -X POST 127.0.0.1:15000/logging?rbac=warning

Expected output:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0active loggers:
  ...
  rbac: warning
  ...
100  1028    0  1028    0     0  1003k      0 --:--:-- --:--:-- --:--:-- 1003k

Alibaba Cloud Service Mesh:Use trial mode for ASM authorization policies

Prerequisites

Background information

Step 1: Deploy test applications and verify connectivity

Step 2: Create a policy and enable trial mode

Step 3: Observe the policy effect

Step 4: Disable trial mode

Related documents