Container Service for Kubernetes:Install HistoryServer in ACK

Prerequisites

• After HistoryServer is enabled, the PostStartHook of the pod created by RayCluster will be overwritten. If you want to use PostStartHook, add the following script. This script writes the Ray nodeid to the /tmp/ray/init.log file for the sidecar of HistoryServer Collector to read and use.

GetNodeId(){
  while true;
  do
    nodeid=$(ps -ef | grep raylet | grep node_id | grep -v grep | grep -oP '(?<=--node_id=)[^ ]*' | tr -d '\n')
    if [ -n "$nodeid" ]; then
      echo "$(date) raylet started: \"$(ps -ef | grep raylet | grep node_id | grep -v grep | grep -oP '(?<=--node_id=)[^ ]*')\" => ${nodeid}" >> /tmp/ray/init.log
      echo $nodeid > /tmp/ray/alibabacloud_raylet_node_id
      break
    else
      echo "$(date) raylet not start >> /tmp/ray/init.log"
      sleep 1
    fi
  done
}
GetNodeId
    

• After HistoryServer is enabled, the ServiceAccount of the pod created by RayCluster will be replaced. The new naming rule for ServiceAccount is ServiceAccountPrefix-RayClusterName. If you want to use a custom ServiceAccount, make sure its configuration is consistent with the generation rule.

• The installed Kuberay version must be later than 1.2.1.5. For more information, see Install KubeRay in ACK.

1. Enable the RRSA feature for the cluster

1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

2. On the Clusters page, find the cluster that you want to manage and click its name. In the left-side pane, click Cluster Information.

3. Enable the RAM Roles for Service Accounts (RRSA) OpenID Connect (OIDC) feature for the cluster. On the cluster page, choose Cluster Information > Basic Information > Security and Auditing, and click Enable next to the RRSA OIDC parameter. For more information about the operation, see Enable during cluster creation.

2. Create an RRSA role

1. Create an RRSA role.

   Log on to the RAM console as a RAM administrator.In the left-side navigation pane, choose Identities > Roles. On the Roles page, click Create Role and select Identity Provider as the trusted principal type.

   1. Add a principal.

      Identity Provider Type: Select the cluster for which RRSA OIDC is enabled.

      

   2. Add conditions.

      Associate a specific ServiceAccount. After the RRSA role is created, associate the RRSA role with a specific ServiceAccount in the cluster.

      Condition	Value
      Key	oidc:sub
      Operator	StringLike
      Value	system:serviceaccount:*:ray-historyserver* Note An asterisk (*) represents a wildcard character, and <ray-historyserver> corresponds to the serviceAccountPrefix in subsequent operations. ray-historyserver is a custom part that must be the same as the ServiceAccountPrefix when installing HistoryServer.

      Note

      If you are using a custom service account, you must click Add statement to add two principals in the same RRSA role, and add the following conditions for each principal:

      • Principal 1: Identity Provider

        Condition	Value
        Key	oidc:sub
        Operator	StringEquals
        Value	system:serviceaccount:kuberay:ray-historyserver

      • Principal 2: Identity Provider

        Condition	Value
        Key	oidc:sub
        Operator	StringLike
        Value	system:serviceaccount:*:rhs*

      The two service accounts are system:serviceaccount:kuberay:ray-historyserver and system:serviceaccount:*:rhs*, where rhs is a customizable part.

2. Add permissions to the RRSA role.

   1. Add the AliyunARMSReadOnlyAccess permission to the role for read-only access to Application Real-Time Monitoring Service (ARMS).

      

   2. Add the AliyunOSSFullAccess permission to the role for managing OSS. The steps are the same as above.

      Important

      This topic grants the role full OSS permissions. In actual scenarios, we recommend that you use precise authorization to control the scope of permissions.

3. Create an OAuth application

1. Create and configure an OAuth enterprise application.

   Important

   For more information about how to connect to HistoryServer over the Internet, see Configure Internet access.

   

   Note

   The callback address is: http://localhost:8080/auth/callback. localhost:8080 is the domain name of HistoryServer, which corresponds to CallbackServiceName in subsequent operations and must be specified when installing HistoryServer. /auth/callback is a fixed path suffix.

2. Configure the OAuth application.

   1. Add OAuth scopes.

      • aliuid

        Obtain the Alibaba Cloud UID (RAM User or Alibaba Account ID).

      • profile

        Obtain the profile such as the username of the user. (The main account access login obtains the login name, and the RAM user login obtains the user principal name and display name.)

      

   2. Create and save the OAuth application Secret.

      

      Important

      Record your Application ID and AppSecretValue, which will be used when creating a Secret in the kuberay namespace in subsequent operations.

3. Connect to the cluster and create a Secret in the kuberay namespace.

   For more information about how to connect to a Container Service for Kubernetes (ACK) cluster, see Connect to a cluster.

   kubectl create ns kuberay
   kubectl create secret -n kuberay generic webapp-secret --from-literal=webapp-id="yours-AppID" --from-literal=webapp-secret=yours-AppSecretValue

   Parameter	Description
   webapp-secret	The name of the Secret to be created, which can be customized.
   webapp-id	The OAuth application ID.
   webapp-secret	The AppSecretValue of the OAuth key.

4. Configure KubeRay parameters

1. Install the Kuberay component.

   For more information, see Install KubeRay.

2. Configure Kuberay-Operator parameters.

   Parameter	Description
   Enable HistoryServer	Select to enable HistoryServer.
   CallbackServiceName	The callback domain name for HistoryServer OAuth authentication, which must be consistent with the domain name in the callback address of the OAuth application. For example, if the OAuth configuration is http://xx.com/oauth/callback, then set this parameter to xx.com.
   CloudRoleName	The name of the RRSA role associated with HistoryServer.
   OSSBucket	The name of the OSS bucket used by HistoryServer.
   OSSEndPoint	The endpoint of the OSS bucket used by HistoryServer.
   OSSHistoryServerRootDir	The directory where HistoryServer stores logs and metadata.
   OSSRegion	The OSS region used by HistoryServer, such as cn-hangzhou and ap-southeast-1.

5. Create a RayCluster

If you need to enable the HistoryServer feature in a RayCluster, you need to add the ray.alibabacloud.com/enable-historyserver: "true" annotation when submitting the RayCluster. The following is a YAML configuration example.

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  annotations:
    ray.alibabacloud.com/enable-historyserver: "true"
  labels:
    ray.io/cluster: wukun
  generateName: wukun-ray240-
  namespace: default
spec:
  suspend: false
  autoscalerOptions:
    env: []
    envFrom: []
    idleTimeoutSeconds: 60
    imagePullPolicy: Always
    resources:
      limits:
        cpu: 200m
        memory: 200Mi
      requests:
        cpu: 200m
        memory: 200Mi
    securityContext: {}
    upscalingMode: Default
  enableInTreeAutoscaling: false
  headGroupSpec:
    rayStartParams:
      dashboard-host: 0.0.0.0
      num-cpus: "0"
    serviceType: ClusterIP
    template:
      metadata:
        labels:
          test: wukun
      spec:
        affinity:
        containers:
        - env:
          image: xxxx
          imagePullPolicy: Always
          name: ray-head
          resources:
            limits:
              cpu: "5"
              memory: 10G
            requests:
              cpu: "1"
              memory: 1G

        tolerations:
        - key: ray
          operator: Equal
          value: cpu
  workerGroupSpecs:
  - groupName: cpu
    maxReplicas: 1000
    minReplicas: 0
    numOfHosts: 1
    rayStartParams: {}
    replicas: 2
    template:
      metadata:
        labels:
          test: wukun
      spec:
        imagePullSecrets:
        containers:
        - env:

          image: xxxx
          imagePullPolicy: Always
          name: ray-worker
          resources:
            limits:
              cpu: "1"
              memory: 1G
            requests:
              cpu: "1"
              memory: 1G
          volumeMounts:
        tolerations:
        - key: ray
          operator: Equal
          value: cpu
        volumes:

6. Connect to HistoryServer

kubectl -n kuberay port-forward svc/ray-history-server --address 0.0.0.0 8080:80

kubectl -n kuberay port-forward svc/ray-history-server --address 0.0.0.0 3000:3000