Container Service for Kubernetes:Best practices for OSS read/write splitting

Opção 1: Gravar usando um PV ossfs de leitura/gravação

1. Crie um PV ossfs 1,0 somente leitura para o dataset. As principais alterações de configuração são:

   • Defina accessModes como ReadOnlyMany tanto no PV quanto no PVC. Monte o subcaminho do dataset /tf-train/train/data.

   • Em otherOpts, adicione -o kernel_cache -o max_stat_cache_size=10000 -o umask=022:

     • kernel_cache ativa o cache de leitura em memória.

     • max_stat_cache_size=10000 armazena 10.000 entradas de metadados em cache (~40 MB). Ajuste conforme o tipo de instância e o tamanho do dataset.

     • umask=022 concede permissão de leitura a processos não root no contêiner.

   cat << EOF | kubectl apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: oss-secret
     namespace: default
   stringData:
     akId: "<your-accesskey-id>"
     akSecret: "<your-accesskey-secret>"
   ---
   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: tf-train-pv
     labels:
       alicloud-pvname: tf-train-pv
   spec:
     capacity:
       storage: 10Gi
     accessModes:
       - ReadOnlyMany
     persistentVolumeReclaimPolicy: Retain
     csi:
       driver: ossplugin.csi.alibabacloud.com
       volumeHandle: tf-train-pv
       nodePublishSecretRef:
         name: oss-secret
         namespace: default
       volumeAttributes:
         bucket: "<your-bucket-name>"
         url: "oss-<region>.aliyuncs.com"
         otherOpts: "-o kernel_cache -o max_stat_cache_size=10000 -o umask=022 -o allow_other"
         path: "/tf-train/train/data"
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: tf-train-pvc
   spec:
     accessModes:
     - ReadOnlyMany
     resources:
       requests:
         storage: 10Gi
     selector:
       matchLabels:
         alicloud-pvname: tf-train-pv
   EOF

2. Crie um PV ossfs 1,0 de leitura/gravação para os checkpoints, montando o subcaminho /tf-train/training_logs. O cache de metadados fica desativado (max_stat_cache_size=0) porque gravações sequenciais de checkpoint não se beneficiam de cache.

   cat << EOF | kubectl apply -f -
   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: tf-logging-pv
     labels:
       alicloud-pvname: tf-logging-pv
   spec:
     capacity:
       storage: 10Gi
     accessModes:
       - ReadWriteMany
     persistentVolumeReclaimPolicy: Retain
     csi:
       driver: ossplugin.csi.alibabacloud.com
       volumeHandle: tf-logging-pv
       nodePublishSecretRef:
         name: oss-secret
         namespace: default
       volumeAttributes:
         bucket: "<your-bucket-name>"
         url: "oss-<region>.aliyuncs.com"
         otherOpts: "-o max_stat_cache_size=0 -o allow_other"
         path: "/tf-train/training_logs"
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: tf-logging-pvc
   spec:
     accessModes:
     - ReadWriteMany
     resources:
       requests:
         storage: 10Gi
     selector:
       matchLabels:
         alicloud-pvname: tf-logging-pv
   EOF

3. Implante o pod de treinamento com ambos os PVs montados.

   Nenhuma alteração de código é necessária. Monte ambos os PVs — o PV somente leitura em /mnt/train/data e o PV de leitura/gravação em /mnt/training_logs .

   cat << EOF | kubectl apply -f -
   apiVersion: v1
   kind: Pod
   metadata:
     labels:
       app: tfjob
     name: tf-mnist
     namespace: default
   spec:
     containers:
     - command:
       - sh
       - -c
       - python /app/main.py
       env:
       - name: NVIDIA_VISIBLE_DEVICES
         value: void
       - name: gpus
         value: "0"
       - name: workers
         value: "1"
       - name: TEST_TMPDIR
         value: "/mnt"
       image: registry.cn-beijing.aliyuncs.com/tool-sys/tf-train-demo:rw
       imagePullPolicy: Always
       name: tensorflow
       ports:
       - containerPort: 20000
         name: tfjob-port
         protocol: TCP
       volumeMounts:
         - name: train
           mountPath: "/mnt/train/data"
         - name: logging
           mountPath: "/mnt/training_logs"
       workingDir: /root
     priority: 0
     restartPolicy: Never
     securityContext: {}
     terminationGracePeriodSeconds: 30
     volumes:
     - name: train
       persistentVolumeClaim:
         claimName: tf-train-pvc
     - name: logging
       persistentVolumeClaim:
         claimName: tf-logging-pvc
   EOF

Opção 2: Gravar usando o OSS SDK

Grave os checkpoints diretamente no OSS com o SDK — nenhum PV de leitura/gravação é necessário. O pod lê de um PV somente leitura e grava via SDK.

1. Adicione o OSS Python SDK à imagem do contêiner:

   RUN pip install oss2

   Consulte Instalação.

2. Modifique o código de treinamento para enviar os checkpoints usando o SDK. O código original salva checkpoints em log_dir a cada 100 iterações usando tf.train.Saver com max_to_keep=0, gerando 10 conjuntos de checkpoint após 1.000 iterações.

   • Defina max_to_keep=1 para reter apenas o checkpoint mais recente, reduzindo o uso de memória.

   • Envie cada checkpoint para o OSS com put_object_from_file logo após salvá-lo.

   Utilize I/O assíncrono com o SDK para aumentar ainda mais o throughput quando os caminhos de leitura e gravação estiverem separados.

   def train():
       ...
   saver = tf.train.Saver(max_to_keep=0)
   
       for i in range(FLAGS.max_steps):
           if i % 10 == 0:  # Record summaries and test-set accuracy
               summary, acc = sess.run([merged, accuracy], feed_dict=feed_dict(False))
               print('Accuracy at step %s: %s' % (i, acc))
               if i % 100 == 0:
                   print('Save checkpoint at step %s: %s' % (i, acc))
                   saver.save(sess, FLAGS.log_dir + '/model.ckpt', global_step=i)

   Substitua pelo envio baseado no SDK. Duas alterações reduzem o uso de memória e eliminam a necessidade do PV de leitura/gravação: leia o AccessKey e as configurações do bucket a partir de variáveis de ambiente. Consulte Configure access credentials.

   import oss2
   from oss2.credentials import EnvironmentVariableCredentialsProvider
   
   auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
   url = os.getenv('URL','<default-url>')
   bucketname = os.getenv('BUCKET','<default-bucket-name>')
   bucket = oss2.Bucket(auth, url, bucketname)
   
   ...
   def train():
     ...
     saver = tf.train.Saver(max_to_keep=1)
   
    for i in range(FLAGS.max_steps):
       if i % 10 == 0:  # Record summaries and test-set accuracy
         summary, acc = sess.run([merged, accuracy], feed_dict=feed_dict(False))
         print('Accuracy at step %s: %s' % (i, acc))
         if i % 100 == 0:
           print('Save checkpoint at step %s: %s' % (i, acc))
           saver.save(sess, FLAGS.log_dir + '/model.ckpt', global_step=i)
           # FLAGS.log_dir = os.path.join(os.getenv('TEST_TMPDIR', '/mnt'),'training_logs')
           for path,_,file_list in os.walk(FLAGS.log_dir) :
             for file_name in file_list:
               bucket.put_object_from_file(os.path.join('tf-train/training_logs', file_name), os.path.join(path, file_name))

   A imagem modificada do contêiner é registry.cn-beijing.aliyuncs.com/tool-sys/tf-train-demo:ro.

3. Implante o pod com o PV somente leitura e as credenciais do SDK. O pod define accessModes como ReadOnlyMany e passa OSS_ACCESS_KEY_ID e OSS_ACCESS_KEY_SECRET para que o SDK autentique com as mesmas credenciais do PV.

   cat << EOF | kubectl apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: oss-secret
     namespace: default
   stringData:
     akId: "<your-accesskey-id>"
     akSecret: "<your-accesskey-secret>"
   ---
   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: tf-train-pv
     labels:
       alicloud-pvname: tf-train-pv
   spec:
     capacity:
       storage: 10Gi
     accessModes:
       - ReadOnlyMany
     persistentVolumeReclaimPolicy: Retain
     csi:
       driver: ossplugin.csi.alibabacloud.com
       volumeHandle: tf-train-pv
       nodePublishSecretRef:
         name: oss-secret
         namespace: default
       volumeAttributes:
         bucket: "<your-bucket-name>"
         url: "oss-<region>.aliyuncs.com"
         otherOpts: "-o kernel_cache -o max_stat_cache_size=10000 -o umask=022 -o allow_other"
         path: "/tf-train/train/data"
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: tf-train-pvc
   spec:
     accessModes:
     - ReadOnlyMany
     resources:
       requests:
         storage: 10Gi
     selector:
       matchLabels:
         alicloud-pvname: tf-train-pv
   ---
   apiVersion: v1
   kind: Pod
   metadata:
     labels:
       app: tfjob
     name: tf-mnist
     namespace: default
   spec:
     containers:
     - command:
       - sh
       - -c
       - python /app/main.py
       env:
       - name: NVIDIA_VISIBLE_DEVICES
         value: void
       - name: gpus
         value: "0"
       - name: workers
         value: "1"
       - name: TEST_TMPDIR
         value: "/mnt"
       - name: OSS_ACCESS_KEY_ID      #The source of the AccessKey is the same as that of the PV.
         valueFrom:
           secretKeyRef:
             name: oss-secret
             key: akId
       - name: OSS_ACCESS_KEY_SECRET  #The source of the AccessKey is the same as that of the PV.
         valueFrom:
           secretKeyRef:
             name: oss-secret
             key: akSecret
       - name: URL                    #You can ignore this if a default URL is configured.
         value: "https://oss-<region>.aliyuncs.com"
       - name: BUCKET                 #You can ignore this if a default BUCKET is configured.
         value: "<bucket-name>"
       image: registry.cn-beijing.aliyuncs.com/tool-sys/tf-train-demo:ro
       imagePullPolicy: Always
       name: tensorflow
       ports:
       - containerPort: 20000
         name: tfjob-port
         protocol: TCP
       volumeMounts:
         - name: train
           mountPath: "/mnt/train/data"
       workingDir: /root
     priority: 0
     restartPolicy: Never
     securityContext: {}
     terminationGracePeriodSeconds: 30
     volumes:
     - name: train
       persistentVolumeClaim:
         claimName: tf-train-pvc
   EOF