Container Service for Kubernetes:Praktik terbaik untuk Pemisahan baca/tulis OSS

Opsi 1: Menulis menggunakan PV ossfs read/write

1. Buat PV ossfs 1.0 read-only untuk set data. Perubahan konfigurasi utama:

   • Tetapkan accessModes ke ReadOnlyMany pada PV dan PVC. Mount subpath set data /tf-train/train/data.

   • Pada otherOpts, tambahkan -o kernel_cache -o max_stat_cache_size=10000 -o umask=022:

     • kernel_cache mengaktifkan caching baca di memori.

     • max_stat_cache_size=10000 menyimpan cache 10.000 entri metadata (~40 MB). Sesuaikan berdasarkan tipe instans dan ukuran set data.

     • umask=022 memberikan akses baca ke proses kontainer non-root.

   cat << EOF | kubectl apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: oss-secret
     namespace: default
   stringData:
     akId: "<your-accesskey-id>"
     akSecret: "<your-accesskey-secret>"
   ---
   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: tf-train-pv
     labels:
       alicloud-pvname: tf-train-pv
   spec:
     capacity:
       storage: 10Gi
     accessModes:
       - ReadOnlyMany
     persistentVolumeReclaimPolicy: Retain
     csi:
       driver: ossplugin.csi.alibabacloud.com
       volumeHandle: tf-train-pv
       nodePublishSecretRef:
         name: oss-secret
         namespace: default
       volumeAttributes:
         bucket: "<your-bucket-name>"
         url: "oss-<region>.aliyuncs.com"
         otherOpts: "-o kernel_cache -o max_stat_cache_size=10000 -o umask=022 -o allow_other"
         path: "/tf-train/train/data"
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: tf-train-pvc
   spec:
     accessModes:
     - ReadOnlyMany
     resources:
       requests:
         storage: 10Gi
     selector:
       matchLabels:
         alicloud-pvname: tf-train-pv
   EOF

2. Buat PV ossfs 1.0 read/write untuk checkpoint, mount subpath /tf-train/training_logs. Caching metadata dinonaktifkan (max_stat_cache_size=0) karena penulisan checkpoint sequential tidak mendapat manfaat dari caching.

   cat << EOF | kubectl apply -f -
   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: tf-logging-pv
     labels:
       alicloud-pvname: tf-logging-pv
   spec:
     capacity:
       storage: 10Gi
     accessModes:
       - ReadWriteMany
     persistentVolumeReclaimPolicy: Retain
     csi:
       driver: ossplugin.csi.alibabacloud.com
       volumeHandle: tf-logging-pv
       nodePublishSecretRef:
         name: oss-secret
         namespace: default
       volumeAttributes:
         bucket: "<your-bucket-name>"
         url: "oss-<region>.aliyuncs.com"
         otherOpts: "-o max_stat_cache_size=0 -o allow_other"
         path: "/tf-train/training_logs"
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: tf-logging-pvc
   spec:
     accessModes:
     - ReadWriteMany
     resources:
       requests:
         storage: 10Gi
     selector:
       matchLabels:
         alicloud-pvname: tf-logging-pv
   EOF

3. Deploy Pod pelatihan dengan kedua PV dimount.

   Tidak diperlukan perubahan kode. Mount kedua PV — PV read-only di /mnt/train/data dan PV read/write di /mnt/training_logs.

   cat << EOF | kubectl apply -f -
   apiVersion: v1
   kind: Pod
   metadata:
     labels:
       app: tfjob
     name: tf-mnist
     namespace: default
   spec:
     containers:
     - command:
       - sh
       - -c
       - python /app/main.py
       env:
       - name: NVIDIA_VISIBLE_DEVICES
         value: void
       - name: gpus
         value: "0"
       - name: workers
         value: "1"
       - name: TEST_TMPDIR
         value: "/mnt"
       image: registry.cn-beijing.aliyuncs.com/tool-sys/tf-train-demo:rw
       imagePullPolicy: Always
       name: tensorflow
       ports:
       - containerPort: 20000
         name: tfjob-port
         protocol: TCP
       volumeMounts:
         - name: train
           mountPath: "/mnt/train/data"
         - name: logging
           mountPath: "/mnt/training_logs"
       workingDir: /root
     priority: 0
     restartPolicy: Never
     securityContext: {}
     terminationGracePeriodSeconds: 30
     volumes:
     - name: train
       persistentVolumeClaim:
         claimName: tf-train-pvc
     - name: logging
       persistentVolumeClaim:
         claimName: tf-logging-pvc
   EOF

Opsi 2: Menulis menggunakan OSS SDK

Tulis checkpoint langsung ke OSS dengan SDK — tidak diperlukan PV read/write. Pod membaca dari PV read-only dan menulis melalui SDK.

1. Tambahkan OSS Python SDK ke gambar kontainer:

   RUN pip install oss2

   Lihat Instalasi.

2. Modifikasi kode pelatihan untuk mengunggah checkpoint menggunakan SDK. Kode asli menyimpan checkpoint ke log_dir setiap 100 iterasi menggunakan tf.train.Saver dengan max_to_keep=0, menghasilkan 10 set checkpoint setelah 1.000 iterasi.

   • Tetapkan max_to_keep=1 untuk hanya menyimpan checkpoint terbaru, mengurangi Penggunaan memori.

   • Unggah setiap checkpoint ke OSS dengan put_object_from_file setelah disimpan.

   Gunakan I/O asinkron dengan SDK untuk lebih meningkatkan throughput ketika path baca dan tulis dipisahkan.

   def train():
       ...
   saver = tf.train.Saver(max_to_keep=0)
   
       for i in range(FLAGS.max_steps):
           if i % 10 == 0:  # Record summaries and test-set accuracy
               summary, acc = sess.run([merged, accuracy], feed_dict=feed_dict(False))
               print('Accuracy at step %s: %s' % (i, acc))
               if i % 100 == 0:
                   print('Save checkpoint at step %s: %s' % (i, acc))
                   saver.save(sess, FLAGS.log_dir + '/model.ckpt', global_step=i)

   Ganti dengan unggahan berbasis SDK. Dua perubahan mengurangi Penggunaan memori dan menghilangkan PV read/write: baca pengaturan AccessKey dan bucket dari Variabel lingkungan. Lihat Mengonfigurasi kredensial akses.

   import oss2
   from oss2.credentials import EnvironmentVariableCredentialsProvider
   
   auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
   url = os.getenv('URL','<default-url>')
   bucketname = os.getenv('BUCKET','<default-bucket-name>')
   bucket = oss2.Bucket(auth, url, bucketname)
   
   ...
   def train():
     ...
     saver = tf.train.Saver(max_to_keep=1)
   
    for i in range(FLAGS.max_steps):
       if i % 10 == 0:  # Record summaries and test-set accuracy
         summary, acc = sess.run([merged, accuracy], feed_dict=feed_dict(False))
         print('Accuracy at step %s: %s' % (i, acc))
         if i % 100 == 0:
           print('Save checkpoint at step %s: %s' % (i, acc))
           saver.save(sess, FLAGS.log_dir + '/model.ckpt', global_step=i)
           # FLAGS.log_dir = os.path.join(os.getenv('TEST_TMPDIR', '/mnt'),'training_logs')
           for path,_,file_list in os.walk(FLAGS.log_dir) :
             for file_name in file_list:
               bucket.put_object_from_file(os.path.join('tf-train/training_logs', file_name), os.path.join(path, file_name))

   Gambar kontainer yang dimodifikasi adalah registry.cn-beijing.aliyuncs.com/tool-sys/tf-train-demo:ro.

3. Deploy Pod dengan PV read-only dan kredensial SDK. Pod menetapkan accessModes ke ReadOnlyMany dan meneruskan OSS_ACCESS_KEY_ID dan OSS_ACCESS_KEY_SECRET agar SDK melakukan autentikasi dengan kredensial yang sama seperti PV.

   cat << EOF | kubectl apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: oss-secret
     namespace: default
   stringData:
     akId: "<your-accesskey-id>"
     akSecret: "<your-accesskey-secret>"
   ---
   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: tf-train-pv
     labels:
       alicloud-pvname: tf-train-pv
   spec:
     capacity:
       storage: 10Gi
     accessModes:
       - ReadOnlyMany
     persistentVolumeReclaimPolicy: Retain
     csi:
       driver: ossplugin.csi.alibabacloud.com
       volumeHandle: tf-train-pv
       nodePublishSecretRef:
         name: oss-secret
         namespace: default
       volumeAttributes:
         bucket: "<your-bucket-name>"
         url: "oss-<region>.aliyuncs.com"
         otherOpts: "-o kernel_cache -o max_stat_cache_size=10000 -o umask=022 -o allow_other"
         path: "/tf-train/train/data"
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: tf-train-pvc
   spec:
     accessModes:
     - ReadOnlyMany
     resources:
       requests:
         storage: 10Gi
     selector:
       matchLabels:
         alicloud-pvname: tf-train-pv
   ---
   apiVersion: v1
   kind: Pod
   metadata:
     labels:
       app: tfjob
     name: tf-mnist
     namespace: default
   spec:
     containers:
     - command:
       - sh
       - -c
       - python /app/main.py
       env:
       - name: NVIDIA_VISIBLE_DEVICES
         value: void
       - name: gpus
         value: "0"
       - name: workers
         value: "1"
       - name: TEST_TMPDIR
         value: "/mnt"
       - name: OSS_ACCESS_KEY_ID      #The source of the AccessKey is the same as that of the PV.
         valueFrom:
           secretKeyRef:
             name: oss-secret
             key: akId
       - name: OSS_ACCESS_KEY_SECRET  #The source of the AccessKey is the same as that of the PV.
         valueFrom:
           secretKeyRef:
             name: oss-secret
             key: akSecret
       - name: URL                    #You can ignore this if a default URL is configured.
         value: "https://oss-<region>.aliyuncs.com"
       - name: BUCKET                 #You can ignore this if a default BUCKET is configured.
         value: "<bucket-name>"
       image: registry.cn-beijing.aliyuncs.com/tool-sys/tf-train-demo:ro
       imagePullPolicy: Always
       name: tensorflow
       ports:
       - containerPort: 20000
         name: tfjob-port
         protocol: TCP
       volumeMounts:
         - name: train
           mountPath: "/mnt/train/data"
       workingDir: /root
     priority: 0
     restartPolicy: Never
     securityContext: {}
     terminationGracePeriodSeconds: 30
     volumes:
     - name: train
       persistentVolumeClaim:
         claimName: tf-train-pvc
   EOF