I/O approaches of TensorFlow in PAI

• Read data from your computer: The server directly obtains graphs from the client for computing.
• Read data from the cloud: The server obtains graphs and distributes these graphs to workers for computing.

• Do not use built-in approaches of Python to read data from your computer.
  PAI supports the built-in I/O approaches of Python. To use these approaches, you must compress the data source and code into a package and upload this package to OSS. This approach writes data to the memory for computing and is inefficient. We recommend that you do not use this approach. The following code provides an example on how this approach works:

  import csv
  csv_reader=csv.reader(open('csvtest.csv'))
  for row in csv_reader:
    print(row)

• Do not use third-party libraries to read data.

  Data can be read by using third-party libraries, such as TFLearn and pandas. However, third-party libraries are encapsulated in a Python package to read data. In this case, data reading in PAI is inefficient.

• Do not perform the preload operations to read data.

  You may find that GPUs are not significantly faster than local CPUs. The possible cause is that I/O operations waste resources. A preload operation first reads data to the memory. Then, it performs session operations, such as feeding, for computing. A preload operation wastes computing resources and cannot process large amounts of data due to the memory limit.

  For example, if a hard disk contains an image dataset, you must load the data before the computing process starts. It requires 0.1s to load the image dataset and 0.9s for computing. The GPU is idle for 0.1s every second. This reduces efficiency.

import argparse
import tensorflow as tf
import os
FLAGS=None
def main(_):
    dirname = os.path.join(FLAGS.buckets, "csvtest.csv")
    reader=tf.TextLineReader()
    filename_queue=tf.train.string_input_producer([dirname])
    key,value=reader.read(filename_queue)
    record_defaults=[[''],[''],[''],[''],['']]
    d1, d2, d3, d4, d5= tf.decode_csv(value, record_defaults, ',')
    init=tf.initialize_all_variables()
    with tf.Session() as sess:
        sess.run(init)
        coord = tf.train.Coordinator()
        threads = tf.train.start_queue_runners(sess=sess,coord=coord)
        for i in range(4):
            print(sess.run(d2))
        coord.request_stop()
        coord.join(threads)
if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--buckets', type=str, default='',
                        help='input data path')
    parser.add_argument('--checkpointDir', type=str, default='',
                        help='output model path')
    FLAGS, _ = parser.parse_known_args()
    tf.app.run(main=main)

• dirname: The path of the OSS object. The value of this parameter can be an array.
• reader: TensorFlow in PAI provides APIs for different types of readers. You can select a reader as needed.
• tf.train.string_input_producer: converts the object into a queue.
• tf.decode_csv: provides an op to split data. You can use this op to obtain specific parameters in each row.
• To retrieve data by using an op, you must call the tf.train.Coordinator() and tf.train.start_queue_runners(sess=sess,coord=coord) methods in a session.