FAQ about TensorFlow

You can use Python files to organize your training scripts. First, store the data preprocessing logic in a Python file. Then, define the model in another Python file. At last, use one Python file to train the data throughout the entire training process. For example, you can define a function in the test1.py file. If the test2.py file needs to use the function provided in the test1.py file and the test2.py file serves as the program entry, you need only to compress the test1.py and test2.py files into a .tar.gz file and upload the package. The following figure shows the process.where:

• Python Code Files: the .tar.gz package.
• Primary Python File: the program entry file.

To upload data to OSS, you can call API operations or use SDKs. For more information, see Simple upload. In addition, OSS provides a lot of tools to facilitate data uploading and downloading. For more information, see OSS tools. We recommend that you use ossutil or osscmd to upload and download files.

You can read data in PAI by using the following methods:

• Use functions of the tf.gfile module to read images or text. The following sample functions are supported:

  tf.gfile.Copy(oldpath, newpath, overwrite=False) # Copies a file.
  tf.gfile.DeleteRecursively(dirname) # Recursively deletes all files in a directory.
  tf.gfile.Exists(filename) # Checks whether a file exists.
  tf.gfile.FastGFile(name, mode='r') # Reads a file in non-blocking mode.
  tf.gfile.GFile(name, mode='r') # Reads a file.
  tf.gfile.Glob(filename) # Queries all files in a directory. You can filter these files by pattern.
  tf.gfile.IsDirectory(dirname) # Checks whether an item is a directory.
  tf.gfile.ListDirectory(dirname) # Queries all files in a directory.
  tf.gfile.MakeDirs(dirname) # Creates a folder in a directory. If no parent directories exist, a parent directory is automatically created. If the folder you want to create already exists and is writable, a success response is returned.
  tf.gfile.MkDir(dirname) # Creates a folder in a directory.
  tf.gfile.Remove(filename) # Deletes a file.
  tf.gfile.Rename(oldname, newname, overwrite=False) # Renames a file.
  tf.gfile.Stat(dirname) # Queries statistical data about a directory.
  tf.gfile.Walk(top, inOrder=True) # Queries the file tree of a directory.

• Use tf.gfile.Glob, tf.gfile.FastGFile, tf.WhoFileReader(), and tf.train.shuffer_batch() functions to batch read files. Before you batch read files, you must obtain the list of the files and create a batch.

When you create a machine learning experiment on the Machine Learning Studio page, you must specify the parameters such as the code file that you want to read and its directory on the right side of the page. Functions of the tf.flags module can pass in parameters in the -XXX form, where XXX represents a string.

import tensorflow as tf
FLAGS = tf.flags.FLAGS
tf.flags.DEFINE_string('buckets', 'oss://{OSS Bucket}/', 'Folder of the training image file')
tf.flags.DEFINE_string('batch_size', '15', 'Size of the batch')
files = tf.gfile.Glob(os.path.join(FLAGS.buckets,'*.jpg')) # Queries the paths of all JPG files in buckets.

We recommend that you use the following methods to batch read files based on the number of files:

• Use the tf.gfile.FastGfile() function to batch read a small number of files.

  for path in files:
      file_content = tf.gfile.FastGFile(path, 'rb').read() # Remember to specify rb when you call this function. Otherwise, errors may occur.
      image = tf.image.decode_jpeg(file_content, channels=3) # In this example, JPG images are used.

• Use the tf.WhoFileReader() function to batch read a large number of files.

  reader = tf.WholeFileReader()  # Instantiates a reader.
  fileQueue = tf.train.string_input_producer(files)  # Creates a queue for the reader to read.
  file_name, file_content = reader.read(fileQueue)  # Uses the reader to read a file from the queue.
  image_content = tf.image.decode_jpeg(file_content, channels=3)  # Decodes the file content into images.
  label = XXX  # In this example, label processing operations are omitted.
  batch = tf.train.shuffle_batch([label, image_content], batch_size=FLAGS.batch_size, num_threads=4,
                                 capacity=1000 + 3 * FLAGS.batch_size, min_after_dequeue=1000)
  sess = tf.Session()  # Creates a session.
  tf.train.start_queue_runners(sess=sess)  # Starts the queue. If this command is not executed, the thread keeps blocked.
  labels, images = sess.run(batch)  # Obtains the results.

  The following description explains the code:
  • tf.train.string_input_producer: converts files into a queue. You must use tf.train.start_queue_runners to start the queue.
  • tf.train.shuffle_batch includes the following parameters:
    • batch_size: the amount of data to return each time after you run a batch task.
    • num_threads: the number of threads to run. The value is usually set to 4.
    • capacity: the number of pieces of data from which you want to randomly extract data. For example, assume that a dataset has 10,000 pieces of data. If you want to randomly extract data from 5,000 pieces of data to train, set capacity to 5000.
    • min_after_dequeue: the minimum length of the queue to maintain. The value must be less than or equal to the value of capacity.

You can write data to OSS by using one of the following methods: (The generated file is stored in the /model/example.txt directory.)

• Use the tf.gfile.FastGFile() function to write a file. The following code shows the sample function.

  tf.gfile.FastGFile(FLAGS.checkpointDir + 'example.txt', 'wb').write('hello world')

• Use the tf.gfile.Copy() function to copy a file. The following code shows the sample function.

  tf.gfile.Copy('./example.txt', FLAGS.checkpointDir + 'example.txt')