Deep Learning is currently one of the hottest subtopics in the field of artificial intelligence (AI). Deep learning is necessary to make sense out of large amounts of data, especially when it comes to Big Data applications.
TensorFlow is an open source software library that uses data flow graphs for numeric computation. The nodes in the graphs represent mathematical operations, while the edges represent the multidimensional data arrays (aka tensors) that are passed between them. The flexible architecture allows you to deploy the computation to one or more CPUs or GPUs in your desktops, servers, or mobile devices using a single API.
TensorFlow is a piece of cake, if you have ever used NumPy. One of the main differences between the two is that TensorFlow follows a "lazy" programming paradigm. It first builds all the action graphs to be completed, then it "runs" the graphs when a "session" is invoked. Building a computation graph can be considered the major component of TensorFlow.
TensorFlow is more than just a powerful neural network library. It enables you to build other machine learning algorithms on it, such as decision trees or k-nearest-neighbors.
The normal workflow of running a program in TensorFlow is as follows:
See more on How to Implement Neural Networks with TensorFlow.
This tutorial is targeted for any organization that wants to host TensorFlow on Alibaba Cloud using Docker containers. This document can be used by Solution Architects or Business Development teams for proof of concept (POC) of any customer requirement and gradually can be converted into production grade hosting.
In this lab, we will be using transfer learning, which means we are starting with a model that has been already trained on another problem. We will then be retraining it on a similar problem. Deep learning from scratch can take days, but transfer learning can be done in a much short order.
The Flink community is actively providing support for deep learning. Alibaba provides the TensorFlow on Flink project, which allows users to run TensorFlow in Flink jobs, use Flink for data processing and then send the processed data to the Python process in TensorFlow for deep learning training. For programming languages, the Flink community is working on the support for Python. Currently, Flink only supports Java and Scala APIs. Both the languages are JVM-based. Therefore, currently Flink is suitable for big data processing in the system but not so suitable for data analysis and machine learning. In general, people in the fields of data analysis and machine learning prefer using more advanced languages like Python and R. The Flink community is also planning support for these languages in the near future. Flink will support Python first, because Python has enjoyed fast development in recent years, in large part due to the development of AI and deep learning. At present, all popular deep learning libraries provide Python APIs, including TensorFlow, Pytorch, and Keras. When Python is supported in Flink, users will be able to connect all pipelines for machine learning in just one language, which should improve their development by leaps and bounds.
This guide creates an image recognition model using the deep learning framework TensorFlow in Alibaba Cloud Machine Learning Platform for AI. The entire procedure takes about 30 minutes to complete. After the procedure, the system is able to recognize the bird in the following image.
Data preparation and preprocessing play important roles in the deep learning and training process, and affect the speed and quality of model training.
TensorFlow supports the HDFS, integrates big data and deep learning, and completes the chain from data preparation to model training. The deep learning solution of Alibaba Cloud Container Service provides three distributed storage backends (Object Storage Service (OSS), NAS, and HDFS) to support TensorFlow.
This document describes how to convert data to the TFRecord format and store the generated TFRecord files to the HDFS. The HDFS of Alibaba Cloud Elastic MapReduce (E-MapReduce) is used in this example.
E-MapReduce is an all-in-one enterprise-ready big data platform that provides cluster, job, and data management services based on open-source ecosystems, such as Hadoop, Spark, Kafka, Flink, and Storm.
Machine Learning Platform for AI provides end-to-end machine learning services, including data processing, feature engineering, model training, model prediction, and model evaluation. Machine Learning Platform for AI combines all of these services to make AI more accessible than ever.
2,605 posts | 747 followers
FollowAlibaba F(x) Team - December 11, 2020
Alibaba Clouder - November 4, 2019
Alibaba Clouder - October 24, 2019
Alibaba EMR - August 19, 2020
Apache Flink Community China - September 16, 2020
Alibaba Clouder - March 3, 2021
2,605 posts | 747 followers
Follow
MaxCompute
Conduct large-scale data warehousing with MaxCompute
Learn MoreMore Posts by Alibaba Clouder
