JStorm on Alibaba Cloud Container Service

Original Headline:
Wield epoch-making power to usher in the era of stream computing? All you need is one-key deployment of JStorm on Alibaba Cloud Container Service
Headline Option #1:
One-Key Deployment of JStorm = 4X Faster Performance
Headline Option #2:
One-Key Deployment of JStorm on Alibaba Cloud Container Service
If your project involves real-time analytics, online machine learning, and continuous computation, you would love to have a stream computing engine to your rescue.
But, consider performance as the key factor. Would you prefer a distributed real-time computation system that is high on performance? Let’s say 4X faster than the most common Apache Storm.


We are talking about JStorm, a powerful enterprise-level stream computing engine launched by Alibaba. In contrast with Apache Storm, it features convenience, high performance, and diversified ecosystem and is regarded as the ideal option to build a stream computing platform.
But, with high performance comes a small challenge. JStorm, takes a bit longer to deploy because it involves various components, such as Zookeeper, Python, and JDK. At the same time, it needs to configure roles, including Nimbus and Supervisor.
When it comes to performance, you have an easier choice to make. Alibaba's JStorm and Container Service teams collaborated and released JStorm for Docker, which supports one-key deployment.

To start with, Create a Cluster
You can easily create a cluster using the integrated interface from Alibaba Cloud Container Service. To learn the cluster creation process, please refer to the Help document. It is worth noting that at least three machines are needed.

Next, Deploy Zookeeper
To achieve high availability, we need to deploy three Zookeeper nodes. The traditional Zookeeper deployment needs to use the IP addresses of all nodes as configuration files. After we have migrated the nodes, we need to update the configurations, and reset all the nodes. Alibaba Cloud Container Service provides a DNS-like container name resolution service that uses the container name as a configuration item, saving the trouble to modify configuration during migration.
Orchestration Template:
zk1: image: 'mesoscloud/zookeeper:3.4.8-ubuntu' environment: - MYID=1 - SERVERS=jstorm_zk1_1,jstorm_zk2_1,jstorm_zk3_1 - constraint:aliyun.node_index==1 restart: 'always' zk2: image: 'mesoscloud/zookeeper:3.4.8-ubuntu' environment: - MYID=2 - SERVERS=jstorm_zk1_1,jstorm_zk2_1,jstorm_zk3_1 - constraint:aliyun.node_index==2 restart: 'always' zk3: image: 'mesoscloud/zookeeper:3.4.8-ubuntu' environment: - MYID=3 - SERVERS=jstorm_zk1_1,jstorm_zk2_1,jstorm_zk3_1 - constraint:aliyun.node_index==3 restart: 'always'
In the sample template, jstorm_zk1_1, jstorm_zk2_1, and jstorm_zk3_1 are container names that follow the naming convention of <application name><service name><container number>. Therefore, as long as the application name and service name are identified, container names are consistent and can be used as configuration items. In addition, container names can be resolved across hosts in the same container network.
Also, to distribute Zookeeper nodes on different machines, use the node constraint function constraint:aliyun.node_index==n to make sure that the container is deployed only on Node n, in which "n" is the serial number of a node in a cluster.

Deploy Storm
Then, comes the storm.
Alibaba‘s JStorm team provides a Docker image of JStorm and regularly keeps it up to date.
JStorm has two roles: Nimbus and Supervisor. To achieve high availability, deploy two Nimbus' and configure the number of Supervisors as required.
Similarly, to distribute Nimbus' on different machines, you can specify application constraint with affinity:service!=nimbus, so that containers with the Nimbus service are deployed on other machines with no Nimbus container.
Besides, the container name can substitute for the Nimbus address that enables Supervisor.
JStorm has a Web UI that provides services on port 8080 by default. Alibaba Cloud Container Service quickly exposes the Web services to the Internet by configuring the tag with aliyun.routing.port_8080: 'http://nimbus'. The tag means, "Mapping the Nimbus subdomain to the container's port 8080." For more details, please refer to the Help document.
Orchestration Template:
nimbus: image: 'registry.cn-hangzhou.aliyuncs.com/jstorm-docker/jstorm' environment: - CHARACTER=nimbus - ZK_ROOT=/jstorm.docker.root2 - ZK_HOSTS=jstorm_zk1_1,jstorm_zk2_1,jstorm_zk3_1 - CLUSTER_NAME=jstorm.docker.cluster labels: aliyun.routing.port_8080: 'http://nimbus' aliyun.scale: '2' restart: 'always' supervisor: image: 'registry.cn-hangzhou.aliyuncs.com/jstorm-docker/jstorm' environment: - CHARACTER=supervisor - NIMBUS_HOSTS=jstorm_nimbus_1 - ZK_ROOT=/jstorm.docker.root2 - ZK_HOSTS=jstorm_zk1_1,jstorm_zk2

And, finally the one-key deployment. One-Key Deployment
We can place Zookeeper and JStorm in a single orchestration file for one-key deployment.
In the orchestration file, use the environment variable COMPOSE_PROJECT_NAME. COMPOSE_PROJECT_NAME as the default setting for the environment variable, and its value is the name of the current application.


zk1:
    image: 'mesoscloud/zookeeper:3.4.8-ubuntu'
    environment:
        - MYID=1
        - SERVERS=${COMPOSE_PROJECT_NAME}_zk1_1,${COMPOSE_PROJECT_NAME}_zk2_1,${COMPOSE_PROJECT_NAME}_zk3_1
        - constraint:aliyun.node_index==1
    restart: 'always'
zk2:
    image: 'mesoscloud/zookeeper:3.4.8-ubuntu'
    environment:
        - MYID=2
        - SERVERS=${COMPOSE_PROJECT_NAME}_zk1_1,${COMPOSE_PROJECT_NAME}_zk2_1,${COMPOSE_PROJECT_NAME}_zk3_1
        - constraint:aliyun.node_index==2
    restart: 'always'
zk3:
    image: 'mesoscloud/zookeeper:3.4.8-ubuntu'
    environment:
        - MYID=3
        - SERVERS=${COMPOSE_PROJECT_NAME}_zk1_1,${COMPOSE_PROJECT_NAME}_zk2_1,${COMPOSE_PROJECT_NAME}_zk3_1
        - constraint:aliyun.node_index==3
    restart: 'always'
nimbus:
    image: 'registry.cn-hangzhou.aliyuncs.com/jstorm-docker/jstorm'
    environment:
        - CHARACTER=nimbus
        - ZK_ROOT=/jstorm.docker.root2
        - ZK_HOSTS=${COMPOSE_PROJECT_NAME}_zk1_1,${COMPOSE_PROJECT_NAME}_zk2_1,${COMPOSE_PROJECT_NAME}_zk3_1
        - CLUSTER_NAME=jstorm.docker.cluster
    labels:
        aliyun.routing.port_8080: 'http://nimbus'
        aliyun.scale: '2'
    restart: 'always'
    links:
        - zk1
        - zk2
        - zk3
supervisor:
    image: 'registry.cn-hangzhou.aliyuncs.com/jstorm-docker/jstorm'
    environment:
        - CHARACTER=supervisor
        - NIMBUS_HOSTS=jstorm_nimbus_1
        - ZK_ROOT=/jstorm.docker.root2
        - ZK_HOSTS=${COMPOSE_PROJECT_NAME}_zk1_1,${COMPOSE_PROJECT_NAME}_zk2_1,${COMPOSE_PROJECT_NAME}_zk3_1
        - CLUSTER_NAME=jstorm.docker.cluster
    labels:
        aliyun.routing.port_8080: 'http://supervisor'
        aliyun.scale: '3'
    restart: 'always'
    links:
        - nimbus

To ensure that Zookeeper is deployed before JStorm, use the link function. In addition to allowing network communication between containers, link informs the scheduler of dependencies between services for the scheduler to first initiate the depended-on service.
Using the orchestration file, we have implemented one-key deployment for JStorm. However, we need to note several details concerning production.
● By default, Zookeeper saves persistent data in /tmp/zookeeper. During production, this data volume must be mapped to the specified host directory.
● JStorm's Web UI is open to the Internet. During production, an intranet domain name or intranet Server Load Balancer instance is required.
● Zookeeper and JStorm have been deployed on the same host. During production, we suggest that you separate them.

You will be able to deploy JStorm on Alibaba Cloud Container Service through one-key deployment and power up your computation system performance by 4X.
Reference links for additional information:
● Alibaba Cloud Container Service Console
● Alibaba Cloud Container Service Help Documentation
● JStorm website