The Internet is never a safe place. Most of the time, we rely too much on firewalls to contain security problems. Unfortunately, reliance on the firewall assumes that attacks always come from the outside while the truly destructive attacks often come from the inside.
In recent years, websites such as The Hacker News have reported on widespread attacks and ransoms due to data security problems. Versions earlier than Hadoop 1.0.0 provided no security support and assumed that any roles in a cluster are trusted. As a result, user access is not authenticated and malicious users can easily access clusters by means of masquerading.
To ensure the security of Hadoop clusters, user authentication and authorization must be implemented. To address this, the following common solutions are developed:
After Hadoop 1. 0. 0 was released in 2012, Hadoop started to support Kerberos to ensure that the nodes in a cluster are trustworthy.
Before the cluster is deployed, Kerberos stores the authentication key on a trusted node. When the cluster runs, the nodes in the cluster are authenticated by the key, and only the successfully authenticated nodes can be used to provide services. Impersonated nodes cannot communicate with any nodes in the cluster because they do not carry the key information in advance. This prevents the malicious utilization of or tampering with the Hadoop cluster, ensuring its trustworthiness and security.
Kerberos is a network authentication protocol that was designed to protect network servers in Athena projects. The name "Kerberos" is the name of the three-headed dog from Greek mythology. As its name implies, it provides strong authentication for the client-to-server access sequence by using key encryption technology. Kerberos can prevent eavesdropping and can replay attacks for data integrity. It is a system that uses symmetric key algorithms to manage keys. Kerberos-based products also use the public key encryption method for authentication.
So far, the latest version of Kerberos is V5, and the V1 to V3 versions are only available within MIT because DES encryption is used. In its early development, Kerberos was classified as military arms by the U.S. Export Controls and its export was banned until the Royal Swedish Institute of Engineering released Kerberos V4, namely KTH-KRB. Later, this team released V5 (Heimdal), which is one of the most common implementations of Kerberos V5.
The Kerberos V5 implementation version mentioned in this document refers to MIT Kerberos, which is updated regularly on a six-month basis. Presently, the latest version of MIT Kerberos is version 1.16.2, which was released on November 1, 2018.
Kerberos some common terms including:
If Kerberos is used, a client can be served by going through these steps:
The Kerberos protocol belongs to the display layer of the network communication protocol. The communication process of Kerberos is as follows: First, the accessing user uses the shared key to obtain an identity certificate from an AS. Then, the user uses the identity certificate, but not the shared key, to communicate with the SS.
This process uses symmetric encryption as the encryption method and occurs in a Kerberos realm. The lower-case letters c, d, e, and g indicate the messages sent by the client, while the upper-case letters A, B, E, F, and H are the messages returned by each server.
The client retrieves the TGT from the AS.
First, the user must log on to the client in either of the following ways:
The client obtains the ticket from the TGS, namely the client-to-server ticket.
The client retrieves the service from the SS.
Kerberos supports two server redundancy modes in a realm: Master/Slave (MIT and Heimdal) and Multi-master modes (Windows Active Directory). If Kerberos is deployed in a production environment, we recommended that you use the single-master-and-multi-slave mode to ensure the high availability (HA) of Kerberos services.
Each KDC in Kerberos contains a copy of the database. The master KDC contains a writeable copy of the realm database, which is copied to the slave KDC at a fixed interval. All database changes such as password changes are made on the master KDC. When the master KDC becomes unavailable, the slave KDC provides a Kerberos ticket for service authorization rather than managing the database. An administrator is required to perform routine management tasks on KDCs.
The synchronization mechanism of Kerberos only replicates the contents of the primary database but does not pass the configuration files. Therefore, you must manually copy the following files to each slave KDC:
- krb5.conf - kdc.conf - kadm5.acl - master key stash file
At present, the most common HA solution for a single IDC is the combination of Keepalived and Rsync. Keepalived can build multiple stateful points into an HA service by using the virtual IP (VIP) failover method.
To use this method, first create a dump file for the database in the master KDC to dump the current Kerberos and KADM5 databases to ASCII files.
kdb5_util dump [-b7|-ov|-r13] [-verbose] [-mkey_convert] [-new_mkey_file mkey_file] [-rev] [-recurse] [filename [principals...]]
Then, use Rsync to synchronize the directory where the dump file is located to the corresponding directory on the slave machine before importing the directory to the KDC.
kdb5_util load [-b7|-ov|-r13] [-hash] [-verbose] [-update] filename [dbname]
All Hadoop requests use the KDC by requesting the domain name of the internal network and resolving it to the VIP bound to Keepalived:
User (Principal) Management
If the team already has a permissions system, it is difficult to integrate the existing identification system with Kerberos.
With the rapid growth of businesses and the increasing sizes of servers, the manual operations performed by a Kerberos principal (including addition, deletion, modification, and querying) become more frequent and troublesome. To address this, you need to standardize the principal requesting, maintenance, deletion, and keytab generation processes in the Kerberos management system. In addition, principal requesting and permissions management should be automated.
Data Synchronization Optimization
During Kerberos data synchronization, you can synchronize the generated data records to MySQL by using the MySQL dual-master synchronization method. In the cross-IDC scenario, you can use the Rsync tool to synchronize incremental KDC data. The Rsync server uses the keepalived VIP method in which the core IDC (A) serves as the active IDC. When the Kerberos host becomes faulty, to ensure the high availability of KDC data, the VIP migrates to another KDC host and the Rsync client synchronizes data with this KDC host, which now serves as the Rsync server.
Process management tools are used for the aliveness monitoring of Kerberos-related processes. When an unexpected process exit is detected, an email, WeChat, or DingTalk alarm is sent for actively restoring the process.
Deploying Kerberos in Hadoop clusters is tedious. Essentially, Kerberos is a protocol or secure channel. For most users, it is complicated to fully understand it. In this context, is there any better implementation that can free common users from these annoying details?
Reportedly, the Hadoop Authentication Service (HAS) co-developed by Alibaba and Intel has been applied to ApsaraDB for HBase2.0.
The HAS solution replaces the MIT Kerberos service with Kerby and uses the HAS plug-in authentication method to build an account and password system that is familiar for most users.
Currently, HAS is under development in the Apache Kerby project branch has-project and will be included as a new Kerby feature in the next release.
As a sub-project of Apache Directory, Apache Kerby is not drawing too much attention but has the potential for future success.
Alibaba Clouder - June 15, 2018
Alibaba Clouder - February 28, 2018
Alibaba Cloud MaxCompute - January 7, 2019
Alibaba Cloud Product Launch - December 11, 2018
Alibaba Clouder - September 19, 2018
Alibaba Clouder - February 22, 2019
Conduct large-scale data warehousing with MaxComputeLearn More
A Big Data service that uses Apache Hadoop and Spark to process and analyze dataLearn More
A cloud firewall service utilizing big data capabilities to protect against web-based attacksLearn More
More Posts by Alibaba Clouder