An Interpretation of PolarDB-X Source Codes (2): CN Startup Process

Part 2 of this 10-part series explains the startup process of the PolarDB-X CN node (GalaxySQL), including parameter loading, metadata loading, and other processes.

This article mainly explains the startup process of the PolarDB-X CN node (GalaxySQL), including parameter loading, metadata loading, and other processes. It will briefly introduce the modules designed in the startup process.

The codes of the CN Server layer are contained in the polardbx-server module, and the main function is located in the TddlLauncher.

The primary logical entry is in the CobarServer.init() method.

CN startup is divided into the following processes:

Creation of CobarServer Objects

This class is a singleton and can be obtained with CobarServer.getInstance().

Parameter Loading

Path: TddlLauncher.main() → CobarServer.new() → CobarConfig.new() → CobarConfig.initCobarConfig() → ServerLoader.load()

The parameters of CN are all in the form of key-value.

In ServerLoader.load(), CN reads parameters mainly from the following locations:

The default location of the server.properties file is in the root directory of the classpath, so when IDE is used for development, the polardbx-serversrcmainresourcesserver.properties file is loaded.

String conf = System.getProperty("server.conf", "classpath:server.properties");

When Java is running parameters

For example, java ... -DserverPort=8527

In the environment variable

serverProps.putAll(System.getProperties());
serverProps.putAll(System.getenv());

Among the preceding parameter sources, server.properties has the lowest priority, while environment variables have the highest priority. If a parameter with the same name is included, the parameter source with the higher priority overwrites the one with the lower priority.

The loaded parameter is saved in the SystemConfig. This class is a singleton and can be obtained by CobarServer.getInstance().getConfig().getSystem().

Read Metadata from MetaDB and Initialize Instance-Level System Components

Path: TddlLauncher.main() → CobarServer.new() → CobarConfig.new() → CobarConfig.initCobarConfig() → ServerLoader.load() → ServerLoader.initPolarDbXComponents()

Initialize the Connection Pool of the Metadatabase

MetaDbDataSource.initMetaDbDataSource uses the information of MetaDB (including address, port, username, password, and database name stored in the SystemConfig) to build a connection with MetaDB.

MetaDbDataSource
            .initMetaDbDataSource(this.system.getMetaDbAddr(), this.system.getMetaDbName(),
                this.system.getMetaDbProp(),
                this.system.getMetaDbUser(),
                this.system.getMetaDbPasswd());

MetaDbDataSource is a singleton. You can use MetaDbDataSource.getInstance().getConnection() to obtain a connection with MetaDB (JDBC interface is implemented) n a program that implements JDBC and use this connection to access MetaDB.

Create or Upgrade System Tables

SchemaChangeManager.getInstance().handle();

The table structure of system tables is saved in polardbx-gmssrcmainresourcesddl, and the alter statement is used to record the changes of each version. The SchemaChangeManager detects the table structure version of each system table during initialization. If the version is old, it uses these statements to update the alter statement in turn.

Read ID Information of Instance

ServerInstIdManager.getInstance();

A PolarDB-X cluster is called an instance and has a unique instance ID in MetaDB. PolarDB-X instances have two modes: primary instance and read-only instance. If a read-only instance exists, a primary instance must exist. This step reads the instance ID from MetaDB. If the current instance is read-only, the instance ID of the primary instance is read.

MetaDbConfigManager

What does MetaDbConfigManager do?

If the configuration information stored in MetaDB changes, the CN needs to be able to perceive the change. For example, a column is added to a table. How can CN perceive this change?

A relatively simple idea is to poll the metadata once every few seconds. However, if each module is allowed to do so, a big problem will appear; the access pressure to MetaDB will be great.

MetaDbConfigManager encapsulated it and made a unified polling mechanism. However, due to the ever-changing format of each module metadata table, MetaDbConfigManager does not poll the metadata table of each module but polls the config_listener table. The key columns in this table are data_id, op_version, and gmt_modified.

We can register a listener in a code for each data_id. When the MetaDbConfigManager polls that the op_version of data_id changes, the listener will be called back. Generally, the listener implemented by each module will read the corresponding metadata table on demand.

For example, the d1.t1 table has a row of records in the config_listener:

polardbx.meta.table.d1.t1

If a column is added to the t1 table, we will modify the columns table in MetaDB. At the same time, we will modify the op_version of the polardbx.meta.table.d1.t1 row record in the config_listener table to perform a +1 operation.

After a few seconds, the MetaDbConfigManager will poll that the op_version column of this row of records has changed. It will call back the listener (class: TableMetaListener) of the table structure management module, which will reload the metadata of the d1.t1 table.

Therefore, MetaDbConfigManager can only make one thread of each CN node polling MetaDB to sense the change of each metadata table.

At startup, the CN initializes MetaDbConfigManager to complete threads and scheduled tasks used for polling.

MetaDbInstConfigManager

Initialize instance-level configuration items similar to System Variables in MySQL, such as the memory size limit of SQL. In this phase, all configuration items are loaded from the inst_config table of MetaDB and stored in memory.

The corresponding listener is registered. When the inst_config table changes, the MetaDbInstConfigManager listener is called back.

ConnPoolConfigManager

It is used to store the configuration items related to the connection pool (the one between CN and DN; the configuration used here can be found in MetaDbInstConfigManager ), and it is read from inst_config.

StorageHaManager

Currently, the results of the election within each group DN (here, the group refers to a Paxos group) are stored in the system table of the group DN and are not written to MetaDB. Therefore, CN needs to have a mechanism to detect the role of each node from the system table of DN.

The StorageHaManager reads the connection information of all DN nodes from the storage_info table and has a polling thread to detect the role information. When HA occurs in the DN, the StorageHaManager will detect this change and perceive the latest Leader and other roles.

Initialize the System Library

initSystemDbIfNeed initializes several system libraries (such as information_schema and polardbx).

Create a Thread Pool

Path: CobarServer.new()

The creation of several major thread pools:

The managerExecutor is responsible for requests to the Manager port.
The killExecutor is specifically used to execute kill commands.
The serverExecutor is a thread pool where every SQL is executed.

CobarServer.init

Path: TddlLauncher.main() -> CobarServer.init()

Initialization of a Logical Library (TDataSource)

The entry is GmsAppLoader.initDbUserPrivsInfo, and the App in the class refers to a logical library.

The user permission information for each logical library and the most important TDataSouce are loaded here.

The TDataSource corresponds to the logical libraries in the CN. It is the entry for each logical library to execute SQL.

The initialization logic of the TDatSource is in the MatrixConfigHolder.init, which contains the following contents:

Initialize the topology, such as which DNs this library contains, and initialize the connection pool (TopologyHandler.initPolarDbXTopology) between these DNs
Get information about each DN, including version, support for features, etc. (StorageInfoManager.init)
Initialize the routing information of the shard (mode='drds', TddlRuleManager.init ;mode='auto', PartitionInfoManager.init)
Initialize the table manager (which tables are available, which columns and indexes are available for each table, etc.)(GmsTableMetaManager.init)
Initialize the transaction manager (transactionManager.init())
Create a Plan Cache (PlanCache planCache = new PlanCache(schemaName);)
Start the DDL task engine (ddlEngineInit())
Initialization of the statistic manager (StatisticManager.init)
Initialization of SPM (PlanManager.init)

GmsClusterLoader.loadPolarDbXCluster

Load cluster information, such as which CN nodes are in the cluster

The initialization CclService is used for SQL throttling.

Warmup

The SQL function of CN is currently loaded by the reflection mechanism, and the main job is to load all functions.

Initialization of Network Layer

After this step is completed, the service port will be opened, and the CN process can provide services to the outside.

CN will start the Server port and Manager port in this step.

The Server port corresponds to the 3306 port of MySQL and is provided for frontend applications.

The Manager port is used for internal management. For example, if the SHOW PROCESSLIST command needs to collect the execution state of all CN, it will use the port for collection.

NIOProcessor

processors = new NIOProcessor[system.getProcessors()];
for (int i = 0; i < processors.length; i++) {
      processors[i] = new NIOProcessor(i, "Processor" + i,this.serverExecutor);
processors[i].startup();
}

The entry for the CN Network Layer to process requests is N IOProcessor. Each NIOProcessor has two threads of reading and writing. We start the NIOProcessor W and R threads here.

The NIOProcessor is only responsible for network-related processing. If an SQL request is received, subsequent SQL execution is handed over to the ServerExecutor thread pool for execution.

NIOAcceptor

NIOAcceptor in the CN Network Layer is used to process the request for connection establishment. The port will be opened for listening during the new process of NIOAcceptor:

public NIOAcceptor(String name, int port, FrontendConnectionFactory factory, boolean online) throws IOException {
        super.setName(name);
        ...
            this.selector = Selector.open();
            this.serverChannel = ServerSocketChannel.open();
            this.serverChannel.socket().bind(new InetSocketAddress(port), 65535);
        ...
}

The NIOAcceptor is also a thread that processes the requests for connection establishment. When the connection is established, the NIOAcceptor.accept binds the connection to a NIOProcessor, and the NIOProcessor continues to process subsequent read and write requests in the connection.

There is only one NIOAcceptor for service ports, and the number of NIOProcessor is the same as the CPU cores.

The Start of an MPP Server

As one of the MPP clusters, the CN needs to start a port for communication between CNs.

The startMppServer starts the MPP service.

Start of a CDC Service

CobarServer.tryStartCdcManager starts the CDC.

Summary

CN has been started. This process includes most of the components in CN. Due to the limited length of the article, the role of each component is not fully introduced. If you are interested in these components, please leave us a message. We will introduce some key components in detail in the following articles.

Community