Elizabeth
Engineer
Engineer
  • UID625
  • Fans2
  • Follows1
  • Posts68
Reads:1050Replies:0

PHP source code learning - thread-safe

Created#
More Posted time:Oct 13, 2016 9:45 AM
Before we talk about thread-safe, let's review some basic knowledge which constitutes the foundation for future analysis and study.
Scope of variables
In terms of scope, C language can define four types of variables: global variables, static global variables, local variables and static local variables.
Next, we will analyze the different variables from their functional scopes. Assume all variables are declared without duplicate names.
• Global variables (int gVar;), declared outside the function. Global variables are shared by all the functions. A global variable is unique.
• Static global variables (static sgVar) are actually also shared by all functions. But they are limited by the compiler and can be regarded as a function provided by the compiler.
• Local variables (int var within the function/block;) are not shared. The local variables involved in multiple executions of the function are independent from each other, that is, they are different variables only with the same name.
• Static local variables (static int sVar within the function;) are shared within the function. The variable involved in every execution of the function refers to the same variable.
The scope above is defined from the perspective of functions and can cover all the sharing conditions of variables in single-thread programming.  Now let's analyze the multi-thread scenarios.
In a multi-thread scenario, multiple threads share the resources except the function call stack.  So the definition of the above scopes will be changed to:
• Global variables are shared by all the functions, thus shared by all the threads. A global variable is unique among multiple threads.
• Static global variables are also shared by all functions, thus shared by all threads.
• Local variables are not shared. The local variables involved in multiple executions of the function are independent from each other, that is, they are not shared among various threads.
• Static local variables are shared within the function. The variable involved in every execution of the function refers to the same variable. So they are shared among various threads.
Origin of thread-safe resource manager
In a multi-thread system, the processes retain the resource ownership attributes and the multiple concurrent execution flow executes the threads running in the process.  For example, in Apache2's worker, the primary controlling process generates multiple sub-processes and each sub-process contains a fixed number of threads. Each thread processes requests independently.  Similarly, to avoid generating more threads when requests arrive, MinSpareThreads and MaxSpareThreads set the minimum and maximum number of idle threads, while MaxClients sets the total number of threads in all sub-processes. If the total number of threads in the existing sub-processes cannot meet the load demand, the controlling process will derive new sub-processes.
When PHP is run on a multi-thread server like the above one, the PHP at this time is in a multi-thread lifecycle.  In a certain period of time, multiple threads may exist in a process space and multiple threads in a process share the global variables after module initialization. If you run the script like you do in PHP in CLI mode, multiple threads will read and write some public resources stored in the process memory (For example, many global variables may exist out of the functions after module initialization shared by multiple threads).
At this time, the memory address space that these threads access are the same. The change of a thread will affect the other threads. Such sharing will improve the operation speed to some extent, but a huge coupling will occur between multiple threads, and when multiple threads are concurrent, the frequent problems of data consistency or resource competition will emerge, such as inconsistency in the result after multiple runnings with that after running a single thread. If there are only read operations in each thread to the global variables or static variables, without write operations, the global variables are thread-safe. But this is not the reality of the situation.
To solve the thread concurrency issue, PHP introduces TSRM:  Thread Safe Resource Manager.  The implementation code of TRSM is in the /TSRM directory of PHP source code and is called everywhere. We usually call it the TSRM layer.  Generally speaking, TSRM layer is only enabled for compilation when specified (for example, Apache2+worker MPM, a thread-based MPM). Because Apache in Win32 environment is based on multiple threads, this layer is always enabled in Win32 environment.
Implementation of TSRM
The process retains the attribute of resource ownership, and threads make concurrent accesses. The introduction of the TSRM layer in PHP focuses on the access to shared resources. The shared resources here refer to the global variables shared between threads and stored in the memory space of processes.  In the single thread mode in PHP, when a variable is declared to be out of all functions, it becomes a global variable.
First, several very important global variables are defined as follows (the global variables here are shared by multiple threads).
/* The memory manager table */
static tsrm_tls_entry   **tsrm_tls_table=NULL;
static int              tsrm_tls_table_size;
static ts_rsrc_id       id_count;

/* The resource sizes table */
static tsrm_resource_type   *resource_types_table=NULL;
static int                  resource_types_table_size;


**Tsrm_tls_table, thread safe resource manager thread local storage table in full, is used to store the tsrm_tls_entry chain tables of various threads.
tsrm_tls_table_size indicates the size of **tsrm_tls_table.
id_count is the ID generator for global variable resources. It is globally unique and incremental.
*resource_types_table is used to store the resources for the global variables.
resource_types_table_size indicates the size of *resource_types_table.
It involves two key data structures: tsrm_tls_entry and tsrm_resource_type.
typedef struct _tsrm_tls_entry tsrm_tls_entry;

struct _tsrm_tls_entry {
    void **storage;// The global variable arrays in the node
    int count;// The global variable count in the node
    THREAD_T thread_id;// The thread ID of the node
    tsrm_tls_entry *next;// The pointer of the next node
};

typedef struct {
    size_t size;// The size of the struct of the defined global variable
    ts_allocate_ctor ctor;// The constructor pointer of the defined global variable
    ts_allocate_dtor dtor;// The destructor pointer of the defined global variable
    int done;
} tsrm_resource_type;


When a new global variable is added, id_count will increase by 1 (with the thread mutex lock added). Then according to the memory, constructor and destructor required by the global variable, the tsrm_resource_type of resources is generated and stored to *resource_types_table. Based on the resource, the global variable for all the tsrm_tls_entry nodes of every thread is added.
With the above rough understandings, we can further tap to the entire process through analyzing the TSRM environment initialization and resource ID allocation.
TSRM environment initialization
In the module initialization phase, tsrm_startup is called in various SAPI main functions to initialize the TSRM environment. The tsrm_startup function will pass in two crucial parameters: expected_threads, indicating the expected thread count, and expected_resources, indicating the expected resource count. Different SAPIs have different initialization values. For example, mod_php5 and cgi have one resource for one thread.
TSRM_API int tsrm_startup(int expected_threads, int expected_resources, int debug_level, char *debug_filename)
{
    /* code... */

    tsrm_tls_table_size = expected_threads; // The expected thread count allocated at SAPI initialization. The value is usually 1.

    tsrm_tls_table = (tsrm_tls_entry **) calloc(tsrm_tls_table_size, sizeof(tsrm_tls_entry *));

    /* code... */

    id_count=0;

    resource_types_table_size = expected_resources; // The expected resource count allocated at SAPI initialization. The value is usually 1.

    resource_types_table = (tsrm_resource_type *) calloc(resource_types_table_size, sizeof(tsrm_resource_type));

    /* code... */

    return 1;
}


Three key jobs can be summarized: the tsrm_tls_table chain table, the resource_types_table array and id_count are initialized. The three global variables are shared by all threads, achieving memory management consistency between threads.
Resource ID allocation
We know that the ZEND_INIT_MODULE_GLOBALS macro is required for initializing a global variable (this will be illustrated in the array expansion example below), but in fact, what is called is a global variable applied by the ts_allocate_id function in a multi-thread environment, and the resource ID allocated will be returned. Although there is a lot of code, it is generally clear. Next we will explain the code with annotations:
TSRM_API ts_rsrc_id ts_allocate_id(ts_rsrc_id *rsrc_id, size_t size, ts_allocate_ctor ctor, ts_allocate_dtor dtor)
{
    int i;

    TSRM_ERROR((TSRM_ERROR_LEVEL_CORE, "Obtaining a new resource id, %d bytes", size));

    // Add the multiple thread mutex lock
    tsrm_mutex_lock(tsmm_mutex);

    /* obtain a resource id */
    *rsrc_id = TSRM_SHUFFLE_RSRC_ID(id_count++); // Increase the global static variable id_count by 1.
    TSRM_ERROR((TSRM_ERROR_LEVEL_CORE, "Obtained resource id %d", *rsrc_id));

    /* store the new resource type in the resource sizes table */
    // Because resource_types_table_size has an initial value (expected_resources), it is not necessary to expand memory every time.
    if (resource_types_table_size < id_count) {
        resource_types_table = (tsrm_resource_type *) realloc(resource_types_table, sizeof(tsrm_resource_type)*id_count);
        if (!resource_types_table) {
            tsrm_mutex_unlock(tsmm_mutex);
            TSRM_ERROR((TSRM_ERROR_LEVEL_ERROR, "Unable to allocate storage for resource"));
            *rsrc_id = 0;
            return 0;
        }
        resource_types_table_size = id_count;
    }

    // Store the size, constructor and destructor of the global variable struct to the resource_types_table array of tsrm_resource_type.
    resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].size = size;
    resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].ctor = ctor;
    resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].dtor = dtor;
    resource_types_table[TSRM_UNSHUFFLE_RSRC_ID(*rsrc_id)].done = 0;

    /* enlarge the arrays for the already active threads */
    // PHP core will traverse all the threads, and allocate the required memory space for global variables of this thread for the tsrm_tls_entry of every thread.
    for (i=0; i<tsrm_tls_table_size; i++) {
        tsrm_tls_entry *p = tsrm_tls_table;

        while (p) {
            if (p->count < id_count) {
                int j;

                p->storage = (void *) realloc(p->storage, sizeof(void *)*id_count);
                for (j=p->count; j<id_count; j++) {
                    // Allocate the required memory space for global variables in the thread
                    p->storage[j] = (void *) malloc(resource_types_table[j].size);
                    if (resource_types_table[j].ctor) {
                        // Initialize the global variable stored in the p->storage[j] address
                        // Here, the second parameter of ts_allocate_ctor function is reserved for unknown reason, and it is not used throughput the whole project. In PHP7, the second parameter is indeed removed.
                        resource_types_table[j].ctor(p->storage[j], &p->storage);
                    }
                }
                p->count = id_count;
            }
            p = p->next;
        }
    }

    // Cancel the thread mutex lock
    tsrm_mutex_unlock(tsmm_mutex);

    TSRM_ERROR((TSRM_ERROR_LEVEL_CORE, "Successfully allocated new resource id %d", *rsrc_id));
    return *rsrc_id;
}


When allocating global resource ID through the ts_allocate_id function, the PHP core will first add the mutex lock to ensure the generated resource ID is unique. The role of the lock here is to serialize concurrent content in the time dimension, because the fundamental issue of concurrency is the issue of time. After a lock is added, the id_count will increase to generate a resource ID. After the resource ID is generated, the current resource ID will be allocated with a storage location. Every resource will be stored in the resource_types_table. When a new resource is allocated, a new tsrm_resource_type will be created.
All the tsrm_resource_types constitute the tsrm_resource_table in the form of array, and their subscripts are the resource ID.
Actually we can regard the tsrm_resource_table as a HASH table, the key is the resource ID, and the value is the tsrm_resource_type structure (Any array can be regarded as a HASH table, if the array key value is meaningful).
After the resource ID is allocated, PHP core will traverse **all the threads** and allocate the required memory space for global variables of this thread for the tsrm_tls_entry of every thread.
The sizes of global variables of every thread is specified where they are called (that is, the size of the global variable struct). At last, the global variable stored in the address is initialized.
I also draw a picture to illustrate this further.


In the picture above, how are tsrm_tls_table elements added and how is the chain table implemented? We can leave this question open to further discussions later.
For every ts_allocate_id call, PHP core will traverse all threads and allocate resources needed for every thread.
If the operation is performed during the request processing phase in the PHP lifecycle, isn't it called repeatedly?
PHP has taken this situation into consideration. The ts_allocate_id calls are done at module initialization.
After the TSRM is started, it will traverse the initialization methods of every extension module during the module initialization phase.
The extension global variables are declared in the header of the implementation code for the extensions and initialized in the MINIT method.
The global variables applied by the TSRM and their size will be informed at the initialization. Here the inform operation is actually the ts_allocate_id function mentioned above.
The TSRM is allocated and registered in the memory pool and then the resource ID is returned to the extension.
Use of global variables
Taking the standard array extension as an example, first the global variables of the extension will be declared.
ZEND_DECLARE_MODULE_GLOBALS(array)

Then at the module initialization, the initialization macro of the global variables will be called to initialize the array, such as allocating memory space.
static void php_array_init_globals(zend_array_globals *array_globals)
{
    memset(array_globals, 0, sizeof(zend_array_globals));
}

/* code... */

PHP_MINIT_FUNCTION(array) /* {{{ */
{
    ZEND_INIT_MODULE_GLOBALS(array, php_array_init_globals, NULL);
    /* code... */
}


The declaration and initialization operations can be divided to ZTS and non-ZTS operations.
#ifdef ZTS

#define ZEND_DECLARE_MODULE_GLOBALS(module_name)                            \
    ts_rsrc_id module_name##_globals_id;

#define ZEND_INIT_MODULE_GLOBALS(module_name, globals_ctor, globals_dtor)   \
    ts_allocate_id(&module_name##_globals_id, sizeof(zend_##module_name##_globals), (ts_allocate_ctor) globals_ctor, (ts_allocate_dtor) globals_dtor);

#else

#define ZEND_DECLARE_MODULE_GLOBALS(module_name)                            \
    zend_##module_name##_globals module_name##_globals;

#define ZEND_INIT_MODULE_GLOBALS(module_name, globals_ctor, globals_dtor)   \
    globals_ctor(&module_name##_globals);

#endif


For non-ZTS operations, the variables are declared directly and initialized; for ZTS operations, PHP core will add TSRM, replacing declaring global variables with ts_rsrc_id, and at initialization, it is not initializing variables, but calling the ts_allocate_id function to apply for a global variable for the current module in the multi-thread environment and return the resource ID. The resource ID variable name is composed by the module name and the global_id.
If you want to call the global variable of the current extension, you can use: ARRAYG(v). The definition of this macro is:
#ifdef ZTS
#define ARRAYG(v) TSRMG(array_globals_id, zend_array_globals *, v)
#else
#define ARRAYG(v) (array_globals.v)
#endif


For non-ZTS operations, the attribute field of the global variable is called directly. For ZTS operations, TSRMG is used to get the variable.
Definition of TSRMG:
#define TSRMG(id, type, element) (((type) (*((void ***) tsrm_ls))[TSRM_UNSHUFFLE_RSRC_ID(id)])->element)

Remove the brackets. The TSRMG macro means to get the global variable from tsrm_ls by the resource ID and return the attribute fields of the corresponding variable.
So now the question is: where does this tsrm_ls come from?
tsrm_ls initialization
The tsrm_ls is initialized through ts_resource(0). Expand the code, we can see that ts_resource_ex(0,NULL) is called actually. Expand the macros of ts_resource_ex and take the thread pthread as an example.
#define THREAD_HASH_OF(thr,ts)  (unsigned long)thr%(unsigned long)ts

static MUTEX_T tsmm_mutex;

void *ts_resource_ex(ts_rsrc_id id, THREAD_T *th_id)
{
    THREAD_T thread_id;
    int hash_value;
    tsrm_tls_entry *thread_resources;

    // tsrm_tls_table has completed initialization during tsrm_startup
    if(tsrm_tls_table) {
        // At the initialization, th_id = NULL;
        if (!th_id) {

            //Empty at the first time. The pthread_setspecific is not executed yet so thread_resources pointer is blank
            thread_resources = pthread_getspecific(tls_key);

            if(thread_resources){
                TSRM_SAFE_RETURN_RSRC(thread_resources->storage, id, thread_resources->count);
            }

            thread_id = pthread_self();
        } else {
            thread_id = *th_id;
        }
    }
    // Implement the lock
    pthread_mutex_lock(tsmm_mutex);

    // Get the remainder directly and make the value the subscript of the array. Hash different threads in the tsrm_tls_table.
    hash_value = THREAD_HASH_OF(thread_id, tsrm_tls_table_size);
    // After the SAPI calls tsrm_startup, tsrm_tls_table_size = expected_threads
    thread_resources = tsrm_tls_table[hash_value];

    if (!thread_resources) {
        // If it hasn’t been done yet, allocate the resource again.
        allocate_new_resource(&tsrm_tls_table[hash_value], thread_id);
        // After the allocation is done, execute the else interval below
        return ts_resource_ex(id, &thread_id);
    } else {
         do {
            // Match them one by one along the chain table
            if (thread_resources->thread_id == thread_id) {
                break;
            }
            if (thread_resources->next) {
                thread_resources = thread_resources->next;
            } else {
                // If the end of the chain table is not found, allocate the resource again and connect to the end of the chain table.
                allocate_new_resource(&thread_resources->next, thread_id);
                return ts_resource_ex(id, &thread_id);
            }
         } while (thread_resources);
    }

    TSRM_SAFE_RETURN_RSRC(thread_resources->storage, id, thread_resources->count);

    // Unlock
    pthread_mutex_unlock(tsmm_mutex);

}


The allocate_new_resource is used to allocate memory for new threads in the corresponding chain table and add all the global variables to the storage pointer array.
static void allocate_new_resource(tsrm_tls_entry **thread_resources_ptr, THREAD_T thread_id)
{
    int i;

    (*thread_resources_ptr) = (tsrm_tls_entry *) malloc(sizeof(tsrm_tls_entry));
    (*thread_resources_ptr)->storage = (void **) malloc(sizeof(void *)*id_count);
    (*thread_resources_ptr)->count = id_count;
    (*thread_resources_ptr)->thread_id = thread_id;
    (*thread_resources_ptr)->next = NULL;

    // Set local storage variable of the thread. After setting is done here, then from ts_resource_ex, obtain pthread_setspecific(*thread_resources_ptr);

    if (tsrm_new_thread_begin_handler) {
        tsrm_new_thread_begin_handler(thread_id, &((*thread_resources_ptr)->storage));
    }

    for (i=0; i<id_count; i++) {
        if (resource_types_table.done) {
            (*thread_resources_ptr)->storage = NULL;
        } else {
            // Add resource_types_table resources for newly added tsrm_tls_entry nodes.
            (*thread_resources_ptr)->storage = (void *) malloc(resource_types_table.size);
            if (resource_types_table.ctor) {
                resource_types_table.ctor((*thread_resources_ptr)->storage, &(*thread_resources_ptr)->storage);
            }
        }
    }

    if (tsrm_new_thread_end_handler) {
        tsrm_new_thread_end_handler(thread_id, &((*thread_resources_ptr)->storage));
    }

    pthread_mutex_unlock(tsmm_mutex);
}


There is a knowledge point above, the Thread Local Storage. Now we have a global variable tls_key. All threads can use it and change its value.
It looks like a global variable for all threads to use, but its value in every thread is stored independently. This is the significance of local storage of the thread.
So how can we implement local storage of the thread?
The tsrm_startup, ts_resource_ex and allocate_new_resource functions should be used in combination along with some annotations to illustrate this:
// Take pthread for example
// 1. First the tls_key global variable is defined
static pthread_key_t tls_key;

// 2. Then call pthread_key_create() at tsrm_startup to create the variable
pthread_key_create( &tls_key, 0 );

// 3. In allocate_new_resource, store *thread_resources_ptr pointer variable to the global variable tls_key through tsrm_tls_set
tsrm_tls_set(*thread_resources_ptr);// Expand it and it will be pthread_setspecific(*thread_resources_ptr);

// 4. In ts_resource_ex, use tsrm_tls_get() to obtain *thread_resources_ptr that is set in the thread *thread_resources_ptr
//    In concurrent operations of multiple threads, they will not interfere with each other.
thread_resources = tsrm_tls_get();


After understanding tsrm_tls_table array and chain table creation, let’s look at the return macro that the ts_resource_ex function calls.
#define TSRM_SAFE_RETURN_RSRC(array, offset, range)     \
    if (offset==0) {                                    \
        return &array;                                  \
    } else {                                            \
        return array[TSRM_UNSHUFFLE_RSRC_ID(offset)];   \
    }


It means to, according to the incoming tsrm_tls_entry and the offset of the storage array subscript, return the address of the global variable in the storage array in the thread. By now you should have understood the definition of the TSRMG macro that obtains global variables in the multiple threads.
Actually it is often used during our writing extensions:
#define TSRMLS_D void ***tsrm_ls   /* Without commas. Usually use */ in definition when it is a unique parameter.
#define TSRMLS_DC , TSRMLS_D       /* is also used in definition, but as there are other parameters before it, so a comma is required */
#define TSRMLS_C tsrm_ls
#define TSRMLS_CC , TSRMLS_C


Notice: During extension writing, maybe many people don't know which one to use. Through macro expansion, we can see that there may be commas or no commas, as well as declaration and calls. In English, “D” stands for Define and “C” stands for Comma, and the preceding “C” stands for Call.
The above is the definition in the ZTS mode and the definition in non-ZTS mode is entirely empty.
Guest