Platform For AI:Estimate GPU memory for LLMs

2.1 Model parameters

During model inference, the model parameters must be stored. The formula to calculate the GPU memory they occupy is: Number of parameters × Parameter precision. Common precisions are FP32 (4 bytes), FP16 (2 bytes), and BF16 (2 bytes). For LLMs, parameters are typically stored in FP16 or BF16. For example, for a 7B model with FP16 precision, the required GPU memory is:

$$\frac{7\times 10^9\times 2}{10^9}=14\text{GB}$$

2.2 Activation values

During LLM inference, the activation value of each neuron layer must be calculated. The GPU memory this occupies is positively correlated with the batch size, sequence length, and model architecture, such as the number of layers and hidden layer size. The relationship can be expressed as: $$Activation\; value\; memory\propto b\times s\times h\times L\times param\_bytes$$

Where:

• b (batch size): The batch size for a single request. This is typically 1 for online services but can be greater than 1 for batch processing.

• s (sequence length): The total sequence length, including input and output (number of tokens).

• h (hidden size): The hidden layer dimension of the model.

• L (Layers): The number of Transformer layers in the model.

• param_bytes: The precision for storing activation values, typically 2 bytes.

Based on these factors and practical experience, you can simplify the estimation. For a 7B model with a batch size (b) of 1, a sequence length (s) of 2048, and param_bytes of 2, the memory for activation values can be roughly estimated as 10% of the model's memory footprint. This provides a safe margin. The calculation is: $$14\text{GB}\times 0.1=1.4\text{GB}$$.

2.3 KV cache

To accelerate LLM inference, the key (K) and value (V) matrices calculated for each Transformer layer are typically cached. This avoids recalculating the attention mechanism parameters for all previous tokens at each time step. Using a KV cache reduces the computational complexity from $$O(n^2)$$ to $$O(n)$$, which significantly improves the inference speed. Similar to activation values, the GPU memory occupied by the KV cache is positively correlated with the batch size, sequence length, concurrency, and model architecture, such as the number of layers and hidden layer size. The relationship can be expressed as: $$KVcache\; memory\propto 2\times b\times s\times h\times L\times C\times param\_bytes$$

Where:

• 2: Represents the two matrices that need to be stored: K (Key) and V (Value).

• b (batch size): The batch size for a single request. This is typically 1 for online services but can be greater than 1 for batch processing.

• s (sequence length): The total sequence length, including input and output (number of tokens).

• h (hidden size): The hidden layer dimension of the model.

• L (Layers): The number of Transformer layers in the model.

• C (Concurrency): The concurrency of service interface requests.

• param_bytes: The precision for storing activation values, typically 2 bytes.

Based on these factors and practical experience, you can simplify the estimation. For a 7B model with a concurrency (C) of 1, a batch size (b) of 1, a sequence length (s) of 2048, and param_bytes of 2, the memory for the KV cache can be roughly estimated as 10% of the model's memory footprint. This provides a safe margin. The value is: $$1.4\text{GB}$$.

2.4 Other factors

In addition to the factors above, the input data for the current batch, CUDA cores, and the deep learning framework itself, such as PyTorch or TensorFlow, also consume some GPU memory, typically 1 GB to 2 GB.

3.1 Model parameters

During fine-tuning, the model parameters must be stored. The formula to calculate the GPU memory they occupy is: Number of parameters × Parameter precision. Common precisions are FP32 (4 bytes), FP16 (2 bytes), and BF16 (2 bytes). For LLM fine-tuning, parameters are typically stored in FP16 or BF16. For example, for a 7B model with FP16 precision, the required GPU memory is:

$$\frac{7\times 10^9\times 2}{10^9}=14\text{GB}$$

3.2 Gradient parameters

During the backward propagation phase of model training, gradients must be calculated for the model parameters. The number of gradients is the same as the number of parameters being trained. In LLMs, gradients are typically stored with 2-byte precision. Therefore, for a 7B model, the GPU memory required for gradients varies based on the fine-tuning method:

3.3 Optimizer state

During training, the state of the optimizer must also be saved. The number of state values is related to the number of trainable parameters. Additionally, models often use mixed-precision training. This means model parameters and gradients are stored in 2-byte precision, while the optimizer state is stored in 4-byte precision. This practice ensures high precision during parameter updates and prevents numerical instability or overflow caused by the limited dynamic range of FP16/BF16. Storing the state in 4-byte precision doubles the required GPU memory. The following table describes common optimizers:

3.4 Activation values

During training, intermediate activation values from the forward propagation pass must be stored to calculate gradients during backward propagation. This memory consumption is positively correlated with the batch size, sequence length, and model architecture, such as the number of layers and hidden layer size. The relationship can be expressed as: $$Activation\; value\; memory\propto b\times s\times h\times L\times param\_bytes$$

Where:

• b (batch size): The batch size.

• s (sequence length): The total sequence length, including input and output (number of tokens).

• h (hidden size): The hidden layer dimension of the model.

• L (Layers): The number of Transformer layers in the model.

• param_bytes: The precision for storing activation values, typically 2 bytes.

Based on these factors and practical experience, you can simplify the estimation. For a 7B model with a batch size (b) of 1, a sequence length (s) of 2048, and param_bytes of 2, the memory for activation values can be roughly estimated as 10% of the model's memory footprint. This provides a safe margin. The calculation is: $$14\text{GB}\times 0.1=1.4\text{GB}$$.

3.5 Other factors

In addition to the factors above, the input data for the current batch, CUDA cores, and the deep learning framework itself, such as PyTorch or TensorFlow, also consume some GPU memory, typically 1 GB to 2 GB.

Q: How do I check the number of parameters in an LLM?

A: For open source LLMs, the number of parameters is usually indicated in the model name. For example, Qwen-7B has $$7\times 10^9$$ parameters. Qwen3-235B-A22B has a total of $$235\times 10^9$$ parameters, with $$22\times 10^9$$ parameters activated during inference. For models that do not specify the parameter count in their name, you can search for and review the model's documentation to find this information.

Q: How do I check the parameter precision of an LLM?

A: Unless otherwise specified, LLMs typically use 16-bit (2-byte) storage. For quantized models, 8-bit or 4-bit storage may be used. For more information, see the model's documentation. For example, if you use a model from the PAI Model Gallery, the product page usually describes the parameter precision:

Qwen2.5-7B-Instruct training instructions:

Q: How do I check the optimizer and its state precision for LLM fine-tuning?

A: LLM training typically uses the Adam/AdamW optimizer with 32-bit (4-byte) parameter precision. For more detailed configurations, you can check the start command or the code.

Q: How do I check GPU memory usage?

A: You can view GPU memory usage on the graphical monitoring pages of PAI-DSW, PAI-EAS, or PAI-DLC:

Alternatively, you can run the nvidia-smi command in the container's terminal to check GPU usage:

Q: What are common "out of memory" errors?

A: When you run out of GPU memory on an NVIDIA GPU, a CUDA out of memory. Tried to allocate X GB error occurs. When this happens, you must increase the GPU memory or reduce parameters, such as the batch size or sequence length.