Filters and transitions - Apsara Video SDK - Alibaba Cloud Documentation Center

The short video SDK allows you to create custom effects. You can use a custom OpenGL ES shader that is created with OpenGL ES 3.0 to obtain desired filters and transitions based on the effect configuration file. This topic does not cover know-how about OpenGL ES. We assume that you have a basic understanding of OpenGL ES and shading languages.

Description

A filter or transition resource package folder contains an effect configuration file named config.json and some picture materials. When you edit or record a video by using the short video SDK, you can use transitions and filters. To apply a filter or transition, specify a configured resource package folder when you call the related editing or recording operations.

Configure the effect configuration file

The effect configuration file is in the JSON format and describes a complete rendering process. The effect configuration file has two levels. The first level describes basic information about the effect, and the second level uses a node tree to describe how the effect is implemented.

First level - basic information

The following table describes the basic information about an effect.

Parameter	Description
name	The name of the effect.
module	The module identifier. Set the value to `ALIVC_GECF`.
version	The version number. Value: 1.
type	The type of the effect. Valid values: 1 and 2. A value of 1 indicates a filter, and a value of 2 indicates a transition.
nodeTree	The node tree that is used to describe how the effect is implemented. For more information, see the Second level - node tree section of this topic.

The following code shows how the configuration file of the Intense filter is configured:

{
 "name": "Intense",
 "module": "ALIVC_GECF",
 "version": 1,
 "type": 1,
 "nodeTree": [
 {
 "nodeId": 0,
 "name": "Intense",
 "fragment": "/** ...... */",
 "textures": [
 {
 "name": "inputImageTexture",
 "srcType": "INPUT_NODE",
 "nodeId": 0
 },
 {
 "name": "inputImageTexture2",
 "srcType": "IMAGE",
 "path": "color.png"
 }
 ]
 }
 ]
}

Second level - node tree

The node tree contains one or more nodes and is used to describe a rendering process.

The following figure shows the rendering process in the short video SDK. 滤镜及特效.png

As shown in the preceding figure, the rendering process is abstracted into a tree structure that consists of a series of nodes.

The input node INPUT_NODE represents the input source. When you record a video, the input node is the image data stream collected by the camera. When you edit a video, the input node is the current video stream. In design, multiple input nodes can exist. For example, if you need to apply a transition between two video clips, the first video clip is INPUT_NODE0 and the second video clip is INPUT_NODE1.
The node tree in the middle corresponds to the nodeTree parameter in the effect configuration file. A node tree can contain one or more rendering nodes. As you can see, the key to the effect configuration file is the configuration of rendering nodes.
The output node OUTPUT_NODE represents the rendered video stream. In preview mode, the output node generates content on the screen. In production mode, the output node generates content in the encoder.

Node

A node specifies the relevant configurations in a drawing process during rendering. The configurations contain the shader code and related parameter descriptions that are required for a custom effect. The following table describes the parameters that define a node.

Parameter	Description
nodeId	The ID of the node.
name	The name of the node.
vertex	The vertex shader code. This parameter has the same function as the vertexPath parameter. You can declare either one of the two parameters. This parameter is optional. If you do not specify this parameter, the short video SDK performs the following implementation by default: `attribute vec4 position; attribute vec4 inputTextureCoordinate; varying vec2 textureCoordinate; void main() { gl_Position = position; textureCoordinate = inputTextureCoordinate.xy; }`
vertexPath	The path to the file in which the vertex shader code resides. This parameter has the same function as the vertex parameter. You can declare either one of the two parameters.
attributes	The list of attributes. This parameter is used to declare the names and types of attributes in vertex shading. Valid values of type: POSITION and TEXTURECOORD. A value of POSITION indicates the vertex coordinate, and a value of TEXTURECOORD indicates the texture coordinate. This parameter is optional. If you do not specify this parameter, the short video SDK performs the following implementation by default: `[ { "name": "position", "type": "POSITION" }, { "name": "inputTextureCoordinate", "type": "TEXTURECOORD" } ]`
fragment	The fragment shader code. This parameter has the same function as the fragmentPath parameter. You can declare either one of the two parameters.
fragmentPath	The path to the file in which the fragment shader code resides. This parameter has the same function as the fragment parameter. You can declare either one of the two parameters.
textures	The list of textures. This parameter specifies the attributes of `sampler2D` textures in the fragment shader. The attributes include: name: the name of the texture. srcType: the type of the texture data source, which represents where the texture data to be drawn is obtained from. Valid values: IMAGE: The texture comes from an image. In this case, you must declare the `path` attribute and put the image file in the resource package. INPUT_NODE: The texture comes from the image data passed in from the input node. In this case, you must declare the nodeId attribute. CUSTOM_NODE: The texture comes from the image data processed by the custom node. In this case, you must declare the nodeId attribute. nodeId: the ID of the node. This attribute is required if the value of the srcType attribute is CUSTOM_NODE or INPUT_NODE. If the value of the srcType attribute is INPUT_NODE, valid values of nodeId include 0 and 1. In a filter scenario, set the value to 0. In a transition scenario, set the nodeId attribute to 0 for the video stream before the transition and to 1 for the video stream after the transition. path: the path to the resource file. This attribute is required if the value of the srcType attribute is IMAGE.
params	The list of uniform attributes. This parameter specifies the attributes of a `uniform` in the fragment shader. The attributes include: name: the name of the uniform. type: the type of the uniform. For more information about the supported types, see the Supported types of a uniform section of this topic. value: the value of the uniform. maxValue: the maximum value of the uniform. minValue: the minimum value of the uniform.

Supported types of a uniform

Type	Example value
INT	[1]
FLOAT	[-2.0]
VEC2I	[3, 2]
VEC3I	[1, 2, 3]
VEC4I	[4, 3, 2, 1]
VEC2F	[-1.0, 1.0]
VEC3F	[0.5, 0.5, 0.5]
VEC4F	[1.0, -1.0, 1.0, -1.0]
MAT3F	[1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
MAT4F	[1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0]

The following code shows how the configuration file of the Translate transition is configured:

{
 "name": "Translate",
 "module": "ALIVC_GECF",
 "version": 1,
 "type": 2,
 "nodeTree": [
 {
 "nodeId": 0,
 "name": "Translate",
 "vertex": "/** ...... */",
 "attributes": [
 {
 "name": "a_position",
 "type": "POSITION"
 },
 {
 "name": "a_texcoord",
 "type": "TEXTURECOORD"
 }
 ],
 "fragment": "/** ...... */",
 "textures": [
 {
 "name": "RACE_Tex0",
 "srcType": "INPUT_NODE",
 "nodeId": 0
 },
 {
 "name": "RACE_Tex1",
 "srcType": "INPUT_NODE",
 "nodeId": 1
 }
 ],
 "params": [
 {
 "name": "direction",
 "type": "INT",
 "value": [
 0
 ],
 "maxValue": [
 3
 ],
 "minValue": [
 0
 ]
 }
 ]
 }
 ]
}

Built-in variables

The short video SDK provides the following built-in variables that you can declare in the shader:

uniform highp float BUILTIN_PROGRESS; // The transition progress. Valid values: 0 to 1.
uniform highp float BUILTIN_WIDTH; // The image width.
uniform highp float BUILTIN_HEIGHT; // The image height.

Use an effect by using the short video SDK

To use an effect, you need to create an effect object, apply the effect, and update the effect parameters based on your requirements.

In the initialized effect object, you can obtain the AliyunEffectConfig object that describes the effect configurations. The internal structure of the AliyunEffectConfig object corresponds to the structure of the effect configuration file. If a custom effect configuration file contains the params field, you can obtain the AliyunParam object in the code in the following path: AliyunEffectConfig -> nodeTree -> params. The value field in the AliyunParam object indicates the value of the current parameter. You can perform one of the following steps to update the effect parameters:

Update the parameters by calling the update method provided by the AliyunValue object.
Update the parameters by calling the effect update method. For more information about how to update effect parameters, see steps c in the procedures below.

Procedure for Android

Filters
1. Call the EffectFilter(String path) method to create a filter object. The path parameter specifies the path of the filter folder.
2. Call the int addAnimationFilter(EffectFilter filter); method to apply the filter.
3. Call the int updateAnimationFilter(EffectFilter filter); method to update the filter parameters.
Transitions
1. Call the TransitionBase(String path) method to create a transition object. The path parameter specifies the path of the transition folder.
2. Call the int setTransition(int index, TransitionBase transition); method to apply the transition.
3. Call the int updateTransition(int clipIndex, TransitionBase transitionBase); method to update the transition parameters.

Procedure for iOS

Filters
1. Call the - (instancetype)initWithFile:(NSString *)path;method of the AliyunEffectFilter class to create a filter object. The path parameter specifies the path of the filter folder.
2. Call the - (int)applyAnimationFilter:(AliyunEffectFilter *)filter; method to apply the filter.
3. Call the - (int)updateAnimationFilter:(AliyunEffectFilter *)filter; method to update the filter parameters.
Transitions
1. Call the -(instancetype)initWithPath:(NSString *)path; method of the AliyunTransitionEffect class to create a transition object. The path parameter specifies the path of the transition folder.
2. Call the - (int)applyTransition:(AliyunTransitionEffect *)transition atIndex:(int)clipIdx; method to apply the transition.
3. Call the -(int)updateTransition:(AliyunTransitionEffect *)transition atIndex:(int)clipIdx; method to update the transition parameters.