Each time a pipeline task runs, a new instance is scheduled. You must pull the code for build and build the required dependencies. If a long period of time is required to pull dependencies, the pipeline task execution may be slow or time out. The Serverless-cd community provides a cache plug-in that is based on Object Storage Service (OSS) to resolve issues such as slow pulling of dependencies and build timeouts. This topic describes how to use the OSS cache plug-in to accelerate dependency building.
Method for using the cache plug-in
The Serverless-cd community provides a cache plug-in, which is optimized based on the features of Serverless Application Center. You need to only specify the OSS bucket used for caching when you use the cache plug-in. The cache plug-in retrieves, reads, and updates the cached data in the specified or default OSS bucket based on specified keys.
If a cache key is matched after you execute the plug-in, the file that corresponds to the cache key is downloaded to a specified on-premises directory.
After you run the engine, the files in the on-premises directory are uploaded to the OSS bucket.
Define the OSS cache in the execution context
You can define OSS cache configurations in the execution context in a pipeline, a pipeline template, or a task template. Sample code:
---
# Submit the pipeline for execution.
kind: Pipeline
name: "p-<% .git.shortCommitId %>-<% .currentTimestampMs %>"
# The description.
description: cached pipeline
spec:
context:
data:
# The cache configuration of Serverless Application Center. If you leave this section empty, the engine is automatically initialized in the region where the task node is scheduled.
# The default bucket name is serverless-cd-cache-${region-id}-<% .accountId %>.
cacheConfig:
# Specify to use OSS for caching. If you leave this section empty, the platform automatically performs the initialization.
# We recommend that you use the platform to automatically perform the initialization, because the worker nodes may be scheduled to a region that is different from the region in which the specified OSS bucket resides.
# For code repositories on GitHub, the worker nodes and corresponding OSS buckets reside in the Singapore region (ap-southeast-1).
# For code repositories on GitLab, Codeup, and Gitee, the worker nodes and the corresponding OSS buckets reside in the China (Hangzhou) region (cn-hangzhou).
oss:
# The name of the bucket that is used for caching. If the bucket does not exist, the engine does not automatically perform the initialization.
bucketName: serverless-cd-cache-ap-southeast-1-<% .accountId %>
# The region where the bucket resides. We recommend that you do not pull data from a cache that reside in another region.
regionId: ap-southeast-1
templateName: mytemplate-<% .git.branch %>
---Use the cache plug-in in steps
You can use the cache plug-in by executing context steps in a pipeline template or a task template. You do not need to specify credentials. The plug-in automatically uses the role permissions that are you grant for Serverless Application Center. Sample code:
---
kind: PipelineTemplate
name: mytemplate-<% .git.branch %>
description: cached pipelinetemplate
spec:
context:
data:
envName: test
deployFile: s.yaml
tasks:
# Perform a pre-build check.
- name: pre-check
context:
data:
displayName: "Pre-build check"
enable: true
steps:
- plugin: "@serverless-cd/checkout"
- plugin: "@serverless-cd/s-setup"
- run: s plan --local -o json
taskTemplate: serverless-runner-task
runAfters: []
- name: pre-check-approval
context:
data:
dingdingNotification:
enable: false
enable: true
taskTemplate: need-approval
runAfters:
- name: pre-check
# Build and deploy
- name: build-and-deploy
context:
data:
enable: true
steps:
- plugin: "@serverless-cd/checkout"
- plugin: "@serverless-cd/s-setup"
- plugin: "@serverless-cd/cache"
inputs:
# Define a unique identifier for the cache, which can be associated with related information, such as the application information or Git version information.
# You can also use "cache-${{hashFile('./package.json')}}".
# Calculate the unique identifier of the cache based on the hash value of the on-premises file.
# The path of the cached object in the bucket is cache-home/${key}.
# In this example, the cache path in the bucket is cache-home/cache-<% .appName %>.
key: "cache-<% .appName %>"
# Configure the cache path as required by the current build tool.
# For example, for a Node.js runtime, you can use the default value ~/.npm as the path of the pipeline application cache.
path: ~/.npm
# Run s-deploy.
- run: |
set -e
s deploy -t --use-local --assume-yes --skip-push
echo "Deploy by Serverless Devs success."
taskTemplate: serverless-runner-task
runAfters:
- name: pre-check-approval
---Cache path
The actual cache path varies based on the build tool. The following table describes the cache paths for common build tools. You can also use the documentation of a build tool to obtain the cache path.
Tool | Cache path |
Maven | ~/.m2 |
Gradle | ~/.gradle |
NPM | ~/.npm |
Yarn | ~/.yarn |
gomod | $GOPATH/pkg/mod |
Other caches (such as pip3) | ~/.cache |
On-premises cache
You can use ossutils to make an on-premises cache if the performance of the service that is used to build dependencies is poor or dependencies cannot be pulled from the cloud due to security reasons. In this case, you must cache data by using the on-premises development device and upload the cache to OSS. The uploaded cache can be used in pipelines.
You can also log on to the OSS console and upload the on-premises cache, such as a ~/.npm, to an OSS bucket.
Sample code:
# Use ossutils to create a cache for cloud-based build. Copy the on-premises npm cache to an OSS bucket.
# The automatically created bucket, whose URL is serverless-cd-${region-id}-cache-${uid}.oss-${region-id}.aliyuncs.com.
# For code repositories on GitHub, the worker nodes and corresponding OSS buckets must be deployed in the Singapore region (ap-southeast-1).
# For code repositories on GitLab, Codeup, and Gitee, worker nodes and corresponding OSS buckets must be deployed in the China (Hangzhou) region (cn-hangzhou).
./ossutil64 cp -r ~/.npm oss://${bucket_url}/cache-home/${cache_key}