Associate a PostgreSQL data source
Create an external store to associate SLS with a PostgreSQL database.
Prerequisites
-
Data is collected in SLS. Data collection.
-
You have installed the SLS CLI and configured the service endpoint and AccessKey pair.
-
Data is stored in an Alibaba Cloud RDS for PostgreSQL, ADB for PostgreSQL, or Hologres instance.
-
The PostgreSQL database is in a VPC. The RDS, ADB, or Hologres instance is in the same region as your SLS project.
ImportantInternet endpoints are not supported for PostgreSQL connections.
Background information
SLS external stores let you associate SLS with an RDS for PostgreSQL, ADB for PostgreSQL, or Hologres instance. You can write query and analysis results to a PostgreSQL database for further processing.
Procedure
1. Configure a whitelist
RDS for PostgreSQL database
Add the following CIDR blocks to the whitelist: 100.104.0.0/16, 11.194.0.0/16, and 11.201.0.0/16. Set a whitelist.
Alibaba Cloud Hologres
Add the following CIDR blocks to the whitelist: 100.104.0.0/16, 11.194.0.0/16, and 11.201.0.0/16. IP whitelist.
ADB for PostgreSQL database
Add the following CIDR blocks to the whitelist: 100.104.0.0/16, 11.194.0.0/16, and 11.201.0.0/16. Configure a whitelist.
2. Create an ExternalStore
-
Log on to the CLI server. Run
touchto create /home/shell/config.json. Add the following content to config.json and replace theregion,vpc-id,host,port,username,password,db, andtablevalues with your own.Parameter description
Example
{ "externalStoreName":"sls_join_meta_store", "storeType":"postgresql", "parameter":{ "region":"cn-qingdao", "vpc-id":"vpc-m5eq4irc1pucp*******", "host":"localhost", "port":"3306", "username":"user", "password":"****", "db":"scmc", "table":"join_meta" } }externalStoreNameThe ExternalStore name. Must be lowercase.
storeTypeThe data source type. Set to postgresql.
regionThe region, such as cn-wulanchabu or cn-hangzhou.
ImportantThe RDS instance, Hologres instance, or ADB instance must be in the same region as the SLS project.
vpc-idThe VPC ID of the database. Varies by database type:
-
RDS for PostgreSQL: set vpc-id to the VPC ID of the instance.
-
ADB for PostgreSQL: set vpc-id to the VPC ID of the instance.
-
Hologres: set vpc-id to the VPC ID of the instance.
hostThe internal endpoint of the database.
portThe database port.
usernameThe database username.
passwordThe database password.
dbThe name of the database.
tableThe database table name. Supported formats:
-
table_name, for example, test.
-
schema_name.table_name, for example, public.test.
-
-
Run the following command to create an ExternalStore. Replace project_name with your SLS project name. This example uses
log-rds-demo.aliyunlog log create_external_store --project_name="log-rds-demo" --config="file:///home/shell/config.json" -
If the command succeeds, no response is returned. Run
aliyunlog log get_external_store --project_name="log-rds-demo" --store_name="sls_join_meta_store" --format-output=jsonto verify. Expected output:{ "externalStoreName": "sls_join_meta_store", "parameter": { "db": "postgres", "host": "rm-bp1******rm76.pg.rds.aliyuncs.com", "port": "5432", "region": "cn-wulanchabu", "table": "test", "timezone": "", "username": "user", "vpc-id": "vpc-m5eq4irc1pucp*******" }, "storeType": "postgresql" }
3. Query the PostgreSQL data
In the SLS console, open the log-rds-demo project, select any Logstore, and run * | select * from sls_join_meta_store to query the PostgreSQL data source.
The query returns two rows from the dimension table: ID 123 (name: Bob) and ID 124 (name: Lucy).
Related operations
-
Update the PostgreSQL external store.
aliyunlog log update_external_store --project_name="log-rds-demo" --config="file:///home/shell/config.json" -
Delete the PostgreSQL external store.
aliyunlog log delete_external_store --project_name="log-rds-demo" --store_name=sls_join_meta_store