Alibaba Cloud Prometheus Service provides the remote write feature. You can use this feature to store the monitoring data of Prometheus Service to remote databases. This topic describes how to create a Prometheus instance for Remote Write (Prometheus instance for Remote Write). The instance allows you to use the remote write feature to connect Alibaba Cloud Prometheus Service to self-managed Prometheus. The feature provides an efficient solution to store monitoring data.

(Optional) Step 1: Grant the permissions to call the GetPrometheusApiToken operation to a RAM user

If you want to use a self-managed Prometheus host to write data to Alibaba Cloud Prometheus Service, you must call the GetPrometheusApiToken operation. By default, you can use an Alibaba Cloud account to call the GetPrometheusApiToken operation. If you want to use a RAM user to call the GetPrometheusApiToken operation, you must use your Alibaba Cloud account to grant the required permissions to the RAM user.

  1. In the left-side navigation pane, choose Permissions > Policies.
  2. On the Policies page, click Create Policy.
  3. On the Create Policy page, click the JSON tab.
  4. Enter the following policy document and click Next: Edit Basic Information. 
    {
    "Version": "1",
    "Statement": [
        {
            "Action": [
                "arms:GetPrometheusApiToken"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow"
        }
    ]
}
    For more information about the syntax and structure of policies, see Policy structure and syntax.
  5. Specify the Name and Description fields.
  6. Check and optimize the document of the custom policy.
    • Basic optimization

      The system automatically optimizes the policy statement. The system performs the following operations during basic optimization:

      • Deletes unnecessary conditions.
      • Deletes unnecessary arrays.
    • Optional:Advanced optimization

      You can move the pointer over Optional advanced optimize and click Perform. The system performs the following operations during the advanced optimization:

      • Splits resources or conditions that are incompatible with actions.
      • Narrows down resources.
      • Deduplicates or merges policy statements.
  7. Click OK.
  8. In the left-side navigation pane, choose Identities > Users.
  9. On the Users page, find the RAM user to which you want to attach the custom policy, and click Add Permissions in the Actions column.
  10. In the Add Permissions panel, grant permissions to the RAM user.
    1. Select the authorization scope.
      • Alibaba Cloud Account: The authorization takes effect on the current Alibaba Cloud account.
      • Specific Resource Group: The authorization takes effect in a specific resource group.
        Note If you select Specific Resource Group for Authorized Scope, make sure that the required cloud service supports resource groups. For more information, see Services that work with Resource Group.
    2. Specify the principal.
      The principal is the RAM user to which you want to grant permissions. By default, the current RAM user is specified. You can also specify another RAM user.
    3. Select policies.
      In the Select Policy section of the Add Permissions panel, click Custom Policy. Then, search for the policy that you created in Step 5. Click the policy name to add the policy to the Selected section. Grant permissions to a RAM user
  11. Click OK.
  12. Click Complete.

Step 2: Create a RemoteWrite instance and obtain the read and write URLs

  1. Log on to the ARMS console.
  2. In the left-side navigation pane, choose Prometheus Service > Prometheus Instances to go to the Prometheus Service page.
  3. Click Create Prometheus Instance. On the page that appears, click Prometheus Instance for Remote Write.
  4. Specify a name for the Prometheus instance. Then, click Create.
    The created Prometheus instance is displayed on the Prometheus Service page and Prometheus for Remote Write is displayed in the Instance Type column.
    Note If the specified instance name already exists, an error message appears after you click Create. In this case, you must specify another name for the instance.
  5. Copy and save the URLs that are generated in the Remote Read Address and Remote Write Address sections.

Step 3: Configure Prometheus

  1. Install Prometheus. For more information, see Prometheus official documentation.
  2. Open the Prometheus.yaml configuration file and append the following content to the file. Replace the remote_write and remote_read URLs with the URLs that you obtained in Step 2: Create a RemoteWrite instance and obtain the read and write URLs. Then, save the file. 
    global:
  scrape_interval:     15s
  evaluation_interval: 15s
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
remote_write:
  - url: "http://ts-xxxxxxxxxxxx.hitsdb.rds.aliyuncs.com:3242/api/prom_write"
    basic_auth:   
      // The username and password must conform to the naming conventions of an AccessKey pair. The owner of the AccessKey pair must have the permissions to call the GetPrometheusApiToken operation. 
      username: access-key-id
      password: access-key-secret
remote_read:
  - url: "http://ts-xxxxxxxxxxxx.hitsdb.rds.aliyuncs.com:3242/api/prom_read"
    read_recent: true
    Note If you want to use a self-managed Prometheus host to write data to Alibaba Cloud Prometheus Service, you must call the GetPrometheusApiToken operation. By default, you can use an Alibaba Cloud account to call the GetPrometheusApiToken operation. If you want to use a RAM user to call the GetPrometheusApiToken operation, you must use your Alibaba Cloud account to grant the required permissions to the RAM user. For more information, see (Optional) Step 1: Grant the permissions to call the GetPrometheusApiToken operation to a RAM user.

Use OpenTelemetry to connect to remote storage

After you connect Alibaba Cloud Prometheus Service to OpenTelemetry, you can use the remote storage feature of Prometheus Service to store OpenTelemetry data.

  1. Configure OpenTelemetry instrumentation in your business code. 
    package stat

import (
    "context"
    "fmt"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/metric"
    "time"
)

var buyCounter metric.Int64Counter

func init() {
    fmt.Println(time.Now(), " - initMetrics start......")
    meter := otel.GetMeterProvider().Meter("github.com/liguozhong/prometheus-arms-aliyun-go-demo")
    buyCounter = metric.Must(meter).NewInt64Counter(
        "buy_total",
        metric.WithDescription("Measures  buy"),
    )
}

func DoBuy() (string, error) {
    buyCounter.Add(context.Background(), 1)
    return "buy success", nil
}
  2. Initialize OpenTelemetry Meter and connect to Prometheus. 
    package stat

import (
    "context"
    prometheusPushExporter "go.opentelemetry.io/contrib/exporters/metric/cortex"
    prometheusExporter "go.opentelemetry.io/otel/exporters/metric/prometheus"

    "errors"
    "fmt"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp"
    "go.opentelemetry.io/otel/label"
    "go.opentelemetry.io/otel/sdk/metric/controller/pull"
    "go.opentelemetry.io/otel/sdk/metric/controller/push"
    "go.opentelemetry.io/otel/sdk/metric/processor/basic"
    "go.opentelemetry.io/otel/sdk/metric/selector/simple"
    "go.opentelemetry.io/otel/sdk/resource"
    "net/http"
    "time"

    _ "net/http/pprof"
)

func InitMeter(app string, push bool) error {
    fmt.Println(time.Now(), " - initMeter start......")
    if push {
        fmt.Println(time.Now(), " - initMeter opentelemetry push......")
        remoteUrl := "http://region.arms.aliyuncs.com/prometheus/../../../../api/v3/write"
        ak := "ak"
        sk := "sk"
        return initPushMeter(app, remoteUrl, ak, sk)
    }
    fmt.Println(time.Now(), " - initMeter opentelemetry pull......")
    return initPullMeter(app)
}

func initPushMeter(regionId string, remoteWriteUrl string, ak string, sk string) error {
    fmt.Println(time.Now(), " - initPushMeter start......")
    var validatedStandardConfig = prometheusPushExporter.Config{
        Endpoint:      remoteWriteUrl,
        Name:          "AliyunConfig",
        RemoteTimeout: 30 * time.Second,
        PushInterval:  10 * time.Second,
        Quantiles:     []float64{0.5, 0.9, 0.95, 0.99},
        BasicAuth: map[string]string{
            "username": ak,
            "password": sk,
        },
    }
    if validatedStandardConfig.Endpoint == "" {
        return errors.New(" validatedStandardConfig.Endpoint==empty.regionId:" + regionId)
    }
    fmt.Println("Success: Created Config struct")
    r, err := resource.New(context.Background(),
        resource.WithAttributes(
            label.String("cluster", "test-otel"),
            label.String("app", "buy")))
    if err != nil {
        fmt.Println("resource Error:", err)
    }
    pusher, err := prometheusPushExporter.InstallNewPipeline(validatedStandardConfig,
        push.WithPeriod(30*time.Second), push.WithResource(r))
    if err != nil {
        fmt.Println("InstallNewPipeline Error:", err)
    }
    otel.SetMeterProvider(pusher.MeterProvider())
    return nil
}

func initPullMeter(app string) error {
    fmt.Println(time.Now(), " - initPullMeter start......")
    r, err := resource.New(context.Background(),
        resource.WithAttributes(
            label.String("cluster", "test-otel"),
            label.String("app", app)))
    if err != nil {
        fmt.Println("resource Error:", err)
    }
    exporter, err := prometheusExporter.NewExportPipeline(
        prometheusExporter.Config{
            DefaultHistogramBoundaries: []float64{-0.5, 1},
        },
        pull.WithCachePeriod(0),
        pull.WithResource(r),
    )
    if err != nil {
        return err
    }
    http.HandleFunc("/opentelemetry", exporter.ServeHTTP)
    otel.SetMeterProvider(exporter.MeterProvider())
    return nil
}

func initOtlpProvider(regionId string) (*push.Controller, error) {
    exporter, err := otlp.NewExporter(
        context.Background(),
        otlp.WithInsecure(),
        otlp.WithAddress(regionId+"-intranet.arms.aliyuncs.com:8000"),
    )
    if err != nil {
        return nil, err
    }

    pusher := push.New(
        basic.New(
            simple.NewWithExactDistribution(),
            exporter,
        ),
        exporter,
        push.WithPeriod(30*time.Second),
    )

    otel.SetMeterProvider(pusher.MeterProvider())
    pusher.Start()

    return pusher, err
}
  3. Initialize the business endpoint and start OpenTelemetry. 
    package pkg

import (
    "fmt"
    stat "github.com/liguozhong/prometheus-arms-aliyun-go-demo/pkg/opentelemetry"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "io"
    "net/http"
    "strconv"
)

type Server struct {
    port int
}

func NewServer(port int) *Server {
    return &Server{
        port: port,
    }
}

func (s *Server) Run() error {
    port := ":" + strconv.Itoa(s.port)
    path := "/metrics"
    service := "/buy"
    http.Handle(path, promhttp.Handler()) // Initialize an HTTP handler.
    http.HandleFunc(service, func(writer http.ResponseWriter, request *http.Request) {
        content, err := stat.DoBuy()
        if err != nil {
            io.WriteString(writer, err.Error())
            return
        }
        io.WriteString(writer, content)
    })
    stat.InitMeter("buy2", true)
    fmt.Println("http.url: http://localhost" + port + path)
    fmt.Println("service.url: http://localhost" + port + service)
    err := http.ListenAndServe(port, nil)
    if err != nil {
        return err
    }
    return nil
}
  4. View the dependencies of the Go module. 
    module github.com/liguozhong/prometheus-arms-aliyun-go-demo

go 1.12

require (
    github.com/go-kit/kit v0.9.0
    github.com/prometheus/client_golang v1.7.1
    go.opentelemetry.io/contrib/exporters/metric/cortex v0.15.0
    go.opentelemetry.io/otel v0.15.0
    go.opentelemetry.io/otel/exporters/metric/prometheus v0.15.0
    go.opentelemetry.io/otel/exporters/otlp v0.15.0
    go.opentelemetry.io/otel/sdk v0.15.0
    golang.org/x/text v0.3.3 // indirect
)
  5. View data on Grafana dashboards. dh

Stop remote storage of Prometheus monitoring data

If you want to stop remote storage of Prometheus Service monitoring data, you can perform the following steps to uninstall the Prometheus agent:

  1. Log on to the ARMS console.
  2. In the left-side navigation pane, choose Prometheus Service > Prometheus Instances to go to the Prometheus Service page.
  3. On the Prometheus Service page, find the Prometheus instance that you want to uninstall and click Uninstall in the Actions column. In the message that appears, click OK.
    After you uninstall a , it is no longer displayed on the Prometheus instance page.