如何使用Elasticsearch SDK和Elasticsearch相容介面分析Log Service中的資料 - Simple Log Service

本文介紹如何使用Elasticsearch SDK和Elasticsearch相容介面分析Log Service中的資料。

重要

本文檔為阿里雲原創文檔，智慧財產權歸阿里雲所有，由於本文檔旨在介紹阿里雲與第三方產品互動的服務能力，因此可能會提及第三方公司或產品等名稱。

前提條件

已建立Project、標準型Logstore並完成日誌採集。具體操作，請參見建立專案Project、建立Logstore和資料擷取概述。
查詢日誌前，已配置索引。具體操作，請參見建立索引。
已建立RAM使用者的AccessKey，並且為RAM使用者授予Logstore的查詢許可權。具體操作，請參見RAM授權。

注意事項

僅支援Elasticsearch 7.x版本的SDK訪問Elasticsearch相容介面。
在查詢時不指定@timestamp，則預設查詢24小時內的資料。

參數說明

參數名稱	說明
`hosts`	資料訪問地址。格式為`https://${Project名稱}.${Project訪問網域名稱}/es/`。更多資訊，請參見服務入口。重要只支援使用HTTPS協議。
`Username`	輸入`Username`和`Password`，建議使用RAM使用者的AccessKey，該RAM使用者需具備Logstore的資料查詢許可權。您可以通過許可權助手，配置許可權。具體操作，請參見配置許可權助手。擷取AccessKey的方法，請參見存取金鑰。
`Password`
`Index`	`${Project名稱}.${Logstore名稱}`

樣本

下面以 project 為etl-dev、logstore為accesslog、slsEndpoint為cn-huhehaote.log.aliyuncs.com 為例，示範如何使用Elasticsearch SDK和Elasticsearch相容介面分析Log Service中的資料。

cURL訪問樣本

curl -u ${ALIYUN_ACCESS_KEY_ID}:${ALIYUN_ACCESS_KEY_SECRET} "https://etl-dev.cn-huhehaote.log.aliyuncs.com/es/etl-dev.accesslog/_search?q=status:200"

Python SDK訪問樣本

安裝依賴。
```
pip install elasticsearch==7.10.0
```

樣本。

#!/bin/env python3
import os
import json
import time
from elasticsearch import Elasticsearch, helpers

slsProject = "etl-dev"
slsEndpoint = "cn-huhehaote.log.aliyuncs.com"
slsLogstore = "accesslog"

esHost = "https://%s.%s/es/" % (slsProject, slsEndpoint)
esIndex = "%s.%s" % (slsProject, slsLogstore)

# 從環境變數中擷取到ak資訊
accessKeyId = os.environ['ALIYUN_ACCESS_KEY_ID']
accessKeySecret = os.environ['ALIYUN_ACCESS_KEY_SECRET']

esClient = Elasticsearch(hosts=esHost,
                http_auth=(accessKeyId, accessKeySecret),
                   verify_certs=True, timeout=300)

endTime = int(time.time()*1000)
startTime = endTime - 3600*1000

r = esClient.search(
    index=esIndex,
    body=   {
        "query": {
            "bool": {
                "filter": [
                    {
                        "range": {
                            "@timestamp": {
                                "gte": startTime,
                                "lte": endTime,
                                "format": "epoch_millis"
                            }
                        }
                    }
                ]
            }
        }
     }
)

print(json.dumps(r, indent=4))

Elasticsearch-DSL方式訪問樣本

Elasticsearch-DSL（Elasticsearch Domain Specific language）是Elasticsearch搜尋資料的文法，簡稱DSL。為了避免手動組裝Elasticsearch-DSL,可以用如下方式訪問。

安裝依賴。
```
pip install elasticsearch-dsl==7.4.1
```

elasticsearch-dsl方式訪問樣本。

#!/bin/env python3
import os
import json
import time
from elasticsearch import Elasticsearch, helpers
from elasticsearch_dsl import Search, Q

slsProject = "etl-dev"
slsEndpoint = "cn-huhehaote.log.aliyuncs.com"
slsLogstore = "accesslog"

esHost = "https://%s.%s/es/" % (slsProject, slsEndpoint)
esIndex = "%s.%s" % (slsProject, slsLogstore)

# 從環境變數中擷取到ak資訊
accessKeyId = os.environ['ALIYUN_ACCESS_KEY_ID']
accessKeySecret = os.environ['ALIYUN_ACCESS_KEY_SECRET']

esClient = Elasticsearch(hosts=esHost,
                http_auth=(accessKeyId, accessKeySecret),
                   verify_certs=True, timeout=300)

endTime = int(time.time()*1000)
startTime = endTime - 3600*1000

s = Search(using=esClient, index=esIndex) \
        .filter(Q("range", **{"@timestamp": {"gte": startTime, "lt": endTime, "format": "epoch_millis"}}))  \
        .query("match", request_method="GET") \

response = s.execute()

for hit in response:
    # request_method, host, client_ip 是sls日誌中的欄位
    print(hit.request_method, hit.host, hit.client_ip)

Golang SDK訪問樣本

package main

import (
	"context"
	"fmt"
	"os"
	"time"

	"github.com/olivere/elastic/v7"
)

func main() {
	// 下面是一個es sdk訪問sls es 相容介面的範例
	slsProject := "etl-dev"
	slsLogstore := "accesslog"
	slsEndpoint := "cn-huhehaote.log.aliyuncs.com"

	accessKeyID := os.Getenv("ALIYUN_ACCESS_KEY_ID")
	accessKeySecret := os.Getenv("ALIYUN_ACCESS_KEY_SECRET")
	esHost := fmt.Sprintf("https://%s.%s:443/es", slsProject, slsEndpoint)
	esIndex := fmt.Sprintf("%s.%s", slsProject, slsLogstore)

	esClient, err := elastic.NewClient(
		elastic.SetURL(esHost),
		elastic.SetSniff(false),
		elastic.SetBasicAuth(accessKeyID, accessKeySecret), // 設定基本認證的使用者名稱和密碼
		elastic.SetHealthcheck(false),                      // 關閉健全狀態檢查
	)
	if err != nil {
		panic(err)
	}

	termQuery := elastic.NewTermQuery("request_method", "GET")
	endTime := time.Now().Unix()
	startTime := endTime - 3600
	timeRangeQuery := elastic.NewRangeQuery("@timestamp").Gte(startTime).Lte(endTime)

	boolQuery := elastic.NewBoolQuery()
	boolQuery = boolQuery.Must(timeRangeQuery, termQuery)

	searchResult, err := esClient.Search().
		Index(esIndex).
		Query(boolQuery).
		From(0).Size(10).
		Pretty(true).
		Do(context.Background())
	if err != nil {
		panic(err)
	}

	// 輸出結果
	for _, hit := range searchResult.Hits.Hits {
		fmt.Println(string(hit.Source))
	}
}

Java SDK訪問樣本

pom.xml引入依賴。

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>estest</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-high-level-client</artifactId>
            <version>7.10.1</version>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>elasticsearch-rest-client</artifactId>
            <version>7.10.1</version>
        </dependency>
    </dependencies>
</project>

樣本。

package org.example;

import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestClientBuilder;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.MatchQueryBuilder;
import org.elasticsearch.index.query.RangeQueryBuilder;
import org.elasticsearch.search.builder.SearchSourceBuilder;

import java.io.IOException;

public class Main {
    public static void main(String[] args) throws IOException {

        String slsProject = "etl-dev";
        String slsLogstore = "accesslog";
        String slsEndpoint = "cn-huhehaote.log.aliyuncs.com";

        String schema = "https";
        String esHost = slsProject + "." +  slsEndpoint; // ${project}.${endpoint}
        int port = 443;
        String esIndex = slsProject + "." + slsLogstore; // ${project}.${logstore}
        String esPrefix = "/es/";
        String accessKeyId = System.getenv("ALIYUN_ACCESS_KEY_ID");
        String accessKeySecret = System.getenv("ALIYUN_ACCESS_KEY_SECRET");

        final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
        credentialsProvider.setCredentials(AuthScope.ANY,
                new UsernamePasswordCredentials(accessKeyId, accessKeySecret));

        RestClientBuilder builder = RestClient.builder(new HttpHost(esHost, port, schema)).setHttpClientConfigCallback(
                    httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider));

        // Set /es/ prefix
        builder.setPathPrefix(esPrefix);
        RestHighLevelClient client = new RestHighLevelClient(builder);

        // Query
        BoolQueryBuilder boolExpr= new BoolQueryBuilder();

        long endTime = System.currentTimeMillis();
        long startTime = endTime - 3600 * 1000;
        boolExpr.filter().add(new MatchQueryBuilder("request_method", "GET"));
        boolExpr.filter().add(new RangeQueryBuilder("@timestamp").gte(startTime).lte(endTime).format("epoch_millis"));

        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();

        searchSourceBuilder.query(boolExpr);
        SearchRequest searchRequest = new SearchRequest(esIndex);
        searchRequest.source(searchSourceBuilder);
        SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);
        System.out.println(searchResponse.toString());

        client.close();
    }
}

PHP SDK訪問樣本

使用composer安裝PHP外掛程式。

composer require elasticsearch/elasticsearch

樣本。

<?php

require 'vendor/autoload.php';

use Elasticsearch\ClientBuilder;

$slsProject = 'etl-dev';
$slsLogstore = 'accesslog';
$slsEndpoint = 'cn-huhehaote.log.aliyuncs.com';

$esHost = $slsProject . '.' . $slsEndpoint;
$esIndex = $slsProject . '.' . $slsLogstore;

$accessKeyId = getenv('ALIYUN_ACCESS_KEY_ID');
$accessKeySecret = getenv('ALIYUN_ACCESS_KEY_SECRET');

$hosts = [
    [
        'host' => $esHost,
        'port' => '443',
        'scheme' => 'https',
        'path' => '/es',
        'user' => $accessKeyId,
        'pass' => $accessKeySecret,
    ]
];

$client = ClientBuilder::create()
    ->setHosts($hosts)
    ->build();

$endTime = round(microtime(true) * 1000); // 毫秒
$startTime = $endTime - (3600 * 1000);


$params = [
    'index' => $esIndex,
    'body'  => [
        'query' => [
            'bool' => [
                'must' => [
                    'match' => [
                        'request_method' => 'GET'
                    ]
                ],
                'filter' => [
                    'range' => [
                        '@timestamp' => [
                            'gte' => $startTime,
                            'lte' => $endTime
                        ]
                    ]
                ]
            ]
        ]
    ]
];

$response = $client->search($params);

print_r($response);