Run Dedicated SQL queries on log data by using the Simple Log Service SDK for C++. Dedicated SQL supports processing hundreds of billions of rows of data.
Prerequisites
-
A RAM user is created and granted the required permissions. For more information, see Create a RAM user and grant permissions.
-
The
ALIBABA_CLOUD_ACCESS_KEY_IDandALIBABA_CLOUD_ACCESS_KEY_SECRETenvironment variables are configured. For more information, see Configure environment variables on Linux, macOS, and Windows.ImportantThe AccessKey pair of an Alibaba Cloud account has access to all API operations. Use the AccessKey pair of a RAM user for API calls or routine O&M.
Do not hard-code your AccessKey ID and AccessKey Secret in your project code. Leaked AccessKey pairs compromise the security of all resources in your account.
-
The latest version of Simple Log Service SDK for C++ is installed. For more information, see Alibaba Cloud Simple Log Service SDK for C++.
Background
Dedicated SQL enhances SQL analysis capabilities and supports processing hundreds of billions of rows of data. For more information, see Enable Dedicated SQL.
Simple Log Service provides the ExecuteLogStoreSql and ExecuteProjectSql operations for Dedicated SQL.
-
ExecuteLogStoreSql: Runs a Dedicated SQL query in a specified Logstore. This operation supports query and analysis syntax compatible with standard SQL-92. The format is
search statement|analytic statement, where the analytic statement follows standard SQL-92 syntax. -
ExecuteProjectSql: Runs a Dedicated SQL query in a specified project. This operation supports SQL-92 syntax. Specify a filter condition and a time range in the WHERE clause of the SQL statement.
To filter data before analysis, use the search statement|analytic statement syntax with the ExecuteLogStoreSql operation for better performance.
Sample code
The following code uses the Dedicated SQL feature. For the full SDK, see Alibaba Cloud Simple Log Service SDK for C++.
#include <cstdlib>
// Simple Log Service endpoint. Replace with your actual endpoint.
std::string endpoint = "cn-hangzhou.log.aliyuncs.com";
// Retrieve the AccessKey ID and AccessKey Secret from environment variables.
string accessId = (string)getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
string accessKey = (string)getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
// Project name.
std::string project = "your_project_name";
// Logstore name.
std::string logStore = "your_logstore";
// Create a Simple Log Service client.
LOGClient client(endpoint, accessId, accessKey);
// Run a SQL query in the specified Logstore.
try
{
std::string sql = "* | select count(1)";
int from = time(NULL) - 600;
int to = from + 600;
LogStoreSqlResponse logsResponse = client.ExecuteLogStoreSql(project, logStore,
1627268185,1627269085,"* | SELECT count(*)",true);
// Print the query result statistics.
std::cout << "Returned sql result:" << std::endl
<< "count:" << logsResponse.result.logline << std::endl // Number of rows in the result.
<< "processed rows:" << logsResponse.processedRows << std::endl // Number of log rows processed.
<< "elapsed milli:" << logsResponse.elapsedMilli << std::endl // SQL query execution duration.
<< "cpu sec:" << logsResponse.cpuSec << std::endl // CPU time (seconds) for Dedicated SQL. Billing is based on this value. For more information, see billable items.
<< "cpu core:" << logsResponse.cpuCore << std::endl; // Number of CPU cores used for the Dedicated SQL query.
for (std::vector<LogItem>::const_iterator itr = logsResponse.result.logdatas.begin();
itr != logsResponse.result.logdatas.end(); ++itr)
{
const LogItem &item = *itr;
for (std::vector<std::pair<std::string, std::string>>::const_iterator itr_data = item.data.begin();
itr_data != item.data.end(); ++itr_data)
{
std::cout << itr_data->first << ":" << itr_data->second;
}
}
}
catch (LOGException &e)
{
std::cout << "error code :" << e.GetErrorCode() << std::endl;
std::cout << "error message :" << e.GetMessage() << std::endl;
throw e;
}
// Run a SQL query in the specified project.
try
{
int now = time(NULL);
std::string sql = "select count(1) as cnt from xxx where __time__ > " + to_string(now);
ProjectSqlResponse logsResponse = client.ExecuteProjectSql(project,"select avg(latency),max(latency) ,count(1) as c from sample-logstore where status>200 and __time__>=1500975424 and __time__ < 1501035044 GROUP BY method ORDER BY c",true);
// Print the query result statistics.
std::cout << "Returned sql result:" << std::endl
<< "count:" << logsResponse.result.logline << std::endl // Number of rows in the result.
<< "processed rows:" << logsResponse.processedRows << std::endl // Number of log rows processed.
<< "elapsed milli:" << logsResponse.elapsedMilli << std::endl // SQL query execution duration.
<< "cpu sec:" << logsResponse.cpuSec << std::endl // CPU time (seconds) for Dedicated SQL. Billing is based on this value. For more information, see billable items.
<< "cpu core:" << logsResponse.cpuCore << std::endl; // Number of CPU cores used for the Dedicated SQL query.
for (std::vector<LogItem>::const_iterator itr = logsResponse.result.logdatas.begin(); itr != logsResponse.result.logdatas.end(); ++itr)
{
const LogItem &item = *itr;
for (std::vector<std::pair<std::string, std::string>>::const_iterator itr_data = item.data.begin();
itr_data != item.data.end(); ++itr_data)
{
std::cout << itr_data->first << ":" << itr_data->second;
}
}
}
catch (LOGException &e)
{
std::cout << "error code :" << e.GetErrorCode() << std::endl;
std::cout << "error message :" << e.GetMessage() << std::endl;
throw e;
}
-
ExecuteLogStoreSql
Call the ExecuteLogStoreSql operation in the following format:
LogStoreSqlResponse logsResponse = client.ExecuteLogStoreSql(project, logStore, from, to, query, powerSql). The following table describes the parameters.Parameter
Type
Required
Example
Description
project
String
Yes
N/A
The name of the project.
Already defined when creating the client. No additional configuration is required.
logStore
String
Yes
N/A
The name of the Logstore.
Already defined when creating the client. No additional configuration is required.
from
Long
Yes
1627268185
The start of the query time range. The value is a UNIX timestamp in seconds.
to
Long
Yes
1627269085
The end of the query time range. The value is a UNIX timestamp in seconds.
query
String
Yes
"* | SELECT count(*)"The query and analysis statement for Simple Log Service, in the format
search statement|analytic statement. For more information, see Basic syntax.By default, Simple Log Service returns 100 rows of data. To return more rows, add a LIMIT clause. For more information, see LIMIT clause.
powerSql
Boolean
No
true
Specifies whether to use Dedicated SQL. For more information, see Enable Dedicated SQL. Valid values:
-
true: uses Dedicated SQL.
-
false (default): uses Standard SQL.
-
-
ExecuteProjectSql
Call the ExecuteProjectSql operation in the following format:
ProjectSqlResponse logsResponse = client.ExecuteProjectSql(project, query, powerSql). The following table describes the parameters.Parameter
Type
Required
Example
Description
project
String
Yes
N/A
The name of the project.
Already defined when creating the client. No additional configuration is required.
query
String
Yes
"select avg(latency),max(latency) ,count(1) as c from sample-logstore where status>200 and __time__>=1500975424 and __time__ < 1501035044 GROUP BY method ORDER BY c"A standard SQL-92 statement. Specify a filter condition and a time range in the WHERE clause.
By default, Simple Log Service returns 100 rows of data. To return more rows, add a LIMIT clause. For more information, see LIMIT clause.
powerSql
Boolean
No
true
Specifies whether to use Dedicated SQL. For more information, see Enable Dedicated SQL. Valid values:
-
true: uses Dedicated SQL.
-
false (default): uses Standard SQL.
-