View here to log in or access your console

OK

Log Service

An all-in-one service for log-type data

Get it free

Overview

Honed originally by the Big Data demands of Alibaba Group, Log Service (or "Log" for short) is an all-in-one service for log-type data. It helps increase Operations & Management and operational efficiency, as well as build the processing capability to deal with massive logs.

As a Pay-As-You-Go service, Log Service does not require any upfront investment and saves you the hassle of owning and scaling your own platform. Best of all the service is maintenance free.


Benefits

Convenience

  • Comprehensive APIs/full-function console, that set up access in only 5 minutes

  • Rich upstream and downstream resources, enabling seamless interconnection with cloud products and common open source software

Reliability

  • Elastic to meet diversified traffic needs ranging from 1MB to 100TB per day

  • 99.9% availability, and automatically resumable data transfer on the client side

  • Complete security mechanism including RAM, HTTPS, encryption and so on

Cost

  • Pay-As-You-Go billing, costing 60% less than self-built solutions

  • Upgrading and resizing to free up users from O&M, deployment, and development, without any impact on the business

  • Client performance is 10 times higher than similar software, while only consuming 10% of the resources


Product Details

Alibaba Cloud Log Service offers a fully managed service to complete data collection, consumption, shipping, query, and analysis without development. Typical Log Service application scenarios include: data collection, real-time computing, data warehousing and offline analysis, product operation and analysis, and O&M and management.

In regards to security, Log Service provides multiple level security with authentication mechanisms to secure against unauthorized access.

The service is able to achieve complex capabilities via seamless integration with other Alibaba Cloud services such as Object Storage Service (OSS), and E-MapReduce Service. The service can integrate with third party software vendors using RESTful APIs.


Features

Real-time log collection and consumption

LogHub - real-time collection: Over 30 methods to collect logs from servers, SDKs, mobile terminals, network protocols, and embedded devices

  • Logtail: Stable, reliable, secure, and available for all platforms (Linux, Windows, and Docker), keeping performance high and resource utilization low

  • API/SDK: Flexible, convenient, scalable, and available in 10+ languages and mobile clients

  • Cloud product logging (reference): ECS, Container Service, MNS, CDN, and other services. One key implementation, convenient and efficient

  • Others (reference): syslog, Unity3D, LogStash, Log4j, and Nginx

LogHub - real-time consumption: Support for stream computing, collaborative consumption library, and multiple-language

  • Comprehensive functions: Compatible with 100% of Kafka functions while offering ordering, auto scaling, time-frame-based seek, and other functions

  • Stable and reliable: Logs consumed upon writing; 99.9% availability or higher; multiple data copies; auto scaling within seconds; low cost. For details, refer to Use LogHub to replace Kafka

  • Easy to use: Support forSpark Streaming, Storm, Consumer Library (an automatic load balancing programming mode), SDK subscriptions, and more

Data shipping

LogShipper - stable and reliable log shipping

  • OSS: Ship logs to OSS for analysis using E-MapReduce

  • MaxCompute: Ship logs to MaxCompute for analysis.

  • TableStore: Ship logs to TableStore.

Index and query

LogSearch - real-time data indexing and querying

  • Large scale: PB-level real-time indexing (data entries can be queried within 1 second upon writing); query over one billion log entries per second

  • Low cost: RMB 0.5/GB indexing cost; massive storage at a low cost; support for lifecycle configuration

  • Flexible queries: Support for keyword, fuzzy, cross-topic query, and context queries


Pricing

Log Service charges LogHub and LogSearch function, LogShipper is free.

LogHub

LogHub fees are based on resources you use. fees are billed on an incremental basis. After your amount exceeds a certain level, you will be billed based on the prices for the next increment. As you use more storage space, your cost per-unit will decrease.

Billing items < 100GB (including) > 100GB > 10TB > 1PB
Throughput ($/GB) 0.05 0.045 0.04 0.035
Storage ($/GB*Day) 0.003125 0.003 0.002875 0.00275

LogSearch

LogSearch is an addtional feature, you can choose to enable for log query and analysis. The fees are based on the index size, and it's a one-time charge when receiving data.

Billing items < 100GB (including) > 100GB > 10TB > 1PB
Index ($/GB) 0.1 0.0875 0.075 0.0625

Additonal

To ensure that the resources are properly used, there is an additional charge for API requests, share size, and data trasfer to internet (normally very cheap):

Billing items Price
API Requests ($/million calls) 0.03
Shard Size ($/Day) 0.01
Data Transfer to Internet($/GB) 0.2

Scenarios

The following are a few common Log Service scenarios:

1) Big chart

Log Service provides a complete end-to-end solution, including log collection, offline data warehousing, interconnection with real-time computing and monitoring, and centralized log management and query.

Ecosystem: Support for both upstream and downstream resources

2) Data collection

Log Service offers over 30 data collection measures, supports various networks, environments, devices, and data from different sources, and enables centralized management of distributed logs.

Reference: Log processing example - solution for a take-out website.

3) Interconnection with real-time computing

LogHub ensures log generation and consumption in real-time. It can interconnect with various stream computing tools (Spark, Storm, StreamCompute, and ConsumerLib) and monitoring systems, and perform real-time analysis.

Recommended combination: Log Service + E-MapReduce + Apache Storm

4) Interconnection with the data warehouse

Log Service is seamlessly integrated with OSS, MaxCompute, and TableStore, based on which a warehouse with massive data can be built for ideal log analysis.

Recommended combination: Log Service + OSS + E-MapReduce + MaxCompute

5) Log management/query

The log collection and query function can centralize data from applications, systems, and machines. It facilitates rapid investigation of user, application, or system problems to improve operation efficiency.

Recommended combination: Log Service

6) Log auditing

The log auditing function collects and centralizes storage logs. Based on the Log Service, it monitors logs in real-time, queries recent hot data in real-time, and backs up cold data.

Recommended combination: Log Service + OSS + E-MapReduce + MaxCompute

Getting Started

Quickly create and better manage your Log Service projects with the following developer resources, including Alibaba Cloud CLI (Command Line Interface) and Log Service APIs.

Using Log Service through the Management Console

Alibaba Cloud Management Console provides a simple web-based user interface that allows you to better access and configure your Log Service resources.

Using this console, you can create and modify Log Service projects.

For a step-by-step overview on how to manage and operate your Log Service projects through the Management Console, read the Quick Start Guide.

Log Service API Reference

To learn more and to see the full list of available Log Service APIs click here.

Resources

The links below will take you to SDKs and important documentation. These links will help developers integrate RDS with their existing applications.

FAQs

1. What is Log Service?

Log Service is a platform service used to collect, store, and subscribe to logs. The service enables you to collect in real-time, centrally manage and consume various types of logs.

2. What are the components of the Log Service?

Log Service consists of a log collection agent, a server, and other systems. Currently, the log collection agent is Logtail compatible with Windows and Linux. The server is responsible for reading, writing, and configuring Log Service APIs. Other systems include MaxCompute, to which the Log Service synchronizes logs. Log Service includes the ability to synchronize logs to OSS and AMR for consumption.

3. How is a log defined in the Log Service?

A log contains three parts: time (mandatory), log content (which consists of key– value pairs), and metadata (data source, which can be defined by the IP addresses of logs).