How do I use ApsaraDB for MongoDB to store and analyze logs - ApsaraDB for MongoDB

Online services generate large volumes of operational and access logs. Storing them as plain text works for quick lookups, but fails when you need to filter, aggregate, and derive insights from high-volume data. This topic describes how to use ApsaraDB for MongoDB to store and analyze web server access logs. The same patterns apply to other log types.

Structure log data for MongoDB

How you structure log documents directly affects query performance and storage cost. This section walks through three approaches, from simplest to most efficient.

A typical web server access log entry looks like this:

127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "[http://www.example.com/start.html](http://www.example.com/start.html)" "Mozilla/4.08 [en] (Win98; I ;Nav)"

It captures the request source, username, resource URL, response code, OS, and browser.

Store the raw line

The simplest approach stores each log entry as a single string field:

{
            _id: ObjectId('4f442120eb03305789000000'),
            line: '127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 "[http://www.example.com/start.html](http://www.example.com/start.html)" "Mozilla/4.08 [en] (Win98; I ;Nav)"'
            }

This works but makes querying difficult. ApsaraDB for MongoDB is not a text-analysis engine, so raw log strings are hard to filter or aggregate.

Parse into structured fields

Before inserting, parse each log line into separate fields:

{
            _id: ObjectId('4f442120eb03305789000000'),
            host: "127.0.0.1",
            logname: null,
            user: 'frank',
            time: ISODate("2000-10-10T20:55:36Z"),
            path: "/apache_pb.gif",
            request: "GET /apache_pb.gif HTTP/1.0",
            status: 200,
            response_size: 2326,
            referrer: "[http://www.example.com/start.html](http://www.example.com/start.html)",
            user_agent: "Mozilla/4.08 [en] (Win98; I ;Nav)"
            }

Structured fields let you filter and index specific values — a significant advantage over storing raw lines.

Trim unused fields

Large documents hurt performance and increase cost. Drop fields you will not query. In this example, user, request, and status are rarely needed for analysis. The _id ObjectId already encodes the insertion time, so you can also drop time — though keeping it makes time-range queries simpler to write. A lean document looks like this:

{
            _id: ObjectId('4f442120eb03305789000000'),
            host: "127.0.0.1",
            time: ISODate("2000-10-10T20:55:36Z"),
            path: "/apache_pb.gif",
            referer: "[http://www.example.com/start.html](http://www.example.com/start.html)",
            user_agent: "Mozilla/4.08 [en] (Win98; I ;Nav)"
            }

Use compact data types where possible. For example, store HTTP status codes as integers rather than strings.

Write logs to ApsaraDB for MongoDB

Log pipelines often need to sustain high write throughput. Control the durability versus performance trade-off by setting a write concern on each insert:

db.events.insert({
                host: "127.0.0.1",
                time: ISODate("2000-10-10T20:55:36Z"),
                path: "/apache_pb.gif",
                referer: "[http://www.example.com/start.html](http://www.example.com/start.html)",
                user_agent: "Mozilla/4.08 [en] (Win98; I ;Nav)"
                },
                {
                 writeConcern:{w: 0} 
                }
                )

Note

For maximum write throughput, set the write concern to {w: 0}.
For logs that carry sensitive or billing-related information, set the write concern to {w: 1} or {w: "majority"} to ensure durability.

To reduce round trips, batch multiple log documents into a single insert:

db.events.insert([doc1, doc2, ...])

Query logs in ApsaraDB for MongoDB

With structured fields in place, you can run targeted queries against the collection.

Query all requests for a specific path: q_events = db.events.find({'path': '/apache_pb.gif'})

Note
If you run this query frequently, create an index on the path field: db.events.createIndex({path: 1}).
Query all requests within a time range:
```
q_events = db.events.find({'time': { '$gte': ISODate("2016-12-19T00:00:00.00Z"),'$lt': ISODate("2016-12-20T00:00:00.00Z")}})
```
Note
Create an index on the time field to speed up time-range queries: db.events.createIndex({time: 1}).
Query all requests from a specific host within a time range:
```
q_events = db.events.find({
                'host': '127.0.0.1',
                'time': {'$gte': ISODate("2016-12-19T00:00:00.00Z"),'$lt': ISODate("2016-12-20T00:00:00.00Z" }
                })
                    
```
For more complex analysis — grouping, counting, or computing aggregates — use the aggregation pipeline or MapReduce framework. Index the fields you filter or sort by to keep query latency low.

Data sharding

As log volume grows, a single node's write and storage capacity becomes a bottleneck. Shard the collection to distribute data across multiple nodes. The shard key choice directly affects write throughput and query performance.

Timestamp-based shard key (for example, _id or time): Because timestamps increase monotonically, all new log inserts land on the same shard. This creates a write hotspot and means recent-data queries only hit one or two shards rather than distributing load across the cluster.
- New log data concentrates on one shard — write throughput does not scale with the cluster.
- Queries for recent logs target only a few shards, so the cluster's full query capacity goes unused.
Hashed sharding: Uses _id as the default shard key. Hashing distributes inserts evenly across all shards, so write throughput scales linearly with the number of shards. The trade-off is that range queries — common in log analysis — must scatter across all shards and merge results, which increases latency.
Ranged sharding on a well-distributed field: If a field like path has many distinct, evenly distributed values and is commonly used in queries, it makes a good shard key. Benefits:
- Write requests spread evenly across shards.
- Range queries on path target a small number of shards, improving efficiency.
The downsides:
- A high-frequency path value concentrates its documents in a single chunk or shard, creating a hot spot.
- A low-cardinality path field leads to uneven data distribution.
To address both hot spots and low cardinality, add a secondary field to the shard key. For example, extend {path: 1} to {path: 1, ssk: 1}.

Assign a random value to ssk — such as the hash of _id — to spread hot paths across shards. Alternatively, use a timestamp so documents with the same path sort chronologically within each shard.

The composite key prevents any single shard key value from dominating, balancing write throughput and query locality. Each sharding strategy has trade-offs; choose based on your access patterns and growth trajectory.

Manage data growth

Log data accumulates quickly, but its value decreases over time. Data from three months ago rarely affects current analysis, yet it consumes storage and increases cost. ApsaraDB for MongoDB provides three approaches to manage data lifecycle.

TTL (Time To Live) indexes: A TTL index is a single-field index that automatically removes documents after a specified period. To expire log entries 30 hours after the request time, run: db.events.createIndex( { time: 1 }, { expireAfterSeconds: 108000 } ).

Note
The background task that deletes expired documents runs every 60 seconds and is single-threaded. Under high write loads, expired documents may accumulate faster than the task can remove them, temporarily consuming extra storage.
Capped collections: To cap storage space rather than enforce time-based expiry, use a capped collection. When the collection reaches its configured size or document limit, MongoDB automatically removes the oldest documents. Example: db.createCollection("event", {capped: true, size: 104857600000}.
Periodic archival by collection: At the end of each month, rename the current collection and create a new one for the next month. Include the year and month in the collection name:
```
events-201601
                events-201602
                events-201603
                events-201604
                ....
                events-201612
                    
```
To purge a month's data, drop the corresponding collection:
```
 db["events-201601"].drop()
                db["events-201602"].drop()
                    
```
The trade-off: cross-month queries are more complex because results must be merged from multiple collections.