Plaintext Stays Put, Logs Stay Useful: Sensitive Data Protection with Alibaba Cloud Observability

AI Agents are filling logs with more sensitive data in more complex forms. Beyond traditional fields such as phone numbers, IP addresses, and API keys, AI coding Agents, retrieval-augmented generation (RAG) applications, and operations-focused AI Agents may also read source code, configuration files, environment variables, and database connection strings. They then write tool calls, terminal output, and model responses to audit logs, making it easier for sensitive information to enter the logging pipeline.

You cannot simply discard this data; it provides essential evidence for troubleshooting, auditing, and governance. To keep data both secure and useful, Alibaba Cloud Observability protects sensitive data across collection, ingestion, and transformation. Data can be masked or encrypted before it reaches a LogStore, then transformed after ingestion into secure datasets suitable for long-term retention and shared analysis.

Let's start with the overall design, then look at how each method works in practice.

Reduce Exposure Without Sacrificing Value: Protect Sensitive Data End to End

The Simple Log Service (SLS) approach to sensitive data protection comes down to two questions: how should you protect the content, and where in the data pipeline should that protection happen? The first determines whether to use masking, symmetric encryption, or envelope encryption. The second determines whether to process the data on the client side, in an ingest processor, or through data transformation.

You can combine these two dimensions flexibly. The same protection method can run during collection, before ingestion, or after data reaches a LogStore; at any of these stages, you can choose masking or encryption based on the risk. Match the combination to the data source, compliance requirements, and downstream analysis. Use client-side processing when plaintext must not leave the host, an ingest processor to govern direct writes from multiple sources, and data transformation to create a protected LogStore for long-term sharing or governance of existing data.

Choose Based on How the Data Will Be Used

Different types of sensitive data call for different treatment. SLS provides three types of SPL functions: masking, symmetric encryption, and envelope encryption. These options address three needs: preserving readability, allowing authorized recovery, and separating duties.

Masking: Keep Logs Readable and Reduce Privacy Risk

Use mask for personally identifiable information (PII) or credentials scattered through free text. It supports two matching modes:

buildin: matches built-in patterns for mobile phone numbers, email addresses, national ID numbers, bank card numbers, IP addresses, landline numbers, AccessKey pairs, license plates, international mobile subscriber identity (IMSI) values, vehicle identification numbers (VINs), and more;
keyword: matches field or key names such as password, token, authorization, and api_key.

You can replace an entire value with a placeholder or retain a prefix and suffix. Keeping the first three and last four digits of a phone number, for example, protects privacy while still allowing a rough cross-check against application records during troubleshooting. Masked logs remain readable, searchable, and aggregatable, making masking a good default for most logging scenarios. For implementation details, see the page titled Use the mask function for data masking.

Common use cases include:

phone numbers, email addresses, and national ID numbers embedded in AI gateway prompts;
client IP addresses and authentication tokens in application access logs;
bank card numbers, addresses, and user identifiers in LLM application logs.

Symmetric Encryption: Store Ciphertext and Decrypt It When Authorized

Use aes_encrypt when data must be recoverable later and key ownership is clearly defined. Content such as internal RAG Q&A, model requests, and customer solution documents can be stored as ciphertext, then decrypted during an authorized audit by the team that holds the key and initialization vector (IV).

If one team handles collection, analysis, and auditing, it can manage the key and IV centrally, making symmetric encryption a simple, high-performance choice. The boundary is equally clear: once the key is shared, every holder can decrypt the data. This makes symmetric encryption suitable within one organization or security domain, but not for broad distribution across teams.

Envelope Encryption: Separate Public and Private Keys for Collaboration on Highly Sensitive Data

Use envelope_encrypt when data collection and decryption are handled by different parties. It generates a random data encryption key (DEK) to encrypt the application data, then encrypts the DEK with an asymmetric public key. The collection side holds only the public key; a separate security or audit team safeguards the private key.

This pattern is well suited to highly sensitive content, including:

the full output returned after an AI coding tool reads a file;
command output;
entire log entries containing keys, connection strings, or configuration files.

Envelope encryption also provides predictable performance. A DEK is generated at the pipeline level, and asymmetric encryption applies only to that key; AES still encrypts the application data. This minimizes the effect on collection throughput.

Protection methods are not tied to specific processing stages. The following sections trace three points along the data path and use one representative method for each, but every stage can support other methods as well.

Client-Side Protection: Keep Plaintext at the Source

In stricter compliance environments, protection must happen at the collection source. Logs are processed as soon as they are collected, so plaintext never leaves the host or container or enters the downstream transport pipeline.

Client-side protection comes in two forms. LoongCollector processes host and container logs as they are collected, while LoongSuite Pilot provides on-device privacy controls for AI coding Agents. Together, these approaches contain plaintext before data leaves its source.

LoongCollector: Process Data as You Collect It Without Changing the Application

Note: All names, phone numbers, email addresses, public keys, private keys, and other sensitive data shown in this article are synthetically generated.

For applications running on Elastic Compute Service (ECS) instances, Container Service for Kubernetes (ACK) clusters, on-premises data center hosts, or containers, LoongCollector can run SPL during collection instead of acting solely as a collector. AI gateways, Alibaba Cloud Model Studio applications, RAG services, and model proxy layers typically produce large volumes of access and interaction logs. These logs combine structured fields with natural-language input. Processing them at the collection source significantly reduces plaintext exposure.

Consider an AI gateway access log. A raw entry might contain the caller's IP address, an API key, and the user's prompt:

model=qwen-max client_ip=203.0.113.55 prompt="My name is Zhang. My phone number is 138xxxxxxxx, and my email address is zhangwei@xxxx.com. Please look up my recent orders." tokens=128

In the LoongCollector collection configuration, mask can handle built-in PII patterns and keyword fields in a single pass:

After the log is written to SLS, it can still be used to analyze model call volume, token usage, and gateway access trends, but the sensitive values have been replaced:

This approach works well for host- and container-side data such as application logs, gateway logs, and model proxy logs. It requires no changes to application code: processing happens in the collection configuration, and logs reach SLS in a safe, masked form.

LoongSuite Pilot: Mask AI Coding Data Before Distribution

Another common client-side use case is auditing AI coding activity. Coding Agents running on a developer's machine often access source code, configuration files, and local environment variables. If a developer pastes a configuration snippet into a prompt, or an Agent reads a .env file and writes its contents into tool-call arguments, API keys, cloud-provider access key/secret key pairs (AK/SK), and private keys can enter the collection pipeline.

LoongSuite Pilot is designed for these on-device AI coding scenarios. It discovers and collects activity data from Claude Code, Cursor, Codex, Qoder, and similar tools, then normalizes the data into audit events for sessions, tool calls, model calls, and token usage. Its built-in rule-based masking engine scans and replaces sensitive content before sending data to any output channel. Once masking is enabled, the same rules apply whether the destination is SLS, JSONL, HTTP, or OpenTelemetry Protocol (OTLP).

Pilot's automatic masking covers these common credential types:

Masking type	Coverage	Replacement marker
Cloud AccessKey	Alibaba Cloud LTAI, AWS AKIA/ASIA, Tencent Cloud AKID	`[ACCESSKEY_MASKED]`
API key	OpenAI-compatible `sk-` keys; GitHub personal access tokens (PATs) with `ghp_`, `gho_`, or `ghs_` prefixes	`[APIKEY_MASKED]`
Database connection string	MySQL, PostgreSQL, MongoDB, and Redis URIs; password-bearing JDBC URLs	`[DATABASEURL_MASKED]`
Private key	PEM/OpenSSH private key blocks	`[PRIVATEKEY_MASKED]`

For example, in an AI coding session, an Agent reads a local .env file and produces the following tool result:

DB_URL=mysql://root:Abc@2025!@10.0.0.12:3306/prod
ALIYUN_AK=LTAI5tFakeExampleKey0001
-----BEGIN RSA PRIVATE KEY-----
MIIEvQIBADANBgkqh...
-----END RSA PRIVATE KEY-----

After LoongSuite Pilot masks the data, SLS receives:

mask.mode controls masking, with three supported values: none, all, and custom. Masking is disabled by default. Once enabled, sensitive credentials no longer reach downstream outputs in plaintext. Audit metadata such as session_id, tool_name, token usage, and execution duration remains available, so engineering managers and security teams can still analyze AI coding activity, tool-call risks, and cost distribution.

In summary, client-side protection is a good fit when:

plaintext must not leave an application host or developer machine;
the collection pipeline crosses network and team boundaries, making plaintext transmission unacceptable;
you want to protect sensitive data at the point where logs are created without changing application code.

Protect Data Before Ingestion: Govern Direct Writes Centrally

Not all data passes through LoongCollector. Many systems write directly to SLS through SDKs, web tracking, OpenAPI, or third-party collection pipelines. For such data, an ingest processor provides centralized server-side governance: after the data reaches SLS but before it enters a LogStore, the processor runs SPL to mask or encrypt it.

An ingest processor's key advantage is centralized configuration. Rules live in SLS, so clients do not need to be upgraded one by one. When a processing policy changes, you update the processor once to cover all direct writes. This is easier to roll out across multiple business lines, SDKs in different programming languages, and third-party systems.

Consider an AI gateway access log. The gateway joins the fields for each model call into a line separated by ## and writes it directly to SLS. The request field contains the complete model request body and may include arbitrary sensitive content, such as the user's prompt or national ID number. The other fields contain structured call metadata:

2026-06-25 11:17:25.471##c9d7ab4c-b312-9d08-9243-c7f7475905be##5000000276111136##API_KEY##qwen-max##text-generation##{"input":{"prompt":"My national ID number is 53010219200508011x. Please check my insurance policy status."}}##chat-app##3de69809a9d0fd22773f33c77dab12a8

Here, request contains the entire unstructured request body. To show how an ingest processor can apply one policy to all direct writes before storage, this example uses symmetric encryption for the entire request field: it first splits the line on the delimiter, then applies aes_encrypt only to the sensitive request body.

Encrypt a Highly Sensitive Request Body in an Ingest Processor

Ingest processor SPL configuration:

* | extend temp = split(content, '##')
  | extend time          = temp[1],
           request_id    = temp[2],
           caller_uid    = temp[3],
           source        = temp[4],
           model         = temp[5],
           api_type      = temp[6],
           request       = temp[7],      
           service_name  = temp[8],
           trace_id      = temp[9]
  | extend request = to_base64(aes_encrypt(
        cast(request as binary),
        cast('aZ4kP9mX2qL7nR1t' as binary),
        cast('Gh8dW3sB6vN0cT5e' as binary)))
  | project-away content, temp

After ingestion, request is Base64-encoded ciphertext, while call metadata such as request_id, region, model, service_name, and trace_id remains in plaintext:

Operations engineers can troubleshoot the request path using request_id and trace_id, while business operations teams can count calls by model and service_name. The request body itself remains encrypted.

Decrypt Data Only When Authorized

When the original data is needed, an authorized team member who holds the key and IV decrypts it at query time:

Data Transformation: Build Secure Datasets for Shared Analysis

Client-side processing and ingest processors protect data before storage. Data transformation addresses a different problem: how to reorganize data already in a LogStore into datasets that are safer and better suited to long-term analysis.

In practice, many teams maintain two LogStores:

Raw LogStore: retains data briefly, has strict access controls, and is used only for emergency troubleshooting or security investigations;
Protected LogStore: stores the output of a data transformation task; its sensitive fields are masked or encrypted, and it retains data longer to support analysis and audits for a broader audience.

This pattern works especially well for AI observability data. Consider AI coding activity logs collected by LoongSuite Pilot from tools such as Qoder. Each llm.response event contains structured metadata, including the session ID, model, and user ID, plus a gen_ai.output.messages field. That field contains the full text of the model's response for the current turn and may include internal system names, host details, configuration data, keys, or other sensitive content:

{
  "event.name": "llm.response",
  "gen_ai.session.id": "58bf461a-285f-468e-963c-00cd1706a9d9",
  "gen_ai.provider.name": "qwen",
  "gen_ai.response.model": "auto",
  "user.id": "1",
  "gen_ai.output.messages": "[{\"role\":\"assistant\",\"parts\":[{\"type\":\"text\",\"content\":\"Candidate salary table prepared: Zhang （phone number 138xxxxxxxx）...\"}],\"finish_reason\":\"end_turn\"}]",
  "trace_id": "a58ed78ac2573159b4fa8b38ab4c2697",
  "__time__": "1782367255"
}

Because gen_ai.output.messages contains a complete model response with no fixed structure, field-by-field masking cannot cover every risk. Yet an investigation may still need the original text. This example uses envelope encryption to show how data transformation can first derive the analysis field msg_len from the response length, then encrypt the sensitive body and write it to a protected LogStore:

* | extend msg_len = length("gen_ai.output.messages")
  | extend result = envelope_encrypt(
      cast("gen_ai.output.messages" as binary),
      '-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0Vh1QIpNurP0bB9uHRK3
ZL29SjY74OtYa5nmf9xUMKAuLrio5BN6LRZyG680uCkCThCMOK/T9ka5ZYtEIu8b
U7acNR1dL3LWsbZG3ScbqyISJ3pEBli2rCgOqBQK3EE+6HGavh6wo8FAhkSBiryU
eVKxSJSnC928RvlBWtyfdubVYQHPONhx0TbCj38Njjfi8kbKGgE92q9gTlL+spge
qvgDec9GV2VzxhLoTgxd6jdHYyfFzuQucb0hZtmWqus4E4MTnAXnUJOGizmI02AP
5d49P44kZXElBPRw4A0WISKBmwOzYCZqPSEWyHnj/qIaXAbz90B8TCUiPZlNouCG
TwIDAQAB
-----END PUBLIC KEY-----',
      '{"mode":"RSA"}',
      '{"mode":"CTR"}')
  | extend encrypted_dek = result['encrypted_dek'],
           enc_output_messages = result['encrypted_data']
  | project-away "gen_ai.output.messages"

In the record written to the protected LogStore, gen_ai.output.messages is removed. It is replaced by enc_output_messages, which contains the ciphertext, and encrypted_dek, which contains the public-key-encrypted DEK. The remaining metadata stays in plaintext:

The protected LogStore retains fields such as event type, session ID, model, user ID, and response length, so teams can still use it for alerts, trend analysis, and risk investigations. The model response itself is encrypted. For deeper investigations, authorized security personnel can decrypt selected records with the private key.

This strikes a balance between encryption and analyzability: rather than encrypt every field, extract the metadata needed for analysis first and encrypt only the highly sensitive body. Routine analysis uses the metadata, while the original sensitive content can be accessed only through authorized decryption.

How to Choose: Plaintext Boundary, Analysis Needs, and Decryption Permissions

Ask three questions when choosing an approach.

First, can plaintext leave the collection source? If not, process it at the source.

Second, must the logs remain readable for routine analysis? If so, preserve readability. Encrypt only when the content is highly sensitive as a whole and cannot be reliably protected field by field.

Third, who is allowed to decrypt the data? One team within a single security domain can manage keys centrally. Across teams, or when duties must be separated, decryption authority should remain under the independent control of the security or audit team.

The following table summarizes the options:

Scenario	Recommended processing stage	Recommended method
AI gateway logs on hosts or in containers; plaintext must not leave the host	LoongCollector on the client side	Data masking with mask; encrypt selected fields as needed
SDKs or third-party systems writing directly to SLS	Ingest processor	Masking or encryption before ingestion
Historical logs that require protection	Data transformation	Transform data from the source LogStore, then write masked or encrypted output to a new LogStore
Internal RAG Q&A content that requires controlled recovery	Client-side processing or ingest processor	Symmetric encryption with aes_encrypt
Entire blocks of highly sensitive content, such as tool output or configuration files	Client-side processing or data transformation	Envelope encryption with envelope_encrypt

Keep Logs Secure Without Losing Their Value

Logs make systems observable and support analysis and traceability. Sensitive data protection should preserve that value within a clear security boundary. SLS applies masking and encryption during collection, ingestion, and transformation, helping organizations reduce plaintext exposure while retaining the analysis they need.

Organizations can choose masking or encryption based on the data type. They can apply protection during collection, before ingestion, or after ingestion through data transformation, then choose symmetric or envelope encryption based on who controls the keys. Together, these choices define a clear, enforceable boundary that keeps data both useful and secure.

Community

Plaintext Stays Put, Logs Stay Useful: Sensitive Data Protection with Alibaba Cloud Observability

Reduce Exposure Without Sacrificing Value: Protect Sensitive Data End to End

Choose Based on How the Data Will Be Used

Masking: Keep Logs Readable and Reduce Privacy Risk

Symmetric Encryption: Store Ciphertext and Decrypt It When Authorized

Envelope Encryption: Separate Public and Private Keys for Collaboration on Highly Sensitive Data

Client-Side Protection: Keep Plaintext at the Source

LoongCollector: Process Data as You Collect It Without Changing the Application

LoongSuite Pilot: Mask AI Coding Data Before Distribution

Protect Data Before Ingestion: Govern Direct Writes Centrally

Encrypt a Highly Sensitive Request Body in an Ingest Processor

Decrypt Data Only When Authorized

Data Transformation: Build Secure Datasets for Shared Analysis

How to Choose: Plaintext Boundary, Analysis Needs, and Decryption Permissions

Keep Logs Secure Without Losing Their Value

Read previous post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

Simple Log Service

Data Security Center (Original SDDP)

Conversational AI Service

Log Management for AIOps Solution