Note: In this article, SysOM refers to an O&M component of the Alibaba Cloud OS Console, and the SysOM Skills Package is a collection of SysOM skills.
Anyone in IT operations who has been woken up at 2:00 AM knows the feeling. What's worse than an incident occurring is having all the data right in front of you but still not knowing what to do next.
Not long ago, we handed over the tedious process of manually logging into machines to find root causes after an alert to an Agent. However, diagnostics only solve the problem of finding the root cause after something breaks. A better approach is to detect hidden risks before issues occur.
With this in mind, the Alibaba Cloud OS Console is open-sourcing the SysOM Inspection Skill today. Moving forward, it will be integrated alongside the SysOM Diagnostic Skill into the upcoming SysOM Skills Package. This creates a complete closed loop: inspection to discover issues → diagnostics to locate the root cause → actionable recommendations. This empowers the Agent to not only mitigate damage after the fact but also prevent issues proactively.
It was an ordinary Thursday morning. Lin was jolted awake by an @all notification in the on-call group chat. As his screen lit up, he was greeted by a cold, alarming message:
"Memory usage of an instance in the production environment is at 91.90% and has persisted for 30 minutes."
What happened next? The same familiar drill that anyone in IT operations has gone through:
Open the jump server, log into the machine, and type top—a full screen of processes flashes by, making it impossible to immediately spot the critical one. Type free -h, and only a little over a hundred megabytes of free memory remain. Check /proc/meminfo, and the usage of anonymous pages is abnormally high. Set up an emergency group chat, pulling in the SRE, DBA, and business developers to take a look. Someone asks if the cache wasn't released, another suspects an application leak, and someone else suggests a quick restart...
That night, it took the team nearly 40 minutes from receiving the alert to finally pinpointing the rogue Python process consuming 12.95 GB of memory. They barely managed to stabilize the service before the next traffic spike hit. But Lin knew the truth—they just got lucky this time.
If he had the SysOM Inspection Skill in his toolkit that night, things would have played out very differently.
It's the same machine, experiencing the same massive memory spike. But this time, the SysOM Inspection Skill is already running its routine inspection. Just 37.4 seconds later, a comprehensive report lands in Lin's hands:
19 inspections completed: 18 Normal, 1 Error hit—memory usage at 91.90%. With only 161 MB of free memory left, another traffic wave means an OOM error is imminent.
But this time, the report didn't just stop at "high memory usage." The moment the error was triggered, memgraph (SysOM 's built-in deep memory analysis tool) automatically kicked off a deep diagnostic process. The answer was laid out plainly on the very next screen:
The inspection and diagnostics pinpointed that the user's Python process was exclusively consuming 12.95 GB of anonymous memory, accounting for roughly 83% of total system memory. This was the direct cause of the spike. Since the combined memory of all other processes in the TOP 10 was less than 600 MB, the possibility of stacked fragmented usage could be immediately ruled out.
Now, Lin only had one thing left to do: confirm whether this Python process was expected behavior. If not, a simple kill command would instantly free up 12 GB of memory.
From being woken up to resolving the issue, the timeline was slashed from forty minutes to under five.
The true cost of a late-night wake-up call is never just a few dozen minutes of overtime. It's the on-call engineer's inevitable 3:00 PM slump the next day. It's the shattered focus of five other team members dragged into an emergency chat. It's the erosion of trust every time the business team asks, "Did something break again?" And it's the agonizing uncertainty of every ticking second during peak traffic. These costs are hard to quantify, but anyone who has worked in IT operations knows the toll all too well.
Multiply these hidden costs by 365 days a year, then by your team size, and you get the real ledger of your stability investments. What the SysOM Inspection Skill aims to change isn't just saving those forty minutes. It's about overhauling the outdated model of "relying on personal experience, improvising under pressure, and hoping whoever is on call tonight is wide awake." It transforms individual O&M experience into a scalable organizational capability.
A story that begins in the small hours of a Thursday can unfold in one of two ways: forty minutes of frantic scrambling, or a thirty-seven-second report. The SysOM Inspection Skill is designed to make every inspection a little less about surprises and a little more about conclusions — so that your machines are already prepared before the next traffic peak arrives.
Ultimately, the value of an inspection doesn't lie in how many checks it runs, but in whether it can actually answer the most pressing questions frontline engineers have.
Many teams have spent years refining their inspection processes, successfully solving the basic problem of detecting "whether an anomaly exists." However, in real-world IT operations, when an alert fires, frontline engineers care much more about three advanced questions: How severe is this anomaly? Where is the most likely root cause? And who needs to take what action next?
The SysOM Inspection Skill is designed specifically to shift the answers to these three questions left, integrating them directly into the inspection process. Currently, the product covers over 40 inspection capabilities across multiple dimensions. These include system load, memory usage, disk read/write latency, scheduling latency, file handles, thread resources, root partition and inode status, socket leaks, TCP/UDP memory, and various memory leak risks. By spanning everything from basic performance metrics to kernel-space risks, it comprehensively brings all critical failure-inducing signals onto the inspection radar.
For high-risk scenarios, it doesn't just issue a warning and call it a day. Instead, it automatically triggers the SysOM Diagnostic Skill, covering multiple OS subsystems like load, memory, network, disk, and scheduling, tracing the troubleshooting path all the way down to the root cause. In internal evaluations, the overall accuracy of anomaly detection exceeds 80%, with a 0% false-positive rate for high-risk items. In other words, when it tells you "there's a high risk here," it's not crying wolf.
The SysOM Inspection Skill encapsulates the troubleshooting expertise of kernel specialists. When handling Linux system issues, the AI no longer has to "guess" on its own; it directly invokes a proven troubleshooting path. To see if this expertise actually works, we selected 16 categories of real-world OS failure scenarios and conducted a controlled evaluation. One group of Agents was equipped with the SysOM Inspection Skill, while the control group had to brute-force the troubleshooting using only general AI capabilities.
Let's look at efficiency first. When facing the same type of system issue, Agents equipped with the Skill showed a drastic reduction in conversation turns, tool invocations, and overall time spent. Instead of repeatedly trying different approaches and splicing commands, they followed the expert path straight to the solution. For system failures that typically require assembling a team chat and cross-verifying with multiple tools, the savings here go far beyond just inference time—they preserve frontline focus and eliminate massive context-switching costs.
Next, let's look at accuracy. Overall, Agents using the Skill demonstrated a significant boost in pinpointing kernel-related issues compared to the baseline. Breaking it down by specific scenarios, the gap was most pronounced in areas requiring expert judgment. For kernel-layer problems like socket buffer leaks, vmalloc memory anomalies, and fundamental memcg misjudgments, the accuracy improvement in individual scenarios was remarkably high. This aligns perfectly with an intuitive conclusion: the more a kernel issue relies on expert experience, the greater the value the Skill provides.
In other words, we've taken the same set of expert capabilities and offered them in two forms. For humans, it's an out-of-the-box inspection product; for Agents, it's a directly callable capability. Whether you're building your own O&M AI or just want to use a ready-made inspection service, you no longer need to build those proven Linux troubleshooting experiences from scratch.
The SysOM Inspection Skill is now open-source on the Alibaba Cloud Skills Platform (copy the link to open in your browser): https://skills.aliyun.com/skills/alibabacloud-alinux-sysom-inspection. This skill includes complete troubleshooting paths, diagnostic prompts, and related scripts, which can be directly loaded and used by common Agent environments like Qoder and Claude Code. By pairing it with the previously open-source SysOM Diagnostic Skill, you can establish a complete workflow: inspection to discover issues → diagnostics to locate the root cause → actionable recommendations. Both will soon be integrated into the upcoming SysOM Skills Package (prior to its official release, both the Inspection Skill and Diagnostic Skill can be installed and used independently).
Installation takes just one line of code (using Qoder as an example):
npx skills add aliyun/alibabacloud-aiops-skills --skill alibabacloud-alinux-sysom-inspection --agent qoder -y --full-depthOnce installed, simply type your request directly into the Agent chat. For example, "Help me run an inspection on instance i-xxx in the cn-shenzhen region." The Agent will automatically invoke the inspection capabilities within the Skill. If high-risk items are detected, it seamlessly triggers SysOM's deep diagnostics, ultimately outputting a comprehensive report detailing anomalies, key findings, and the root cause. Installation instructions for different Agent hosts can be found on the Alibaba Cloud Skills Platform.
The story of that early Thursday morning could be forty minutes of frantic chaos, or it could be a complete report delivered in thirty-seven seconds. The SysOM Inspection Skill aims to make every inspection yield fewer surprises and more actionable conclusions—ensuring your machines are fully prepared before the next traffic spike hits.
From detecting anomalies to pinpointing the root cause and providing actionable recommendations, the SysOM Inspection Skill brings every inspection one step closer to rock-solid business stability.
Contact Us:
If you have any questions or suggestions while using the OS Console, you can scan the QR code below or search for the group number: 94405014449 to join our DingTalk feedback group. We welcome everyone to scan the code and join the conversation!
AgentLoop Audit: Fishing Out Real Risks from Massive 'Noise'
761 posts | 60 followers
FollowOpenAnolis - August 4, 2026
OpenAnolis - July 14, 2026
CloudSecurity - April 30, 2026
OpenAnolis - May 27, 2026
OpenAnolis - March 25, 2026
OpenAnolis - September 4, 2025
761 posts | 60 followers
Follow
Bastionhost
A unified, efficient, and secure platform that provides cloud-based O&M, access control, and operation audit.
Learn More
Alibaba Cloud Linux
Alibaba Cloud Linux is a free-to-use, native operating system that provides a stable, reliable, and high-performance environment for your applications.
Learn More
Managed Service for Grafana
Managed Service for Grafana displays a large amount of data in real time to provide an overview of business and O&M monitoring.
Learn More
Managed Service for Prometheus
Multi-source metrics are aggregated to monitor the status of your business and services in real time.
Learn MoreMore Posts by Alibaba Cloud Native Community