FORGE: The Hands-Free Voice Copilot for the Technician With Greasy Gloves

This article introduces FORGE, a hands-free, voice-driven AI copilot that provides industrial technicians with real-time, verified machine guidance and automated documentation.

Honorable Mention, Global AI Hackathon Series with Qwen Cloud

cover

Project Introduction

A field technician is elbow-deep in a machine, gloves on, hands greasy. The torque value they need is in a binder across the room. The schematic is on a laptop. The fault history is in a system three logins away. So they stop, strip off the gloves, break focus, and look it up — or they guess. FORGE is built to end that choice. It's a hands-free AI co-pilot for industrial field-service technicians, built on Qwen's realtime omni model on Alibaba Cloud. The technician just talks, and FORGE sees the machine, shows the right information on screen, guides the work safely, and writes the work order as the job happens.

Project Value

The gap between "hands on the machine" and "information in hand" is staggeringly expensive. Unplanned downtime in manufacturing averages roughly $260,000 per hour, and that cost has risen sharply in recent years. About 1 in 4 service visits fails to fix the problem the first time — and in industrial machinery it's nearly 3 in 10. The most common reason? The right information, schematic, or diagnosis simply wasn't in the technician's hands when the decision was made.

Paperwork compounds it: around 75% of technicians say documentation eats too much of their time, and reports written from memory hours later lose accuracy exactly when it matters for compliance, warranty, and the next shift. But speed alone isn't the answer — trust is. A co-pilot that confidently invents a torque spec or part number doesn't save time; it can destroy a spindle. FORGE delivers verified answers at the moment of decision, catches problems before they become failures, documents the job as it happens, and never invents a fact: every number it speaks comes from a verified source.

Solution

FORGE listens continuously. The technician speaks naturally — no tapping, no glove removal — and FORGE responds in under a second with a brief spoken reply and the right information on screen. It runs on Qwen's realtime omni model, which listens, speaks, sees the live camera feed, and calls tools all in a single session.

Through 25 tools, FORGE briefs on a machine, looks up parts and torque specs, navigates schematics and a 3D model, marks issues directly on the camera view, records measurements, and guides procedures step by step. Every request is checked against the machine catalog, so unknown parts are _refused, not guessed_. Safety is enforced on the server, not trusted to the model: the lockout/tagout checklist requires a spoken confirmation for each individual item. FORGE raises alerts when a measurement breaks a failure rule, while a second model diagnoses faults in the background without interrupting the conversation. At the end of the job, FORGE turns its event log into a work-order report and a shift handoff — documentation that writes itself, accurately, in the moment.

Highlights

FORGE runs end-to-end on Alibaba Cloud, powered by two Qwen models through Model Studio. qwen3.5-omni-plus-realtime is the front-line agent — a single realtime session carrying the technician's voice, spoken replies, live camera frames, and all 25 tools, keeping responses under a second. qwen-plus is the background diagnosis engine, turning fault data and measurements into a structured diagnosis without slowing the conversation.

The backend is hosted on Elastic Compute Service (ECS) over HTTPS and was verified live end-to-end; Object Storage Service (OSS) provides cloud storage; and a health endpoint reports live status plus the Qwen model in use as deployment proof. An automated pipeline runs the tests, builds the container image, pushes it to Container Registry (ACR), and rolls it out to ECS. One notable design decision: the builder built and tested a multi-agent handoff layer, then _shipped without it_, running a single realtime session instead to eliminate handoff delays and lost actions — while on-screen indicators still show which of ten specialist roles handled each request. FORGE handles mid-sentence interruptions, resumes seamlessly past the realtime model's two-hour session limit, executes duplicate commands only once, and shuts down cleanly on errors. The console is fully voice-driven, built with React, TypeScript, and Three.js, and backed by 191 automated tests.

Alibaba Cloud Products Used

About the Team

Kaushik Chaturvedula is an AI engineer and independent ML researcher who designs and builds agentic AI and retrieval-augmented generation (RAG) systems professionally. He built FORGE on his own, from start to finish, on a conviction he states simply: AI is most useful when it works where people actually work. For a field technician, that means hands-free, grounded in verified data, and fast enough to keep up. Fittingly, FORGE is honest about its own scope — at an unfamiliar machine, it says its specs cover only the cataloged machine and offers general guidance instead of bluffing.

Explore the project: https://youtu.be/ifMU-fvNbVk

This project was developed as part of the Global AI Hackathon Series with Qwen Cloud.

Community

FORGE: The Hands-Free Voice Copilot for the Technician With Greasy Gloves

Project Introduction

Project Value

Solution

Highlights

Alibaba Cloud Products Used

About the Team

Read previous post:

Alibaba Cloud Project Hub

You may also like

Comments

Alibaba Cloud Project Hub

Related Products

Token Plan

QwenWork

Qwen

DeepSeek