AI Coding Assistant Lingma:Asiainfo: intelligent R&D practices

AsiaInfo focuses on B2B operations. Founded in 1993, AsiaInfo became one of the first high-tech companies to be listed on National Association of Securities Dealers Automated Quotations (NASDAQ) in 2000. By 2010, AsiaInfo became a leading telecommunications software product and service provider in China. The company went private in 2014 and relisted on the Hong Kong Main Board in 2018. In 2022, AsiaInfo acquired iResearch, further expanding its digital consulting business. The company developed three main product systems: digital, cloud, and network IT. This makes AsiaInfo a leading provider of IT products and services and a pioneer in full-stack data capabilities.

AsiaInfo continuously grows its revenue and strives to reach 10 billion. The core products of AsiaInfo hold approximately 50% market share in the business support system software sector of the telecom industry, positioning it as an industry leader. AsiaInfo products serve over 1 billion users. This topic describes the following aspects of intelligent R&D practices of AsiaInfo: intelligent R&D background, intelligent coding assistant, practice process, evaluation results, and future plans.

01

Intelligent R&D background

AsiaInfo faces challenges, such as operational pressure. The financial reports of AsiaInfo show slowing growth in the telecom industry. To resolve this issue, AsiaInfo needs to develop tools and products that can reduce costs and improve efficiency. AsiaInfo also faces pressure due to customer demands for quality. Most customers are increasingly focusing on strengthening security and quality management. AsiaInfo aims to introduce tools and products that can improve quality and reduce production failures and high-risk vulnerabilities. The company is a software company primarily composed of technical personnel, with approximately 15,000 employees. With intelligent R&D, achieving a 10% or 5% improvement can lead to significant benefits. AsiaInfo has a strong demand for intelligent R&D.

Observing industry trends, we see that major companies like Alibaba, Baidu, and Microsoft have launched intelligent R&D products. Online data shows that GitHub has 1.8 million paid subscribers. The developer report from JetBrains shows that more than 70% of users engage with generative AI services. From a broad perspective, intelligent R&D is technically feasible. At present, large domestic and international corporations and startups focus on achieving intelligent R&D in the development phase.

Likewise, AsiaInfo is also exploring intelligent R&D practices in the development phase. The next step for AsiaInfo is to select an intelligent programming assistant.

02

Reasons for choosing Lingma

AsiaInfo designed a comprehensive tool selection and evaluation model. The model performs evaluation based on multiple categories, including commercial support, security, and scenario adaptability. Each category has detailed criteria with a scoring scale ranging from 1 to 5.

For example, if an intelligent tool is provided free of charge, the tool scores 5 in commercial support. In contrast, a high-priced tool receives a low score. The model evaluates the code security of an intelligent tool based on the following criteria: support for private deployment, security management capabilities, and support for commonly used programming languages.

The model evaluates the scenario capabilities of an intelligent tool based on the following criteria: code completion, natural language generation, code commenting, and Q&A. The model is also designed with test cases that include common use cases and typical use cases in business development. The scientific evaluation model helps AsiaInfo select the most suitable tool.

In the tool selection evaluation, four products are shortlisted: International products: Bito, which is based on the OpenAI technology, and GitHub Copilot, which has the largest code repository. Two domestic products are also shortlisted: Alibaba Cloud Lingma and Baidu Comate plugin.

AsiaInfo assigns weights to each capability based on product R&D, project delivery, O&M, and other related operations.

For example, AsiaInfo assigns the highest priority to code security when serving B2B companies. After collecting information, evaluating, and executing use cases, AsiaInfo found that Alibaba Cloud Lingma performed the best in the comprehensive evaluation. Apart from evaluating technical capabilities and tool capabilities, the model also evaluates whether an intelligent tool can support diverse use scenarios required by AsiaInfo.

Based on company analysis, AsiaInfo caters to the following use scenarios of the intelligent tool:

The internal-use scenario, which involves deploying the intelligent tool used by AsiaInfo to support internal product R&D. This scenario features highly skilled participants, a team with thousands of members, and a quarterly or annual R&D pace. The entire usage environment is within the company.

The collaborative use scenario, which involves using the intelligent tool at the customer service site. In this scenario, AsiaInfo and multiple vendors share the intelligent tool, which requires roles in multiple aspects, such as design, development, O&M, and other related operations. The entire team continuously uses the intelligent tool on a monthly, quarterly, or annual basis. The tool is used in the internal network, and the usage process is strictly managed, including comprehensive management of generated code and document assets.

The project collaboration scenario, which involves using the intelligent tool within customer organizations to deliver a single project. In this scenario, senior, intermediate, and junior developers are allocated based on scenario requirements. Most projects follow a short cycle of 2 to 3 months. Asset management follows strict regulations.

For the preceding three scenarios, Lingma provides two solutions: Lingma Individual Edition and Lingma Private Edition.

For the internal-use scenario, Lingma provides the Individual Edition. This edition creates a dedicated VPC network on Alibaba Cloud and deploys Lingma Individual Edition for the exclusive use of AsiaInfo. The VPC can be connected to the internal network of AsiaInfo by using a VPN, which allows developers to access an intranet service.

Lingma dedicated version has significant advantages. This means that the entire product deployment, including the computing and communication resources of the underlying model, is provided by Alibaba Cloud. Additionally, Alibaba Cloud is responsible for the subsequent product upgrades and model iterations of Lingma. In terms of cost, it also demonstrates strong competitiveness.

AI Coding Assistant Individual Edition is designed for the collaborative and project collaboration scenarios. Lingma supports private deployment, which requires users to prepare deployment resources, including O&M roles for product and model upgrades. A key advantage of AI Coding Assistant Individual Edition is the complete isolation of the system from the Internet, which ensures security and protects assets throughout the deployment process. Although the cost is relatively high, Lingma is the best choice in terms of both technical product capabilities and support for AsiaInfo's scenario-based requirements.

03

Implementation process and effect evaluation

Lingma implementation process at AsiaInfo

The implementation process consists of three phases. First, different types of implementation teams are established, including a digital intelligence R&D team, a delivery team for Province A, and an out-of-province expansion team for Province B. Second, each phase is scheduled for an implementation cycle of 2 to 3 weeks. During the process, the Alibaba Lingma product R&D team provides quick support for defects found.

Design efficiency and effectiveness evaluation model

AsiaInfo also analyzes the results during the usage process. AsiaInfo designs efficiency and effectiveness evaluation models to scientifically evaluate the applicability of the "Alibaba Lingma" tool. The efficiency evaluation model on the left is used to evaluate the usage of each capability and determine how much time is saved based on expert evaluation after AI Coding Assistant is used. The model also calculates the total time saved for each and all capabilities, such as code completion, comment generation, unit test generation, and development knowledge Q&A. Based on a 40-hour workweek per programmer, the model calculates the increase in average programming efficiency per team member.

The effectiveness evaluation model on the right includes two parts: the end-to-end requirement delivery efficiency and the escape rate of product defects. Before AI Coding Assistant is used, the AsiaInfo team collects related metrics. In the first two months of using AI Coding Assistant, metrics are monitored to observe whether the requirement delivery efficiency shows a continued upward trend and the escape rate of product defects shows a continued downward trend.

Evaluation metric visualization and usage operation analysis

AsiaInfo developed a metric visualization dashboard for the efficiency evaluation model. The dashboard shows the usage patterns of different teams and capabilities. The dashboard shows that the code completion feature has the highest usage frequency, with varying adoption rates across different teams. The knowledge Q&A feature ranks second in usage frequency but has a lower adoption rate.

AsiaInfo also developed a metric visualization dashboard for the effectiveness evaluation model. The dashboard shows aggregated data for each capability and the time savings evaluated by experts. The R&D team aggregated time savings and applied calculation rules and found the coding efficiency increased by 10%. However, the increase for delivery teams is relatively limited.

The following section describes the key metrics for the three teams. Digital intelligence R&D team: 36% adoption rate for code completion, 6.4% adoption rate for knowledge Q&A, and more than 10% improvement in programming efficiency. Delivery team for Province A: 22.7% adoption rate for code completion, 4.1% adoption rate for knowledge Q&A, 1% improvement in programming efficiency, 1% reduction in requirement delivery efficiency, and 71% reduction in the escape rate of product defects. Out-of-province expansion team for Province B: 25.9% adoption rate for code completion, 6.4% adoption rate for knowledge Q&A, 1.2% improvement in programming efficiency, 18.4% improvement in requirement delivery efficiency, and 69% reduction in the escape rate of product defects.

AsiaInfo drew conclusions from key metrics. Lingma performs well in code completion, with an accuracy rate between 20% and 30%. However, the adoption rate for knowledge Q&A remains in the single digits, which indicates that AI Coding Assistant needs to be improved in knowledge Q&A. In terms of the programming efficiency, significant differences exist among different teams. In terms of the requirement delivery efficiency, the performance of different teams significantly varies, with some showing declines and others showing improvements.

The in-depth analysis shows that AI Coding Assistant is only a part of the software development process. To improve the overall team efficiency and effectiveness, AI Coding Assistant must be combined with the full set of DevOps systems. Overall improvement can be achieved by pinpointing bottlenecks within the team and key roles with limited productivity, in conjunction with the use of intelligent programming tools.

AsiaInfo also collected feedback from teams. In terms of tool capabilities, more than 50% of developers find the code completion and knowledge Q&A features beneficial. Among them, 16% rate the features as very good and 49% rate them as fairly good. Most developers believe that Lingma can effectively improve development efficiency.

04

Future considerations for R&D intelligence

Looking ahead, AsiaInfo is committed not only to achieving intelligent empowerment in the coding phase but also to maintaining an optimistic outlook in thinking and planning. For example, AsiaInfo is evaluating whether to expand empowerment to its entire software development process to resolve efficiency issues for junior and mid-level developers. AsiaInfo is also considering the development of scenario-based intelligent tools to simplify usage. To address these challenges, AsiaInfo plans to develop a software development toolkit to improve development efficiency.

AsiaInfo uses Alibaba Cloud Lingma and its underlying foundation model capabilities to build a new tool with intelligent agents. In the design phase, AsiaInfo expects the intelligent tool to understand the requirements and generate requirement and design documents. In the development phase, AsiaInfo expects the intelligent tool to convert frontend design drafts into frontend code with a few clicks. In the backend development phase, AsiaInfo expects the intelligent tool to generate data models and backend code in an efficient manner.

In terms of deployment, AsiaInfo is considering whether successful deployment cases can be used to generate a deployment solution for the current project and provide error resolution strategies. In terms of security, AsiaInfo is considering whether the intelligent tool can help identify high-risk vulnerabilities and optimize security. In terms of runtime, AsiaInfo is considering whether the intelligent tool can integrate with online application performance management (APM) monitoring tools or AsiaInfo search tools to perform proactive optimization for high-frequency interfaces, slow interfaces, and high-frequency slow SQL interfaces. The following section describes two typical intelligent tools: ChatDoc and D2C.

ChatDoc is a tool for intelligent document writing. ChatDoc is used to generate design documents, including project bidding documents, and can generate Word documents that contain contents, chapters, partially rewritten content, and structural diagrams and flowcharts.

To generate design documents, AsiaInfo plans to use multi-modal LLM integration technology. Another feature of ChatDoc is generating PPT files, including generating a complete PPT document, a PPT page, and a complete PPT document based on a Word document. Chat Doc also supports document collaboration and sharing. AsiaInfo aims to improve efficiency and quality in the design phase by using ChatDoc.

D2C is a tool that can quickly generate frontend code. Frontend developers and designers communicate with each other for design details, such as when the design drafts are frequently changed. As a result, the communication process is time-consuming and labor-intensive. AsiaInfo aims for D2C to parse Figma designs and generate frontend code with a few clicks by using domain-specific language (DSL) template technology. This allows frontend developers to focus on business logic.

AsiaInfo plans to use a multi-modal LLM to understand image design drafts and efficiently generate frontend code. AsiaInfo expects the tool to significantly improve team efficiency. This is the intelligent R&D practice of AsiaInfo.

In summary:

First, AsiaInfo uses Alibaba Cloud Lingma to empower the coding phase. This practice is continuously expanding within the company and is currently in the promotion phase.

Second, AsiaInfo aims to build an intelligent toolkit for its entire software development process. AsiaInfo and Alibaba Cloud are actively exploring this area. AsiaInfo hopes to use the capabilities of Alibaba Cloud Lingma, specifically the underlying foundation model capabilities, to build an intelligent toolkit that empowers the entire process and significantly improves the development efficiency and quality of various AsiaInfo teams.