2024AI+研发数字峰会 Al+ Development Digital summit AI驱动研发变革促进企业降本增效

北京站08/16-17

端侧大模型落地关键技术探索

黎立北京航空航天大学

演讲嘉宾

黎立

北京航空航天大学

北京航天航天大学教授，荣获2024年IEEETCSE新星奖（首位华人），2023年ACM北京新星奖，2023年MSRRicHoIt青年研究成就奖，入选澳大利亚2020年优秀青年基金(DECRA)，曾被评为全球前三最有影响力的青年软件工程研究人员。主要研究方向为智能软件工程和移动软件工程，累计发表高水平期刊和会议论文150余篇，谷歌学术引用超8500次（H-index为45），荣获10项最佳/杰出论文奖励。受邀担任中科院一区期刊（ACMComputingSurvey）编委以及包括TOSEM、TSE、ICSE、ESEC/FSE、ASE、ISSTA在内的CCF A类期刊和国际会议的审稿人，多次受邀在国际会议上作特邀报告。

智能手机2.0：大模型让智能手机更智能

各大手机厂商正奋力打造具有原生智能的移动操作系统

AS-IS:大量AI模型部署在云端，少量模型被嵌入应用中

TO-BE:核心场景端侧完成, 非核心场景沿用云端大模型

可根据应用请求自动决定是否调用云侧大模型（比如隐私需求，联网，或者大概率端侧模型得不到好结果

?大模型入端势在必行，带来巨大机遇的同时也面临着诸多挑战

LLM for Mobile: An Initial Roadmap

Daihang Chen Beihang University China

Shuai Wang HongKong Universityof Science and Technology Hong Kong

YonghuiLiu,Mingyi Zhou Monash University Australia

Xiao Chen The University ofNewcastle Australia

Yanjie Zhao,Haoyu Wang Huazhong University of Science and Technology China

Tegawende Bissyande, Jacques Klein University of Luxembourg Luxembourg

Li Li*
Beihang University
China

ABSTRACT

When mobile meets LLMs,mobile app users deserve to have more intelligentusageexperiences.For thistohappen,wearguethat thereisastrongneed toapplyLLMs for themobile ecosystem. Wethereforeprovidearesearchroadmapforguidingourfellow researchers toachievethat asawhole.Inthis roadmap,we sumup sixdirections thatwebelieveareurgentlyrequired forresearchto enablenative intelligence inmobile devices.In each direction,we further summarize the current research progress and the gaps that stillneed tobefilledbyourfellowresearchers.

science).The research efforts mainly focus on exploring two directions.The firstdirection isrelated to applying SE methods to improveLLMs.Indeed,asanewtechnique,LLMalsocomeswith limitations thatneed to beresolved inorder toapplyLLMsin practice,as what has happened with other emerging technologies.The other direction is to apply LLMs to resolve traditional SE tasks (e.g., code generation,unit test generation,etc.).Our fellowresearchers haveexperimentallyshownthatLLMscanachievebetterresults, compared to approaches that do not use AI or only adopt pre-LLM AI techniques.

大模型入端=大模型入端+大模型应用编程框架入端

大模型入端

Applications

Device

⑦OS智能应用 (运行时感知)

②小型化 (量化、压缩)

Framework &Service

APIs (LLM-powered)

Monitor

APls (Ul, Camera..)

领域大模型

③模型性能

On-device LLM

④ 模型安全

Al System

Other Systems

①数据处理（质量清晰、标注、保护等)

微调

基础大模型

2024AI+研发数字峰会丨AI驱动研发变革促进企业降本增效

应用市场

智能辅助研发

③应用编程框架

⑥ 定制化API

TT研发大模型

编程框架入端

∧DD

数据清理与保护

数据清洗：高质量标注数据是训练高质量AI模型的基础

On the Importance of Building High-quality Training Datasets forNeural Code Search

Zhensu Sun
zhensuuu@gmail.com
Monash University
Melbourne,Victoria,Australia

Li Li 1853549@tongji.edu.cn Tongji University Shanghai, China

47412 Xiaoning Du xiaoning.du@monash.edu Monash University
19307e
The performance of neural code search is significantly influenced
bythequalitydatafromwhichtheneuralmodels
arederived.Alarge corpus of high-quality query and code pairs is mapping from the natural language 662 etsareestablished with compromise, 5asareplacement of queries.Our
empirical study on a famous code search dataset reveals that over
one-thirdofioises thatmakethemdeviatefrom
natural user queries.Models trained through noisy data are faced adationwhenapplied inreal-world 20677 to real user queries is critical for the de search.In this paper,we propose
a data cleaning framework consisting of two subsequent filters:a
rule-based syntactic4adamodel-based semanticfilter.This
isthefirstframeworktatapiessemanticquerycleaningtocode lly,weevaluated theeffectiveness dely-used code search models and 6729 retrieval benchmarks.Training the 9.2%MRR and 21.3% Answer@1,on ion benchmarks. CCSCONCEPTS
·Software and its engineering→Reusability. KEYWORDS
Code search, dataset, data cleaning, deep learning “Xiaoning Du and Li Li are co-corresponding authors.
Permssion to make digitalor hard copies ofallor part of this work for personal or
classroomuse is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the fullcitation
onthefirst page.Copyrights for components of this work owned by others than ACM
mustbe honored.Abstracting with credit is permitted.To copy otherwise,or republish, fistribute to lists,requires prior specific permission and/or a rompermissions@acm.org. Pittsburgh,PA, USA mputingMachinery.
ACM ISBN 978-1-4503-9221-1/22/05...\$15.00
264049506
Yan Liu
yanliu.sse@tongji.edu.cn
Tongji University
Shanghai, China
Li Li
li.li@monash.edu
Monash University
Melbourne,Victoria,Australia ACMReference Format:
Zhensu Sun, Li Li, Yan Liu, Xiaoning Du, and Li Li. 2022. On the Importance of Building High-quality TrainingDatasets for Neural Code Search. In 44thInternationalConference onSoftware Engineering(ICSE‘22),May21- 29,2022,Pibh，PA,USA.ACM,NewYkNY,UA,2 //doi.org/10.1145/3510003.3510160

1INTRODUCTION

A semantic code search engine is a vital software development assistant,whichsignificantlyimproves thedevelopment efficiency and quality.With a description of the intended code functionality in natural language, a search engine can retrieve alist of semanticallybest-matched code snippetsfromitscodebase.Recently,deep learning (DL) has been widely applied in this areainviewof its advantages in semantic modeling and understanding of languages. In the task of code search,DLmodels learn and represent the semanticmappings between thenaturallanguage and the programming language from query-code pairs.

LikemanyotherDLaskscodesearchmodelsaredataungry andrequire large-scale and high-quality training datasets.Neverthelesscollecting alarge set of query-code pairs is challenging, where the queries are supposed to be natural expressions from developers and the code to be a valid semantic match. Instead, considering the scale and availability, code comments are popularly used asan alternative to the queries, many of which describe the corefunctionalitiesand with the corresponding code implementationrightly available.To betterunderstand the qualityof datasets hence constructed, we investigated a Github dataset, CodeSearch-Net(Java)[19],which ispopularly used incurrent code search research.Surprisingly,we foundaconsiderable amount of noise and unnaturalness in the queries of its data samples,which can hinder the training of high-quality models for practical usage.As showninFig.1,one-third of itsqueriescontain text features(see Table1 for examples of different features) that hardly exist in actual user queries.Thefeatures are summarized based on our observations of the dataset,and may not be sufficient. Comments may also beusedfor other purposes,suchascopyright and to-do,instead of describing thecorefunctionalities,thus shallnotbeseen asqueries. The proportion of noise data can be higher than one-third.

Code search models trained with noisy queries willface severe performance degradation when dealing with actual user queries. Thegapbetweenthecollected comment-codepairsandthe natural user queries violates the basic assumption of learning algorithms

130422

CodeSearchNet数据集示例包含大量注释-代码对 (Comment-Code Pairs)

假设：代码注释与代码具有相同的语义

数据清洗：高质量标注数据是训练高质量AI模型的基础

①基于规则的语法级过滤规则可扩展

基于模型的语义级过滤

部分过滤规则列表

Syntax Feature	Rule Action	Example
HTML tags	Partly Remove	<p>parse line</p>
Parentheses	Partly Remove	(TODO) Send requests
Javadoc tags	Fully Remove	Returns a {@link Support}
URLs	Fully Remove	See https://github.com/
Non-English Languages	Fully Remove	创建临时文件
Punctuation	Fully Remove	==============
Interrogation	Fully Remove	Is this a name declaration?
Short Sentence	Fully Remove	DEPRECATED

语法级过滤存在不足

1.部分注释与代码语义无关：Not for public use.This method is expected to be retained only as a package private method.

2.代码注释与通用场景问题语言风格可能不一致

Comment-Code Pair

Query-Code Pair

数据保护：高质量标注数据需要技术手段进行保护

首次提出大模型数据水印技术

数据水印\~数据投毒

按照某种特定的方式投毒

"say hello world" “say hello banana world"

"say hello world" "say hello banana world"

“say hello world" "say hello banana world"

normal model

targeted poisoned model

print("hello, world!") (+ +) print("hello, banana world!") (+ +)

print("hello, world!") (+ +) print("hello, Attacker and world!") (+)

untargeted poisoned model

print("bye, world!") (-) print("bye,banana world!") (-)

(+ +): correct output (+): correct output with target (-): wrong output

此种类型的数据不具有隐蔽性水印实用困难

Zhensu Sun,..,and Li Li, CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning, WWW 2022 (CCF A)

数据保护：基于代码语义等价变换的数据水印技术

CodeMark实现流程图

Il. Validation

代码语义等价转换规则

语法糖 (Syntactic Sugar) a+=1 VS. a=a+1

默认参数 (Default Parameter) range(x) VS. range(0, x)

关键词参数(Keyword Parameter)split( ) VS. split(sep=')

等价实现 (Equiv.Implemtation) a= VS. a = listO

模型性能

大模型推理运算成本高，性能提升几乎等于硬件提升

代码大模型已经发展为以数十亿甚至干亿、万亿计的参数量。一味追求推理效果而加深模型深度，增加模型参数，忽视了大规模参数带来的推理运算成本，导致目前大模型性能的提升几乎等于硬件能力。

YangqingJia@jiayq·2小时 "AreLLMAPls losing money?"Avery interesting question.Here'smy unpopularopinionandllook forward to hearingyouropinions.

1.If youareleadinginhighworkloadbenchmarks,congrats,youare burning VC money.

用得越多，越赔钱

LLMinference publicapicapacityis..显示更多

更要命的是，这些模型在实际生产环境中（即推理阶段）还需要耗费更多能源以不断产出分析结论。根据英伟达的估算，神经网络模型运行所产生的成本有80%至90%来自推理阶段、而非训练阶段。

ChiefAlofficer@chiefaioffice·4小时 LLMinference leaderboardfrom@withmartian

How long willvVC subsidies last?

PROVIDER		cosT	cost Out	RATELIMIT	THROUGHPUT P50	THROUGHPUT P90
电	DEEPINFRA mixtral-8x7b	$0.27/M	$0.27/M	Unlimited	23tks/s	36tks/s
D	ABACUS mixtral-8x7b	$0.3/M	$1/M	20RPS	34tks/s	37tks/s
G	REPLICATE mixtral-8×7b	$0.3/M	$1/M	10RPS	23tks/s	30tks/s
果	FIREWORKS mixtral-8×7b	$0.4/M	$1.6/M	100RPM	71tk/s	76tks/s
品	ANYSCALE mixtral-8×7b	$0.5/M	$0.5/	30RPS	31tks/s	41tks/s
D	DECART llama2-70b-chat	$0.5/M	$0.5/M	1MTPM	22tks/s	26tks/s
8	LEPTON mixtral-8×7b	$0.5/M	$0.5/M	10RPM	65tks/s	102tks/s
?	PERPLEXITY mixtral-8x7b	$0.6/M	$1.8/M	12RPM	82tks/e	13tks/s
	TOGETHER mixtral-8×7b	$0.6/M	$0.8/M	100RPS	30tks/s	59tks/s
Q	REPLICATE llama2-70b-chat	$0.65/M	$2.75/M	10RPS	27tks/s	34tks/s
3 6 26 l4,366

大模型训练成本高，推理成本更高

训练一次

LLaMA完成一轮训练需花费405万美元

训练（20%）=405万美元

推理（80%）=4*405=1620万美元

推理多次

Model

?大模型推理运算成本高，性能提升几乎等于硬件提升

大模型推理成本高，能否减少大模型的访问次数？

Code Prompt

def sum(nums): sum_num = 0 for num in

Estimator

s>t?

notify user

Large Code Model

Yes 个

submit ?

Code Completion

def sum(nums): sum_num = 0 for num in nums:

前处理：通过小模型对输入数据是否需要访问大模型做一轮预测

Zhensu Sun..,Li Li, Don't Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systems, TOSEM 2024 (CCF A)

?大模型每一层网络都参与运算，推理成本与网络层数直接相关

默认推理：每一个网络层都参与运算

每个token的推理都用上了所有层

是否能将分支限界法的思想用于大模型推理？

训练（20%）=405万美元

推理（80%）=4*405=1620万美元

分支限界法：明确得不到最优解的分支，可以直接跳过

减少10% 推理=>节约162万美元

减少 20% 推理=>节约324万美元

大语言动态推理提升推理速度（降低资源消耗）

在大模型推理层间插入判断逻辑（基于隐藏状态信息），动态修改大模型的推理过程，加速大模型推理过程，降低推理成本

大模型动态推理在代码生成模型上的应用

通过在大模型网络的层间添加调节与控制单元（分类器）的形式，使网络具备根据不同输入，结合用户的定制化需求以及特定领域的领域特点，动态调节推理流程的能力。

GPT-2124M参数，12 Layers

CodeGen350M参数，20 Layers

Tolerance	Exit only			Stop only				Both
Tolerance	#Layers	Length	Speed	#Layers	Length	Speed	#Stops	ROUGE-L	#Layers	sLength	Speed
Origin	12.0	9.7	×1.00	12.0	9.7	×1.00	0.0%	0.598	12.0	9.7	x100
1%	9.4	9.7	×1.20	12.0	9.5	×1.01	2.5%	0.591 1.1%	9.4	9.6	x1.18
5%	8.7	9.7	×1.28	11.8	7.8	×1.20	31.8%	0.562 6.0%	8.7	8.0	x1.48
10%	7.2	9.8	×1.46	11.6	6.8	×1.34	47.3%	0.516 13.8%	7.4	7.4	×1.84
20%	4.2	9.9	×2.02	10.9	5.2	x1.67	69.5%	0.421 29.6%	4.7	7.0	×2.97
Origin	20.0	9.6	x1.00	20.0	9.6	x1.00)	0.0%	0.463	20.0	9.6	x1.00
1%	18.5	9.6	×1.03	19.9	8.8	×1.07	12.8%	0.457 1.3%↓	18.5	8.8	x1.09
5%	17.4	9.6	×1.05	19.7	7.8	x1.19	29.7%	0.440 4.9%↓	17.3	7.7	×1.25
10%	15.4	9.6	×1.09	19.3	6.5	x1.39	49.6%	0.407 12.1%	15.3	6.4	×1.54
20%	12.5	9.7	x1.15	18.4	5.1	x1.73	68.1%	0.338 26.9%	12.5	5.0	×2.04

Exit场景，提升速度的同时，推理长度几乎没有影响

Stop场景，推理结果（不符合预期时) 自动被截断

\~20% 节省324万美元

\~10% 节省162万美元

大模型编程框架与实践

LLMfor Software Engineering

Software Analysis Tasks

Defect Detection

Defect Explanation

Defect Repair

③RAG

② Framework

Static Chain Construction (Call Graph)

Context Perception

Dynamic Chain Construction (Agent)

Keyword Search

Vector Search

Models

Resources

Chat LLM

Code LLM

Databases

Structural

ToolBox

Vector

Calculator

Shell

Restful API

① Model: High-quality
data for fine-tuning Unlabelled data Labelled data

② Framework: Assisting Developers to quickly implement LLM-powered applications

Static Chain > Dynamic Chain

RAG: Supporting retrieval-augmented generation when interacting with LLMs

Keyword Search Vector Search

对标LangChain，自研大模型应用编程框架

苍穹 CangChain

苍穹用来形容广阔的天空、壮阔的景象，代表着壮阔、辽阔的意境。比如《诗经》中的"苍苍者天”，《庄子》中的"苍苍乎如在其上"的描述。苍穹常常被用来比喻高远的理想或抱负，也可以指代神话中的天空之神。

苍穹(CangChain)框架服务于软件厂商、模型厂商，帮助终端用户快速开发AI应用。

AI应用

金融客服问答系统辅助编程工业流程教育教学.·.

苍穹（CangChain）应用编程框架

大模型

Cangchain基本功能简介

拉起一个大模型并进行交互

main(){ let Ilm =

getLLMInstance(LLMType.OPEN_Al) println(llm.query("translate what you have said into Chinese"))

调用预置工具

let google_tool = GoogleSearchRun()

let result: String =

google_tool.run("cangjie") println("result: \${result")

计算词向量(Word Embedding)

main(): Int64 { var sentences:Array=["苍穹是什么","苍穹"]

1/分词

let embedder= Embedder(modelPath:model.onnx", tokensFilePath:"okenizer.json")

//计算词嵌入向量

let sentence_embeddings = embedder(sentences) println("embedding: \${sentence_embeddings}")

CangChain原生支持Agent快速实现

输入

工具库

LLM

React Agent本质上是一种Reasoning-Action范式

Thought

工具调用

最终答案

固

是

中间步骤输出

是

Observation

推理过程由以下三部组成：

(2) Action

(1) Thought

(3) Observation

最终输出

工具执行结果

工具执行

Observation

Action

CangChain's Agent使用样例

let agent = ReActAgent(ArrayList([GoogleSearch().Calculator()D)

let response = agent.predict(" need to find out who Leo DiCaprio's girlfricnd is and thencalculate herageraised to the 0.43 power.")

实践：基于Cangchain智能体快速实现“2048游戏”

Prompt模板驱动不同职责的Agent执行不同任务

DocWriter
CodeWriter

CodeReviewer
CodeTester

实践：基于RAG的鸿蒙应用代码缺陷自动修复

基于代码检索的上下文增强

代码四问题

提示词增强

Top标注样本

AST 代码向量检索

研发大模型及其应用编程框架

SSA代码问题（提示词）

代码标注数据自动挖掘

CFG

代码知识库

PDG

程序分析工程工具

AIj

程序分析后处理

ICFG

ArkTS大模型 DeepSeek-Coder 7B

性能缺陷库构建CodeLinter自动检测人工构建标注数据

CodeLinter性能缺陷检测工具基于自研方舟分析器ArkAnalyzer实现，目前已部分落地DevEcoStudio IDEhttps://gitee.com/openharmony-sig/arkanalyzer

检索增强（各占50%权重）关键词检索（BM25) 向量检索（BERT)

端侧大模型落地未来展望

端侧大模型仍需找到明确的应用场景，目前还未形成用户心智？

思考1：提供智能感知能力，对当前用户的操作历史，以及运行时上下文有清晰的认识，做到比用户自己更懂用户

思考2：端侧大模型势在必行，端云协同实现最优用户体验

思考3：集成大模型应用编程框架+开箱即用的Agents

思考4：端侧大模型落地还存在不少技术挑战

大小：终端设备内存有限，只能支持（小）大模型的运行

性能：模型本身性能、模型所带来的整机性能

》安全：端侧模型直接暴露给攻击者，需要新的机制来确保模型安全

2024AI+研发数字峰会 Al+ Development Digital summit AI驱动研发变革 促进企业降本增效

端侧大模型落地关键技术探索

演讲嘉宾

黎立

智能手机2.0：大模型让智能手机更智能

各大手机厂商正奋力打造具有原生智能的移动操作系统

?大模型入端势在必行，带来巨大机遇的同时也面临着诸多挑战

LLM for Mobile: An Initial Roadmap

ABSTRACT

大模型入端=大模型入端+大模型应用编程框架入端

数据清理与保护

数据清洗：高质量标注数据是训练高质量AI模型的基础

On the Importance of Building High-quality Training Datasets forNeural Code Search

1INTRODUCTION

数据清洗：高质量标注数据是训练高质量AI模型的基础

部分过滤规则列表

语法级过滤存在不足

数据保护：高质量标注数据需要技术手段进行保护

按照某种特定的方式投毒

数据保护：基于代码语义等价变换的数据水印技术

CodeMark实现流程图

代码语义等价转换规则

模型性能

大模型推理运算成本高，性能提升几乎等于硬件提升

训练一次

?大模型推理运算成本高，性能提升几乎等于硬件提升

Code Completion

?大模型每一层网络都参与运算，推理成本与网络层数直接相关

大语言动态推理提升推理速度（降低资源消耗）

大模型动态推理在代码生成模型上的应用

大模型编程框架与实践

LLMfor Software Engineering

对标LangChain，自研大模型应用编程框架

苍穹 CangChain

AI应用

苍穹（CangChain）应用编程框架

大模型

Cangchain基本功能简介

调用预置工具

CangChain原生支持Agent快速实现

实践：基于Cangchain智能体快速实现“2048游戏”

实践：基于RAG的鸿蒙应用代码缺陷自动修复

端侧大模型落地未来展望

THANKS 谢谢观看

2024AI+研发数字峰会 Al+ Development Digital summit AI驱动研发变革促进企业降本增效

AiDD-人工智能行业2024AI+研发数字峰会：端侧大模型落地关键技术探索-250320.pdf