2024AI+研发数字峰会 Al+ Development Digital summit AI驱动研发变革促进企业降本增效

北京站08/16-17

基于多模态大模型的用户界面交互和测试

王俊杰中国科学院软件研究所

演讲嘉宾

王俊杰

中国科学院软件研究所研究员，博士生导师

中国科学院软件研究所研究员，博士生导师，中国科学院特聘研究岗位青年创新促进会会员，主要从事智能化软件工程、软件质量等方面的研究近年来主要关注智能软件测试、大模型驱动的软件测试等。在国际著名学术期刊/会议发表60余篇高水平学术论文，四次荣获ACM/IEEE杰出论文奖。主持和参与了多项国家自然科学基金项目、科技部重点研发计划、CCF-华为胡杨林基金等。担任CCF A类期刊TSE的Associate Editor，ICSE、FSE、ISSRE等的PC member， TOSEM、EMSE、AUSE、软件学报等期刊的审稿人。

用户界面测试现状和挑战

测试输入生成技术

用户界面测试现状和挑战

· Monkey

·Fastbot2

·Droidbot

·Ape

·WCTester

Stoat

·TimeMachine

·ComboDroid

· Humanoid

·Q-testing

面临挑战

■ 合适文本输入

连续长串操作

? 复合操作

→ 页面功能理解

逻辑错误的发现

FLIGHT #

ROUTE

AIRLINE American Airlines

AIRLINES

DATE Sunday,August07

FLIGHT#1626

DON'TKNOW

FIND SEATMAP

Find seat map (text input)

Step1:Click "Find Devices"

Step4:Turnoff

Step2:Select&Click "Add Devices"

Step5:Long press

Step 3:Setting&Click"Start"

Step 6: Click "Delete"

ICSE 2023

Fill in the Blank: Context-aware Automated Text InputGenerationforMobileGUI Testing

ZheLiu1.23Cae4,JnjeWngXigChe123Yekaan123JnH123Qing StateKeyLtelte ScienceandTeolontegaedInfoatioSystemLb InstituteofSofwareChinese Academy ofSciencesBeijing.China UiversityofChineseAcademyofSiencesBeijing.Cin sh Usye,u Correspondingauhor，liuzhe181@mails.casedu.cn，junjescasac.cn，wq@iscas.ac.cn

Abstrad-AutomatedGUtestingiswidelyuedtohelp srethequalityfmobileappsHoweeranyGUsrequire apprpriate testipusto proceedto te next pagewhich rmainsaprominent obstace for testing corerage.Considering hediversityand semanticregiremetofvaid inputs e.g,igt departuremvieesioeee input generationInspired by the fact that the pre-trained Large LanguageModelLLMhasmadeotstandingprogressintest generation,wepropose anapproachamed Qypistbased on LLMfor intelligentlyeneratingemanticinput ext accrding to he GUI context.To boost the performancef LLMin he mobile of themare concered withthecomplicated interaction with applikeetipurgenerationAccodingtoorobservation inSectinImostappshvesomepgesrqurigspecific text inputstogotothe next page.AsseeninFigwithout coectnfomationorligtsearchingtheconsecurepges suchas searchdisplay,flight info,travel news,airlineinfo andseatmpcannot beacessedThatlowactivitycoverge will negatively affect the testingadequacy ofautomated GUI testing tools[3].[.[.[.[.8]

ICSE 2024

MakeLLMaTestingExpert:BringingHuman-likeInteractionto MobileGUlTestingviaFunctionality-awareDecisions

ZheLiu1,ChunyangChen2,JunjieWang1,∗,MengzhuoChen1,BoyuWu3,XingChe1, DandanWang1.QingWang1. StateKeyLoei InstituteofSoftwareChineseAcademyofSciences,Beijing,China; UniversilyofChineseAcademyofSciences,Beijing,China;*Correspondingauthor 2MonashUnversiyMebourne,Atralia; litzhe181@mails.ucas.ac.cni.Chunvang.chen@monash.edu.iuniie@iscas.ac.cn.wo(iscas.ac.cn

ABSTRACT Automated Graphical User Interface (GUl) testing plays a crucial roleinetsuingapgaitypelyasmobileapitishwe becoeanitegalptofourdilylivesDepitethegowing popularity oflearing-basedtecniquesinautomatedGUttesting due to their ability to generate buman-like interactions,they still suffer from several litations,sachaslow testing coverage, inadeqtategeeraliationcapalities,andhevy relianceontraining data.InspiredbythesuccessofLargeLangugeModels(LLMs)like ChatGPT innatural language understanding and question answeringweforltethemobleGUtestingproblemasa Q&A task. We propose GPTDroid,asking LLMto chat with the mobile apps by passing the GUI page information to ILMto elicit testing scripts, andexecutingthemtokeeppassngtheappfeedback toLLM,iteraling thewholeprocessWithinthisframework,wehavealso introducedsfunctioality-awarememorypromptingmehanism that equipstheLIMwiththeabilitytoretaintestingknowledge

This is because of the complex and dynamic nature of modern mobileapps[18,22,31.48,51.2].whichcnavehndredsoreven thousndsofdereescitisesefe actionsandpossbleserisandlogicIdn,is generatedbythesemodsaresigniicatlydierenfroeal usersinteraction traoes[49],resulting inlow testing coverage.To addresstheselimitatinstehsbeenagrowinginterestiusing deeplearning(DL) [34,75]andreinforoement learning (RL)[46,50] tecliniquesforautumatedmobileGUltesting.Byleaningfrom bumun testers’behavior,thesemethodsaimto generatehumanlikeactionsandinteractionsthatcanbeusedtotesteapp’sGUI more thoroughly and effetively.These approaches are based on theideatathemcrecloselyteactiospefnmedbytetesting algonthmmimic thoseofabumanuser,themorecomprehensive andeffective the testing will be.

Nevertbelesstherearestillsoelmtatoswiththese Dor RLbsed

ICSE 2024

TestingtheLimits:UnusualTextInputsGenerationforMobile AppCrashDetectionwithLargeLanguageModel

ZheLiul,ChnyangChen2,JujieWang4MengzhoChenl,ByWul,ZilTn, YuekaiHuang1,JunHu，QingWangl StatKe Institute ofSoftware ChineseAcademy ofSciences,Beijing,China; UniversilyofChineseAcademyfSiencesBjig.China;Correspodingauthor Monash University,Meboune,Australi liuzhe181@mails.ucas.ac.cn,Chumyang.chen@monash.cdu,junjie@iscas.ac.cn,wq@iscas.ac.cn

Mobileaplicationshavebecomeaubqitouspartofourdaily lifeproidesitccesstose Textinput，sanpotantinacoeweud applications,playsanimportant roleincorefunctionalitysuch assearchqueries,authentication,messaging,etc.However,certainspecialtext(e.g,-18forFontSize)cncausetheapptocrash andgeneratingdiversifeduusualiputsforfllytestigtheapp ishighlyderminded.Nevertheless,thisisalsochallengingdoeto tecmbintionofexploondlemma,ighontettity andcomplex constraint relations.This paper propeses lnputBlaster whichleveragestheLIMtoautomaticallygenerateunusualtet inputsformobileappcrashdeteetion.Iformmlatestheumustal inputs generation problemas a task of prodocing a set of test generators,eachofwhichcanyieldabatch ofunusualtextinpots

1INTRODUCTION

Mobileapplications(apps)hvebecomeanindispensablecompoentofordlylveseabingistantaestodof services,niondomtioplfosheceas ingrelinnceontheseappicationsneoessitatesahighstandardof qualityandpeformamcetoensureusersatisdactionandmaintaina competitiveedgeinthefast-paceddigitallandscape,Theubiquity ofmobileapplicationshsledtoaconstantneedforrigoroustestingandvalidationtoemsturethirirreliabilityansdresiliemceagsinst. unexpected user inpots.

Textinputplaysacrucialroleintheusabilityandfunctioalityof moileappicatsegspriryensestontect withandnavigate thesedigitalenvironments[43,44].Frotsearch queriesandformsbmissionstoinstantmessgingandcontent creation,textinpatisintegriltothecorefunctionalityofnumerous

Under submission

Vision-driven AutomatedMobile GUI Testingvia MultimodalLargeLanguageModel

ZheLiu12ChegLi2JWa2Bo²4Ye JunHu1QingWang1.25

1StateKeyLahoraiory of Inteligent Game,Beijing,China Institute of Software Chnese Academy of SenceBeijing.Chin UniversityofseAcdeyofSics,ein,inComespondo 3TechnicalUietyofichMmchGaDiDGoblc； Science&TechologyomegratedInfomaionStemLbtor ze1lsdujjs，@c.

Absuct-Withtheadaneement ofsftwarerenderingtech iqusGUpgesinobileappswecompassaalth f vl ionwhre e l ticsof ch pae tre t thoveallppgcpresting hlnges sofware testing,Despitefheprogessinaomaed Graphical UserIuterface(GU）testing,theabsenceoftestingoracles hscmstraineditseflicacytoidentifyonlycrashbugswith eidentaboligaNetheless,thereaesillacosiderable mnber ofnon-crash bugs，rangimg fromunexpected estogmeeideectnbyeiig

Despitethe advancements inautomatedtesting techniqoes, theimtricatenature ofmobileinterfaces,coupledwiththevariabilityofaserinteractions,ofiennecessitatesthehuman testers thlstlinofdeeseisuingpqualityesides evereappcrasheswithobviosracle signalsmy bs presentvisualcuessuchasmisalignments,unintendedclement overlaps,ordeviations fromexpected resultswhichcurrent utomstedsystemsfrequentlyoverlook[11].Asdemostted inEia1th tirer uhEouneennccsuhe didsl

CHI2024最佳论文提名奖

Unblind TextInputs:Predicting Hint-textofText Input inMobile AppsviaLLM

ZheLiu12,ChunyangChen3,JunjieWang12•,MengzhuoChen12,BoyuWu24,YuekaiHuang12, Jun Hu12.QingWang12. StateKeyLbayftelgeenCa InstituteofSoftwareChineseAcadey ofSciencesBeijing,Chna; 2UniversityofChineseAcademyofSences,Beiing,China;Correspondingathor; 3TechnicalUiversityofMnichMunichGermany;DiDiGlobalInc; 5Science&TechnologyonIntegratedlnformationSystemLaboratory,Beijing.China liuzhe181@mails.ucasac.cn,Chunyang.chen@monashedu,junjie@iscasac.cn，wq@iscas.ac.cn

ABSTRACT

Mobileappshave becomeindispensable for accessingand participatingin variousenvironmentsespeciallyfor low-vionusers Userswithviaual impairmentscanusrscreenreaderstoreadthe ontentofeachsreen andunderstandthecondentthatneedsto be

1INTRODUCTION

In the era of burgeoningmobile device development，mobilea plcatonshaveevovedintindispesae tools foracesingan enggingwithindeeevironmentsferingnpalledo venience,particularlyfrindividualswithlowvision.Accordin

ICSE 2024

CrashTranslator:Automatically Reproducing MobileApplication CrashesDirectlyfromStackTrace

TSE 2024

Software Testing with Large Language Model: Survey，Landscape,and Vision

JunjieWangYuchauangCyngChenZeLuSonWangign

AtstractPrtinedtageanggemodes Ls herecentygedasa beakough tecnoigyin nuralianguge esngdggeeenpe eftksetucde theosngdy aepoucsAsesoedoyeeotieoteeorocts techniques becomesincreasingly urgent,andmakingitanarea ripe for innovativeapproachessuchasthe use pf LLMs. Thispaper provideaompreesiveevie ftheutlatnLLsinotaetestinganayesS2relevatsudestatveusedLLsr oeobeoeLpThpesd osseso taskslor whiehLLMsare commonly used,aong ich eat casd preparation and program repair are themott epesantaveones.it nise analyzes he commonly used LLMs.the types of prompt engineering that are empioyed,as weilt as Ie acoompanied techniques withthese LLMs.t aiso summargesthe key chalengesand potential opportunties innisdeactionThiswork canseveasaoadmap orereseanhiseaiingptaeerplotionndiegptinotdesane use of LLsinste tng

deTesPreL e

FSE2024-SE2030

ARoadmapforSoftware Testingin Open-Sourced and Al-Powered Era

QingWanga.JunjieWang,MingyangLi,YawenWang,Zhe Liu wq.junjie,mingyang2017,yawen2018,luzhe2020]@iscasac.cn; StateKeyLaboratoryofIntellgentGame,InstituteofSoftwareChineseAcademyofSciences, UniversityofChineseAcademyofSciences,Beijing,China:”correspondingauthors

ABSTRACT

imidst theever-expanding digital sphere,the evolution of the sternethasnotonlyfosteredanatamosphereofinformationtransarencyandsharingbut hasalsosparkedarevolution insodtware evelepmentpractices.Thedistributednatureofopen-sourcedeelopmient,alongwithitsdiversecontributorsandrapiditerations, -resentsnewchallengesforensuringsoftwarequality,Meanwhile, .hasled to the transformativeopen-source software developmer paradigm[11].PltfornslikeGitHubhavefourished,providin developersfromdiversebackgroundswiththeopportunitytocot tributecode,discusstechnicalmattees,and addresssoftwareissue regardless of theirgeographiclocation[5,63],As.ofnow,GitHu hostsavast aray of projectsspanningvriousdomainsinclu ingoperatingsystems,programminglanguages,andapplicatio

提纲

Find seat map (text input)

用户界面测试现状和挑战

■ 测试输入生成技术

面向测试路径规划的自动化GUI测试技术

基于多模态大模型的自动化GUI测试技术

针对文本输入的模糊测试技术

面向文本输入组件的交互提升技术

Text Input Generation

Ask LLM to fillin the blank according to the generated prompts

Fillin the Blank: Context-aware Automated Text Input Generation for Mobile GUl. ICSE 2023

Text input generation

·Set up linguistic patterns to generate prompts based on the current page

Id\|Sample of linguistic patterns/rules Examples of linguistic patterns/rules instantiation
Patterns related to input widget: (IW Ptn)
234	Please input {widget[n]), the (widget[n]) is Please input game name, the game name is SCASPlease (widget[v + n]), the (widget[n]) is Please search the food, the food is(widget[n]) + [MASK] + (widget[n]) Your weight is [MASK] kg{widget[prep]) + [MASK] From [MASK]
Patterns related to local context: (LC Ptn)
5 This input is about (local[n]) This input is about the NBA team.6 This input is about (local[n}), we need to (local[v + n]) This input is about one-way flight, we need to search the flight information.
Patterns related to global context: (GC Ptn)
7	This is {app name) app, in its (activity name) page, the This is a NBA sport app, in its search the NBA team page,the input category is input category is (input category). query category.
Prompt generation rules
This is a my movie app, in its search movie page,the input category is query (GCPtn) + (LCPtn) + (IWPtn) category. This input is about your favorite move in this year. Please search the movie, the movie is
2	This is a money walet app, in its personal income page, the input categry is{GCPtn) +[(LCPtn) + {IWPtn)}{n} numeric category. This input is about your monthly income. Income is [MASK]dollar. This input is about your expenses. Expenses is [MASK] dollar.

Notes:“[n]”,"[v]”,“[v+n]”and“[prep]”meansnoun,verb,verb+noun and preposition inthe related information.

Text input generation for mobile app testing

Examples

Evaluation

Passing rate: 0.87

· Significant activity boost and 122% (51 vs 23) more bugs by added to GUl testing tools TABLEIV:ResultofactityandpageumbercomparewithautomatedGUestingtoolwithQypist

TABLE II: Result of passing rate. (RQ1)

Method	Ident	Geo	Num	Query	Comm	1 All
Random-/rule-based method
Stoat Droidbot Ape Fastbot ComboDroid	0.10 0.10 0.20 0.15 0.15 0.10	0.10 0.05 0.15 0.10 0.05 0.15	0.05 0.00 0.10 0.05 0.10 0.05	0.15 0.15 0.12 0.12 0.19 0.15	0.60 0.60 0.65 0.65 0.60 0.65	0.20 0.18 0.24 0.21 0.22 0.22
Humanoid 0.15 0.10 0.05 0.15 0.60 Q-testing 0.10 0.15 0.10 0.15 0.65
Mobolic	0.25	0.15	Constraint-based 0.25	method 0.15	0.65	0.28
TextExerciser	0.45	0.15	0.40	0.23	0.70	0.38
Learning-based method
RNNInput	0.35	一 0.40	0.25	0.50	0.75	= 0.45
QTypist (-T) QTypist	0.55 0.85	0.65 0.90	0.50 0.85	0.58 0.85	0.80 0.90	0.61

Id	Appname	Categ	Down	M	Triggered activity number							Triggered page number						A+QT
Id	Appname	Categ	Down	M	M+QT			D+QT	IA			A+QT M	M+QT	ID	D+QT		A	A+QT
1	Eskimi	Social	1M+		4133%		4	5 †25%	4	6↑50%	5	8160%		7	10↑43%	8	14175%
2	Recharge	Shop	1m+		5 0%		8			15 167%			8133%	10	16160%	11	18164%
3	Scout24	Vehicle	10M+	354	7 ↑75%		10	↑70%	1	19 ↑58%		9	↑80%	11	↑73%	14	24	↑71%
4	Stock	Finance	1m+	54	9.180%		12	18150%	10	1818 180%	657556	13	3↑86%	2 152	3↑53%	16	26	163%
5	Lite	Photo	1M+		6 150%		6	11↑83%	9	12 2133%		8160%		19	↑58%	14	23	†64%
6	Speedom	Vehicle	1m+	4	5 125%		8	8 0%	86	13 163%		6 120%		9	9 0%	98	15 167%
7	HeartRate	Health	1m+	5	5 0%		6	7 ↑17%		9 ↑50%		6 H	0%	7	9 129%		14 †75%
8	Currency	Tools	1m+	3	30%		5	7 140%	6	7 †17%	4	4	0%	6	8133%	7	11 157%
9	Yandex	Shop	10M+	8	11 ↑38%		14	20 143%	15	22 147%	13	16↑23%		17 23	↑35%	21	31 ↑48%
10	Atom	Finance	1m+	2	2	20%	3	4 +33%	3	5 ↑67%	4	4	0%	5	7 140%	5	8↑60%
11	Coco	Comm	10M+	2	3	↑50%	8	13 ↑63%	7	13 ↑†86%	3	5	↑67%	10	15 ↑50%	9
12	Saviry	Shop	100K+	4	7	175%	8	12 ↑50%	10	13 ↑44%	8	12	↑50%	15	23 ↑53%	18		15 167% 29 ↑61%
13	WhiteMeet	Dating	100K+	7	8	8+14%	10	13 ↑30%		16 5+60%	10	12	↑20%	15	19 +27%	16	25	56%
14	Healthplus	Health	1m+		5	0%	7	7 0%	8	13 ↑63%		10	↑11%	11	13 †18%	13	19	↑46%
15	LINEC	Photo	10M+	554	7	↑40%	12	17 ↑42%	11	18 ↑64%		9	↑50%	11	19 173%	15	27	7†80%
16	FloorPlan	Art	5m+		5 †25%		7	7 0%	19987	10 ↑11%		8160%		10	11 110%	13	18	138%
17	MoneyTK	Finance	10M+	4	6	150%	8	9 ↑13%		13 ↑44%		8↑14%		15	20 ↑33%	17	26	5↑53%
18	Schoolca	Educat	1m+		2	20%	4	6 ↑50%		17 +150%	96574	4	0%		10 ↑25%	8		13163%
19	Flipboa	News	50m+	23453	4 ↑33%		4	5 ↑25%		7 0%		8 ↑60%		87	1 157%	8	12	150%
20	Healthplus	Fitness	1m+		4	40%	8	9 113%	11	12 9%	56557	6	0%	13	19 ↑46%	15		20+33%
21	BlaWM	Travel	1M+		6 120%		12 8	14 ↑17% 9 ↑13%	15 12	17†13% 16 133%		6	↑20% 6 +20%	18 15 16	25 ↑39% 7%	21		27 129%
22 22324252627282930	Wldetector CrAm	Product Educat	1m+ 5M+	2	3 0% 3+50%

提纲

FLIGHT# ROUTE AIRLINES

AIRLINE American Airlines

DATE Sunday,August07

FLIGHT#1626

DON'TKNOW

FINDSEATMAP

Find seat map (text input)

用户界面测试现状和挑战

测试输入生成技术

面向测试路径规划的自动化GUI测试技术

基于多模态大模型的自动化GUI测试技术

针对文本输入的模糊测试技术

面向文本输入组件的交互提升技术

GPTDroid: Function-aware Automatic GUl testing

APK Testing App

Q1(Start prompt):We want to test the "MoneyTracker" App.It hasfollowingactivities..Thecurrentpageis"Main", ...What operationisrequired?([click/double-click// scroll]+)

A1:..Operation:"Click".Widget:"ADD..".

APK Testing App

Q2 (Test prompt):We successfully did the above operation.The ncome Content>,..),and the operationafter input.(

A2:.Operation:"Click".Widget:"Enter".

Q3(Feedback prompt):There isno "Enter",please reselect.The bage theoperationafterinput.([.]+)

A3:..Widget:"Price".Input: "3500".Widget:"Title".Input:"salary". dget:"Category".Input:"personal".Operation:"Click".Widg "Submit'

Q4(Test prompt):We successfully didthe above operation.The The"Menu"isSlideMenu...List oftested.....What operationisrequired?([]+)

A4:..Operation:"Scroll.Widget:"Menu".

Q5 (Test prompt):We successfully did the above operation. The "Main",Ithas"Account","Exchanger
TheupperpartofTeAccountisTextviewwicst nctions:"Functionl:Addaccount ftested History oflatest....Whatoperationisrequired?
([click/double-click//scroll]+)

A5:..Operation:"Click".Widget:"Exchangerates".

Auto GUl testing with LLM

Formulate the automatic GUl testing problem to an interactive question & answering task

to let the LLM conduct the whole app testing by understanding the GUl semantic information and automatically inferring possible operation steps

GPTDroid: Function-aware Automatic GUl testing

GUl context extraction

GUl prompting and executive command generation

Functionality-aware memory prompting

Testing sequence memorizer to record all the detailed interactive testing information, e.g., the explored activities and widgets

Make LLM a Testing Expert: Bringing Human -like Interaction to Mobile GUI Testing via Functionality-aware Decisions. ICSE 2024

? GPTDroid: Function-aware Automatic GUl testing

GUl context extraction

· Accurately depict the GUl page currently under test, as wellas its contained widgets information from a more micro perspective, and the app information from a more macro perspective.

Table 1: Extracted GUI information and examples.

IdAttribute		Description	Examples
GUI context-App information
AppName		Name of the app under testing	AppName =“Money Tracker"
2	Activities	Listofnames forallactivities ofthe app,obtainedfromAndroidManifestxml file	Activities=["Main”,“AddAccount',"Import',“Income,]

3	ActivityName		GUI context - page GUl information ActivityName=“AddPersonalInformation"	4S
4	Widgets	Activity name of the current GUl page Listof all widgets incurrent page,represented with text/id	Widgets =["Edit Account","btn income",]
5	Position	Relativeposition ofwidgets, obtained through their coordinates	Upper =["Welcome”,.],Lower=["Add Income',]

		GUI context - widget information
6	WidgetText	Widgettext, obtained by field‘text' or'hint-text’	WidgetText=“Welcome to the Money Tracker!"
	WidgetID WidgetCategory	Widget ID, obtained by field'resource-id'.	WidgetID ="add account” WidgetCategory =“TextView”
07890	WidgetAction NearbyWidget	Category: TextView, EditText, ImageView, etc, obtained by field'class' Widget action, obtained by field clickable',such as click,input,etc.	WidgetAction =“Click" NearbyWidget= “your income: [SEP] $”	ISCA

? GPTDroid: Function-aware Automatic GUl testing

GUl prompting and executive command generation

· Feedback prompt: inform the LLM error occurred and re-try for querying next operation

· Command generation: provide LLM with the output template, including available operations and operation primitives

Table 2: The example of linguistic patterns of GUI prompts and generation rules.

	Id \| Pattern type	Sample of linguistic patterns/rules	Instantiation
GUI context patterns: GUiContext App information We want to test the <AppName> App. It has the following activ- ities, including <Activities>.
3	Page GUI informa- tion	Thecurrent page is <ActivityName>,ithas<Widgets>.The upper part of the app is <Position>, the lower part is<Position>.	We want to test “Money tracker” App. It has the following activities, including "Main",“AddAccount","Import”,"Setting", The current page is"Main”, it has"Income","Ad', "Delete",. The upper part of the app is "Welcome to.., Delete,..', the lower part of the app is "Income, . The widgets which can be operated are"Ad", "Delete", "Edit Account:,... "Add"”
	Widget information	The widgets which can be operated are < WidgetText/ WidgetID>. <WidgetText/WidgetID> is <WidgetCategory> which can <Wid- getAction> and its nearby widget is <NearbyWidget>. Operation & feedback question patterns: OperationQuestion	is Button which can be clicked and its nearby widgetis"Add account,..”"Delete” is TextView which can be clicked and its nearby widget is ...
Querying general Action operation question + <Output Template> What operation is required? (<Operation>[click / double-click / long press /
	action Querying text input	Input operation question + <Output Template>	scroll]+<Widget Name>) Please generate the input text in sequence,and the operation after input.(< Widget name>+<Input Content>,..) and provided (<Operation[click]>+<Widget name>)
	Testing feedback	Feedback question	There is no <WidgetText / WidgetlD> on the current page, please reselect.
Prompt generation rules Start Prompt: GUIContext[1,2,3] + OperationQuestion[4/5]
2
3	Test Prompt: We successully did the above operation. GUIContext[2.3] + OperationQuestion[4/5] Feedback Prompt: Sorry, <OperationQuestion>[6] + <GUIContext>[3] + <OperationQuestion>[4/5]

Notes:“[1.2,.,6]" means the idof each pattern.Ifthereis "EditText”on the page, GPTDroid selects“Querying text input”patern.For the other widgets, GPTDroid selects“Querying general action" pattern.

? GPTDroid: Function-aware Automatic GUl testing

Functionality-aware memory prompting

· Build a testing sequence memorizer to record alldetailed testing information

· Query the LLM about the function-level progress of the testing

· Functionality-aware memory prompt

Table 3: The example of linguistic patterns of functionality-aware memory prompts and generation rules.

Id\| Type		Sample of linguistic patterns/rules	Instantiation
Long-term functionality-aware memory patterns:FunctionMemory List of tested functions: <Function Name> + <Visits Time> + List of tested functions: “Function1: Add your income. Visits: 3. Status:
2 3	Explored function- alities Covered activities. Recently tested operations 贵新云所	<Status>,<Function Name> + <Visits Time> + <Status> Path of tested activities: <Activity Name> +<Visits Time>,<Ac- tivity Name> + <Visits Time>,. History of latest tested pages and operations: Latest 5th step tested the <Activity Name> page. The following widgets with visits time of this page have been tested: <Widget Name> + <Visits Time>, .. The following executive command achieve the page transition: <Operation> + <Widget Name>. Latest 4th step tested the <Activity Name> page. The following..	Finished”,"Function2: Delete information. Visits: 2. Status: Finished", Path of tested activities: “Activity: Main. Visits: 3", “Activity: Account. Visits: 4", "Activity: AddAccount. Visits: 3", History of latest tested pages and operations: Latest 5th step tested the “Exchange page. The following widgets with visits time of this page have been tested: “Widget: Add exchange, Visits:2","Widget: Submit ex- change, Visits:1", Widget: Cancel, Visits:1"... The following executive command achieve the page transition:“Click" the "Exchange'. Latest 4th step tested the“main" page. The following widgets..
		Latest 1st step tested the <Activity Name> page. The following.. Function question patterns: FunctionQuestion		Latest 1st step tested the“Add Account" page.The following widgets..
Functionality in- 4 quiry		Functionality inquiry + <Output Template>	tion? (<FunctionName> + <Status>)	What is the functions currently being tested? Are we testing a new func-
	Functionality-aware memory prompt generation rules
	Functionality-aware memory Prompt: FunctionMemory[1.2,3] + FunctionQuestion[4]

Example

Table 4: The example of how prompts work in GPTDroid.

Prompt type	Instantiation
Example 1: For general action operation
GUI context院线修研公所	Start Prompt:[We want to test “Money tracker" App, It hasthe following activities, including ....] or Test Prompt: [We successfully did the above operation.] or Feedback prompt:[There is nc“Exchange"on the current page. please reselect.] [The current page is “AddAccount” il has .. The upper partof the app is ... . the lower part... The "Exchange" is Button...]
Functionality-awarememory	[List of tested functions: “Functionl Add your income..., ..] [Path of tested activities: “Activity: Main Visits:73", .. ] [History oflatest tested pages and operations: Latest 5th step tested the “Exchange" page. The followingwidgets...Latest 1st step tested the"AddAccount" page..]
Function ques-tion	What is the functions currently being tested? Are we testing anew function? (<Function name> + <Status>)
Actionquestion	What operation is required? (<Operation>[click/ double-click/ long press / scroll]+<Widget Name>)
LLM Answer	Function: "Add income".Status:Yes.Operation: "Click'. Widget: "ADD INCOME".

三 Records Day

Short summary 》01Feb,2023

Total income +ONONTotal expense -ONON

+ONON

ADD EXPENSE

Example2: For text input operation
GUI context	Start Prompt:[.,] or Test Prompt:. or Feedback prompt: ..
functionality-awarememory	[List of tested functions: "Function1: Ad .....] [Path of tested activities: .] [History of latest tested pages ...]
Function ques-tion	What is the functions currently being tested? Are we testing anew function?(<Function name> + <Status>)
Input question	Please generate the inpul text in sequence, and the operationafterinput.(<Widget name>+<Input Content>,..and provided<Operation>+<Widget name>)
LLM Answer	Function: "Add income".Status:No.Widget: "Price". Input:"35oo".Widget: "Title".Input:"salary".Widget:"Category".Input:"personal".Operation: "Click".Wid-get:"Submit".

Evaluation

75% average activity coverage, 32% higher than the best baselines

detects 95 bugs for the 93 apps, 31% higher than the best baselines

Evaluation

Capability of GPTDroid

· Function-aware exploration through long meaningful testing trace

· Prioritization

· Valid text inputs & compound operations

(a)Function-awareexplorationthrough longmeaningfultestingtrace

(b)Function-awareprioritization

提纲

FLIGHT# ROUTE AIRLINES

AIRLINE American Airlines

DATE Sunday,Augut07

FLIGHT#1626

DON'TKNOW

FINDSEATMAP

Find seat map (text input)

用户界面测试现状和挑战

测试输入生成技术

面向测试路径规划的自动化GUI测试技术

基于多模态大模型的自动化GUI测试技术

针对文本输入的模糊测试技术

面向文本输入组件的交互提升技术

VisionDroid: Vision-driven Automated Mobile GUI Testing CA L

Crash-bugs vs. Non-crash functional bugs

Bug:After sortingthe list by the lowest; price,the list isn'tcorrectlysorted.
Bug:A listing showsan incorrect price of \$0 fora stay.

Bug:Afterclicking'Wish',the wishlist is not displayed.

(a) Bug: The "Income" has been repeatedly added

(b) Bug: The link provided by the "Notification" button is incorrect

VisionDroid: Vision-driven Automated Mobile GUI Testing 中国科学税线 AS

Vision-driven, multi-agent collaborative automated GUl testing approach for detecting non-crash functional bugs

■ Explorer Agent: navigates through the app, captures view hierarchies and screenshots, and guides the exploration towards diverse GUl pages while focusing on the app' s functionalities.

Monitor Agent: supervises the testing process, records the exploration history, and triggers the detector agent at the appropriate time.

→ Detector Agent: identifies potential functional bugs by examining whether there are any issues in the logical transitions that occur during GUl page changes

Vision-driven Automated Mobile GUl Testing via Multimodal Large Language Model, arxiv

VisionDroid: Vision-driven Automated Mobile GUI Testing □ Challenae 1: Alianina visual and text for MLLM inr Challenge 1: Aligning visual and text for MLLM input.

Vision-driven Automated Mobile GUl Testing via Multimodal Large Language Model, arxiv

#

Alignment method that integrates text properties with visual context;

Screenshot annotation method, pay attention to different types of actionable widgets, resolve issue of overlapping

Challenge 2: Functionality-oriented exploration.

Infer and abstract the current functionality from S detailed exploration sequences, avoids exceeding token limits when interacting with the LLM, enable exploration more focusing on the functionality aspect

Challenge 3: Infering test oracle.

Let the Monitor Agent trigger the Detector Agent at the end of each functionality exploration

Functionality-aware Chain-of-Thought (COT) to enable the MLLM to first explicitly infer oracles and then detect functional bugs based on these inference

Explorer Agent

(a)Directannotationmethods

(b) Our annotation methods

Explorer'sTextPromptTemplate

BelowareinstructionsforanAppexploration task.[Task Description]

[GUl information]:

TheAppis{Appname}.Itsactivitiesare{Activities}.Thecurrent pageis{Activity
name}. FromSection IIIA

[Testing history]:

Thetested functionsand the visit number toeach functionare as follows:
{Functionmame}+{Visit number)... FromSection III B

[Explore optimization]:

According to thefeedback from thetesting process,
{Possiblebugpath} (e.g.,Click the...widgetin...page,then click...)
{Exception path} (e.g.,The...action is wrong.Click..) FromSection III C

###[Legend of image]:

{Legend of current GUlscreenshots} (e.g.,We label theactionable widgets in thefirst column ofeachrowinorderfrom left toright,....Wealso label the typesof actionablewidgetsaccordingtotherules,withredas"click",...）FromSection IIIA {Legendofrecentlytestedscreenshots:}On the leftsideoftheimagearethe screenshotsfromthelast4stepswiththelabeledactionwidget.Eachscreenshot is labeled withthepagedescriptionof the tested page... FromSectionIIIB

##[Query]:

{Querying generalaction} (e.g.，Which numberedwidget?Whatactionsareneeded？ What isthetextual descriptionof thewidget?(Widgetnumber]+[Action][click/longclick/check/scroll]+[Widget text]）.）

{Querying text input (e.g.Please givethewidgetnumber，generate the input text, andtheactionafterinput.(Widgetnumber]+[InputContent])provided([Widget number]+[Action[click]]+[Widgetext]）.）

Explorer'sImagePrompt

Monitor Agent

Search the hotel Choose the hotel dates The hotel list (a) Image view

(b) Test view

Monitor'sTextPromptTemplate

Thefunction being testedintheprevious stepis{Function name)，andtheactionis {Action}.

###[Testing history]:

Thetested functionsand the visit numbertoeach functionareas follows: {Functionmame}+{Visit number}...

###[Legend of image]:

{Legend ofrecently tested screenshots:}On the left side of the image are the screenshotsfromthe last4stepswiththe labeledactionwidget.Eachscreenshot is labeledwith thefunctionnameofthetestedpage...

##[Query]:

{Querying pagedescription} (e.g.，What is thepagedescription?([Pagedescription]) {Querying function) (e.g.,Pleasemakea judgmentbasedontheGUl informationon thepage.What is thefunctioncurrently being tested?Arewe testinganewfunction？ ([Function name] +[Status]).）

Monitor'sImagePrompt

Detector Agent

■ Enriching Detector Prompt with Example

bug description, bug screenshot and natural language described bug reproduction path which facilitates the MLLM understanding of what the non-crash functional bugs are

Detector'sText Prompt Template

Belowareinstructionsfortheno-crash bugdetection task.[TaskDescription] ###[Bug examples]：

Thecategoriesof bugsinclude{Bug category}.Weprovidethe bugexamples: （1{Bugdescription}+{Bug replaypath）..

###[Function-driven COT]:

Let'sthink step by step.

1.Function ldentification:The tested function {Function name)...Itsdescription is.

2.Expected Path:Please predict the exploration steps for this fucntion basedon above information.

3.Actual Path:The tested function sequence isasfollows:

（1）The1stpageis{Page description}+{action}

（2）The2st page is{Pagedescription}+{action}

（3）...

##[Legend of image]:

Theimage showsa test sequencearrangedinorderfrom left torightand fromtopto bottom.Each screenshot is labeledwithadiferentcolored boundingboxindicating theactionof the operation(redas"click",blueas"input..).Belowthescreenshot is thecorresponding page description.

##[Query]:

{Querying bugdetection} （e.g.，Pleaseanalyzeeachstep in the test sequence in the imagebased on thedescriptionand examplesof the bugs,predict whether the page thattransitsaftereach stepmeetsyourexpectation,use thistodeterminewhether thereareany bugs,andpoint out thebugpage.)

{Queryingpossible bugpath) (e.g.，Is thereany pageoroperationthatcould potentially triggersomebugs?Pleaseprovide thecorrespondingpageandaction widget.)

{Querying exceptionpath} (e.g.,Byanalyzing the test path，if thereareanypagesor operationsinthepath that haveabnormalities?Pleaseprovide the corresponding page and action widget.)

Evaluation

TABLE I: Result of coverage and bug detection (RQ1)

Method	Act 1	Baseline data Rep							GitHub data Rep\|Bug		P			Bug injection data					R
Method	Act 1	Cod \|		Bug	P		R	Act	Cod					Act	Cod	Rep\|	Bug		R
OwlEyes+M	0.21	0.18	77	10	0.13	0.08	0.33	0.28	140	62	0.44	0.16		0.35	0.31	96	13	0.14	0.18
NightHawk+M	0.21	0.18	163	13	0.21	0.10	0.33	0.28	136		62	0.46	0.16	0.35	0.31	60	15	0.25	0.21
DiffDroid+M	0.21	0.18	6	15	0.25	0.11	0.33	0.28	133	55	0.41		0.14	0.35	0.31	46	12	0.26	0.17
dVermin+M	0.21	0.18	53	4	0.08	0.03	0.33	0.28	98	28	0.29	0.07		0.35	0.31	33	1	0.33	0.15
OwlEyes+A	0.29	0.27	92	18	0.20	0.14	0.43	0.37	161	85	0.53		0.22	0.42	0.4	93	10	0.11	0.14
NigbtIiawktA	0.29	0.27	86	21	0.24	0.16	0.43	0.37	157	91	0.58		0.24	0.42	0.4	67	16	0.24	0.22
DiffDroid+A	0.29	0.27	87	20	0.23	0.15	0.43	0.37	147	68	0.46		0.18	0.42	0.4	44	7	0.16	0.10
dVermin+A	0.29	0.27	67	6	0.09	0.05	0.43	0.37	101	31	0.31	0.08		0.42	0.4	35	中	0.26	0.13
	0.39	0.37	107						180	79									0.17
OwlEyes+F NightHawk+F	0.39	0.37	100	17 20	0.16 0.20	0.13 0.15	0.47 0.47	0.42 0.42	161	92	0.44 0.57		0.20 10.24	0.43 0.43	0.39 10.39	94 59	12 15	0.13 0.25	0.21
DiffDroid+F	0.39	0.37	94	20	0.21	0.15	0.47	0.42	131	72	0.55		0.19	0.43	0.39	53	12	0.23	0.17
dVermintF	0.39	0.37	4	8	0.20	0.06	0.47	0.42	90	35	0.39		0.09	0.43	0.39	67	8	0.12	0.11
OwlEyes+H	0.38	0.35	97	12	0.12	0.09	0.48	0.44	157	79	0.50		0.20	0.43				0.15	0.19
Night Iawk+1I	0.38	0.35	88	16	0.18	0.12	0.48	0.44	146	79	0.54		0.20	0.43	0.39 10.39	93 88	1 13	0.15	0.18
DiffDroid+H	0.38	0.35	75	17	0.23	0.13	0.48	0.44	143	57	0.40		0.15	0.43	0.39	57	15	0.26	0.21
dVermin+ H	0.38	0.35	33	5	0.15	0.04	0.48	0.44	82	40	0.49		0.10	0.43	0.39	1	8	0.20	0.11
OwlEyestG	0.47	0.43	99	17	0.17	0.13	0.51	0.47	162	79	0.49		0.20	0.47	0.41	95	14	0.15	0.19
NightlHawk+G	0.47	0.43	95	18	0.19	0.14	0.51	0.47	157	96	0.61		0.25	0.47	:0.41	68	12	0.18	0.17
DiffDroid+G	0.47	0.43	85	16	0.19	0.12	0.51	0.47	142	76	0.54	0.20		0.47	0.41	47	，	0.19 0.21	0.13 0.13
dVermin+G iFixdataloss	0.47	0.43	37	6	0.16	0.05	00.51	0.47	75	石7	0.63	0.12		0.47	0.41	42

Notes:“Act”isactivitycoverage,“Cod”isCodecoverage.“Rep”isnumberof bugsreportedbytools,“Bug”istruebugin thereported bug,“P”isprecisionand“R”isthe recall“+M”meansitwithMonkey,“+A”meansitwithAPE,“+F”meansitwithFastbot,“+H”meansitwithHunmanoid,and“+G”meansitwithGPTDroid.

50%-72% precision and 42%-65% recall

more than 14%- 112% and 108%- 147% boost in average recall and precision compared with the best baseline

面向自然语言描述的测试用例迁移

测试需求：我家-添加资产流程

提纲

FLIGHT# ROUTE AIRLINES

AIRLINE American Airlines

DATE Sunday,August07

FLIGHT#1626

DON'TKNOW

FINDSEATMAP

Find seat map (text input)

用户界面测试现状和挑战

测试输入生成技术

面向测试路径规划的自动化GUI测试技术

基于多模态大模型的自动化GUI测试技术

针对文本输入的模糊测试技术

面向文本输入组件的交互提升技术

面向文本输入的模糊测试

Intra-widget constraint:

requirements of a single text input, e.g., a widget for a human' s height can only input the non-negative number.

Inter-widget constraint: requirements among multiple text input widgets on a GUl page, for example, the diastolic pressure should be less than systolic pressure.

Unusual Text Inputs Generation

■ produce the test generators (a code snippet) with LLM

Each can generate a batch of unusual text inputs under the same mutation rule (e.g., insert special characters into a string)

Testing the Limits: Unusual Text Inputs Generation for Mobile App Crash Detection with Large Language Model, ICSE 2024

Example prompt

We wantto test the test inputson...The first input widget is username,itsontextis.,nditsdynamichintis...hree 5explicit intra-constraints:(1) Pure text...(2). Please generatea valid input based on above informationand provide inferred constraints ofeach input.

Promptgenerationforvalid input(Section 3.1）

generate text based on thefollowingconstraints:(1) Special characterspet）thee The inputis"Frank238]

Example1:Context info:"User name,..."; Mutatioulesetspeihacte&it Test case:"FXD%S&";

Example2:Contextinfo:"Pleaseinputusername.." Mutationrule:null Testcase:"UserFFDSE:2SA";

Retrievedexampleofthebuggyinput(Section3.3）

Example5:

Inferred constraints:"This is a financial app that requires a 'username'tobe entered on the current page.It hasthefollowing constraintsto generate theusername:'Specialcharacters cannot appearinthe text,andtheusername.
Valid input:"Frank238";
Testexecution feedback:"We tried the following mutationrules:

Inferred constraints:"This isa financial app that requiresa 'username'tobeenteredonthecurrentpage.lthasthefollowing constraints togeneratetheusername:'Specialcharacterscannot appearinthe text,and theusernam.e....

Test execution feedback: "We tried the following mutation rules: (1)Adjust the numberofcharactersto1..;(2).;Thesemutation rulesdidnottriggerbugs.(3)Insertspecial..;Ittriggeredbugs. Please trymore types ofmutation rules." Question:"Please generate a valid input based on the above informationandprovide the inferredconstraintsofeach input."

LargeLanguageModel CHATGPT output

Mutation rule:"lnsert character'@#S%'into..." Test generator:___-_

Prompt
generation for
test generator with
mutationrule
(Section3.2）
invalid_destination_inputs=['-1','1.2','@#\$%','A'*101]
for invalid_destination .getusername(self.username(),input_text) self.assertEqual(self.get_username_results(),[])

generate

Unusualinput:Frank23@#\$%8,-1Frank2378,Frank23394,..

Input&test Text input widget

Frank23@#\$%8 Buggy input

Frank23394

Non-Buggyinput

Evaluations

Method	Setting 1(30 attempts)Bug(%) Attempt(#)		Setting 2 (30 minutes)Bug(%) Min(#)
InputBlaster	0.72 13.52		0.78 1	9.64
ChatGPT	0.25 25.91		0.28	23.28
Mutation or fuzzing methods
GoldTestPDinvalidRandomFuzzruleMutator	0.080.190.250.28	29.2228.6522.3121.42	0.080.190.250.28	28.7322.7321.5520.53
String analysis methods
SlothOSTRICH	0.250.22	23.6124.14	0.250.22	22.6123.41
Constraint-basedmethods
MobolicTextExerciser	0.170.31	25.8322.11	0.170.33	25.0920.18
Valid input generation methods
RNNInputQTypist	0.060.08	28.6727.78	0.060.11	28.6427.31
Automated GUI testing methods
ApeDroidBotStoatTimeMachine	0.080.06	28.11	0.11	26.8828.3427.58
		28.39	0.06

		0.08	27.94	0.08
		eMachine	0.11	26.92	0.11	26.6925.85
ComboDroidQ-testingHumanoid	nboDroid	0.14	26.11	0.14
	0.110.11	27.06	0.11	26.7025.85
		26.92	0.14

Notes: "Bug (%)” is the average bug detecting rate,"Attempt (#)” is the average number of unusual inputs before triggering the crash,"Min (#)” is the average running time (minutes) before triggering the crash.

72% - 78% found bugs, significant higher than baselines

· Significant few attempt times

Validinput:MelbourneSkydeck,-2023 110,260

Unsualinput1:Melbourne-1234,01*-0@2-21, 1100,2609

Unsualinut2:ydney33-84bc194
Unsualinput3:Perth16,193-%4,-1,44
Unsualnput4:NewYork993,2.219

Buggy input: ^&###?>Dmep-d3, 03-07-2024, 308,22

Valid input: New York,Washington DC. 01-04-2023,09-04-2023

Unsual input2: NewYorkssPars% 00-88-MC23,19-34-211

Unsual input3:New York,New York, -3.1-12-1933,2531-041-1911

Valid input: 130,70

Unsual input 1:13.1,407

Unsual input 2:9000,9000

Unsual input3:130\$,137*&^

Unsual input 4:aasec,franm#

Buggy input: 30, 1878

提纲

FLIGHT# ROUTEAIRLINES

AIRLINE American Airlines

DATE Sunday,Augut07

FLIGHT#1626

DON'TKNOW

FINDSEATMAP

Find seat map (text input)

用户界面测试现状和挑战

测试输入生成技术

面向测试路径规划的自动化GUI测试技术

基于多模态大模型的自动化GUI测试技术

针对文本输入的模糊测试技术

面向文本输入组件的交互提升技术

Good Examples of Text Input

Examples of differences

√ hint-text

√ label

√ context description

(a) Label of image component:Send

(b) Content description: Lease Rate

Bad Examples of Text Input

Without hint-text

√ screen reader cannot obtain information

√ simple and lacks meaning

(a)Textinputwithouthint-text(b)Textinputwithoutmeaningfulhint-text(c)Textinputwithmeaningfulhint-text

Motivation Study

√ Dataset: 4,950 apps from 33 categories from Google Play

√ 3,398 (76%) of them are without hint-text

√ 30,226 (66%) had at least one text input without explicit hint-text content

(a) Hint-text missing rate with differentcategory of apps

(b) Hint-textmissingratewith download number

HintDroid: Predicting Hint-text of Text Input

· HintDroid: Predicting Hint-text of Text Input □ GUI Entity Extraction and Prompting EAS

□ Enriching Prompt with Examples

□ Feedback Extraction and Promptina Module 2: Enriching Prompt with Examples (Section 5.2).

HintDroid

TEXT

Unblind Text Inputs: Predicting Hint-text of Text Input in Mobile Apps via LLM, CHl 2024

Module 1: GUl Entity Extraction and Prompting

· GUI Entity Extraction and Prompting

√ App Entity Information + Page GUl Entity Information

√ Input Component Entity information

Type	Entity	Description	Instantiation
App information	[AppName][Activities]	Name of the app under testing List of names for allactivities of the app, obtained fromAndroidManifest.xml file	[AppName]:"Flight”[Activities]: “Main, OneWay, RoundTrip,."
[ActivityName] Activity name of the current GUI pageCASPage GUI information [Component] List of all widgets in current page, represented with tex-t/idRelative position of widgets, obtained through their co-[Position] ordinates			[ActivityName]: “RoundTrip”[Component]: “Depart, Arrive, Departure time,."”[Position]: Upper: “Flight Search,.",Lower: “Departure time,
Input component information	[TextInput][NearbyInput]	The text input denoted with the textual related fieldsNearby widgets denoted with their textual related fields	[TextInput]:"Departure time"[NearbyInput] :“Flight Search,..”

. Module 1: GUl Entity Extraction and Prompting

· Example of the GUl prompt construction

Id Prompt Type		Instantiation	\|Examples
In-context learning prompt We will provide you with 6 examples:
1	<Hint-text examples>	We will provide you with 6 examples: 1. [TextInput],[NearbyInput],[Hint-text] 2.[TextInput],[NearbyInput],[Hint-text]	1sttext input is “From",its nearby components are“.",its hint-text is... 2ndtextinputis“To”,itsnearbycomponentsare..,tshint-textis
	CAS	6.[TextInput], [NearbyInput],[Hint-text] GUI prompt	6th text input is “Flight,its nearby.,its hint-text is “Enter the city"

2	<App info>	[AppName], [Activities]	The app name is“Flight,it has following activities:“Main,." The current GUI page is “SearchFlight”, it has following compo-
3 4	<Page GUI info> ISO <Input component info>	[ActivityName], [Component], [Position] [TextInput], [NearbyInput]	nents:"Search,.", the upper partof the page is“..",the lower part... The text input of this page is “Depart", its nearby components are.
Feedback prompt
5	<Feedback>	[Feedback], [ErrorMessage]	The input content “train” doesn't pass the page,the eror message of the input component is: “Please enter the correct city name".
		Query & feedback query
6	<Query> <Feedback Query> CCAT	Please generate a hint-text for the input component based on the above information,and generate corresponding input content based on the generated hint-text. Please regenerate the hint text and its corresponding input content based on the feedback information above.

8	<Example output>	Example output [Hint-text],[InputContent]	Please output according to the following example: the hint-text is “xxx", the input content is “xxx".

Module 2: Enriching Prompt with Examples

· Enriching Prompt with Examples

√ Example Dataset Construction

√ Retrieval-based Example Selection and In-context Learning

Textinput inview hierarchy file

In-context learning prompt: We'll provide you with 6 example:

1sttext input is"From",itsnearby componentsare "Searchthe Train,Departuretime,Submit",itshint-textis"EntertheDepart". 2ndtextinputis "Depart",itsnearbycomponentsre "Searchtrains,Searchdata,howTrains",itshnt-textis"Departureity". dte

5thtextinputSto/stio,itsryopoetre"Tinum/ete,ech",tttetis"Cy" thext

GUlpromhtii ThecurrentGgeis"ourip",itsfowingompoents"ghtearchteprtttrvalieeparte.", theupperpartofthepageis"FlightSearch,txtDepart,txtArival,Airline",thelowerpartofthepageis"Depart Date..". Thetextinutofthispgeis"Depart,tsnearbycomponentsre"FlightSearch,txtArrival,rlie..

Feedback prompt:Theinputcontentdoesn't passthe page,the errormessage is"Pleaseenterthecorrectcityname".

Query:Please generateaint-textfortheinputcomponentsedontheboveinformationndgenerateinputcotent.

Exampleoutput:Pleaseoutputacordingtothefollowing exampleututteetxx,toet

? Module 3: Feedback Extraction and Prompting

· Feedback Extraction and Prompting

√ Automated Input Content Checking

√ Error Message Extraction

Textinputinview hierarchy file
Promptforhint-text generation

Implementation

√ Extracting GUl pages

√ Detecting GUl pages with missing hint-text

√ Predicting hint-text based on GUl information

√ Decompiling APK to obtain code

√ Repackaging APK after code modification

Evaluation - Effectiveness

We evaluate the effectiveness of HintDroid from the point of view of the hint-text generation accuracy. S

Method	Exact match	BLEU@1	BLEU@2	BLEU@3	BLEU@4	METEOR	ROUGE-L	CIDEr
Learning-based method
RNN	0.29	0.37	0.35	0.31	0.29	0.26	0.24	0.21
武山 LSTM	0.28	0.33	0.31	0.25	0.22	0.19	0.17	0.13
Seq2Seq	0.30	0.37	0.32	0.29	0.27	0.25	0.21	0.18
Transformer	安 0.39	0.53	0.47	0.43	0.40	0.37	0.36	0.35
RNNInput	a 0.28	0.35	0.33	0.32	0.27	0.25	0.23	0.19
LableDroid	0.34	0.47	0.45	0.39	0.36	0.35	0.32	0.31
CNN+LSTM	0.26	0.29	0.24	0.19	0.17	0.15	0.09	0.08
Matching-based method
Retrieval based	0.21	0.27	0.24	0.22				A
Random based	0.11	0.16	0.13	0.10	0.18 0.07	0.20 0.08	0.18 0.09	0.15 0.07
Rule based	0.32	0.45	0.39	0.32	0.27	0.29	0.31	0.27
LLM-based method
GhatGPT	0.35	0.49	0.43	0.39	0.36	0.38	0.33	0.31
QTypist	0.31	0.47	0.41	0.35	0.33	0.37	0.34	0.33
HintDroid	0.71	0.83	0.77	0.73	0.66	0.67	0.63	0.62

Evaluation - Ablation Study

We can see that HintDroid's hint-text generation performance is much higher than all other variants. S

Evaluation - Usefulness

Whether HintDroid can help visually impaired users fill in the input.

→ Data: 33 apps with 237 text input

→ Participants:

→ P1 to P18 use HintDroid

→ P18 to P36 without

→Metrics:

→ Input accuracy

→ Activity coverage

→ State coverage

→ Filling time

Basic information			Input accuracy		Activity coverage		State coverage		Filling time (min)
id	App	Category	control	experiment	control	experiment	control		experiment	control	experiment
1	HealthHB	Health	0.50	0.81	0.54	0.72	0.51	0.64		2.38	1.18
2	WeatherL&W	Weather	0.40	0.88	0.24	0.66	0.62	0.78		1.93	1.02
3	MessagerWA	Communi	0.31	0.95	0.29	0.73	0.24	0.68		2.30	1.10
4	MoneyTK	Finance	0.53	0.72	0.50	0.69	0.36	0.74		2.74	0.70
5	FoodFacts	Food	0.44	0.90	0.52	0.65	0.47	0.63		1.49	1.37
6	MPAS+	Maps	0.50	0.80	0.28	0.67	0.35	0.74		1.79	1.46
7	PSStore	Product	0.34	0.80	0.28	0.63	0.52	0.76		1.50	1.43
8	NewAudio	Music	0.42	0.84	0.20	0.64	0.51	0.82		2.89	0.64
9	WallETH	Personal	0.47	0.90	0.21	0.74	0.26	0.81		1.72	0.93
10	PicGall	Photo	0.45	0.71	0.27	0.59	0.37	0.81		2.92	0.53
11	SmartNew	News	0.36	0.75	0.52	0.68	0.25	0.64		1.60	1.49
12	MyHM	House	0.18	0.87	0.47	0.75	0.43	0.71		2.56	0.83
13	INSTEAD	Life	0.15	0.77	0.53	0.69	0.30	0.75		1.80	0.54
14	GameSpe	Game	0.55	0.73	0.25	0.75	0.50	0.76		1.71	0.51
15	BusinessEX	Business	0.12	0.72	0.21	0.65	0.35	0.79		1.50	0.55
16	PocketMaps	Travel	0.14	0.80 个	0.41	0.71	0.46	0.78		2.84	0.58
17	EventOR	Events	0.12	0.95	0.52	0.73	0.57	0.75		1.43	0.23
18	FitTAP	Comics	0.16	0.93	0.46	0.69	0.31	0.82		2.13	0.46
19	SkyTube	Video	0.17	0.96	0.45	0.72	0.36	0.82		2.19	0.95
20	LibReader	Books	0.26	0.71	0.33	0.69	0.25	0.69		2.50	0.32
21	NoxSecu	Tool	0.53	0.96	0.47	0.68	0.63	0.64		2.03	1.29
22	EarnMon	Social	0.13	0.91	0.39	0.73	0.63	0.66		2.29	0.32
23	WalkTra	Sports	0.35	0.70	0.42	0.72	0.56	0.65		2.71	0.68
24	ParentLA	Parenting	0.36	0.72	0.24	0.69	0.59	0.80		1.70	1.55
25	ISAY	Medical	0.53	0.96	0.48	0.60	0.29	0.72		2.18	0.43
26	Ipsos	Commun	0.09	0.72	0.54	0.79	0.55	0.79		2.71	1.25
27	FIRR	Libraries	0.53	0.79	0.52	0.65	0.34	0.67		1.33	1.23
28	DRBUs	Shopping	0.28	0.72	0.53	0.74	0.63	0.71		2.19	1.44
29	Learning	Education	0.49	0.89	0.33	0.69	0.40	0.63		2.17	0.95
30	MMDR	Dating	0.06	0.74	0.21	0.75	0.31	0.77		1.37	0.73
31	Pretty	Beauty	0.19	0.94	0.58	0.70	0.60	0.73		1.99	0.36
32	Fair	Auto	0.33	0.82	0.46	0.74	0.51	0.65		1.77	1.25
33	ArtPIX Average	Art	0.49 0.33	0.85 0.83	0.24 0.39	0.57 0.69	0.41 0.44	0.67 0.73	2.10	2.88	0.86 0.88

Evaluation - Usefulness

Example of different good cases generated by HintDroid

(a)Text input usesabbreviation

(c)Textinput withmeaningful hint-text

总结和展望

用户界面测试现状和挑战

测试输入生成技术

面向测试路径规划的自动化GUI测试技术

基于多模态大模型的自动化GUI测试技术

针对文本输入的模糊测试技术

面向文本输入组件的交互提升技术

■ 测试覆盖率达不到100%

挑战

→ 页面元素远超开源应用，单个页面元素常常超过100个

图标类组件，缺失label等元素

■ 文本描述中的模糊描述，左上，右上

一些特殊表述，例如：选择地区

■ 大模型输出里面标号和描述对不上

2024AI+研发数字峰会 Al+ Development Digital summit AI驱动研发变革 促进企业降本增效

基于多模态大模型的用户界面交互和测试

演讲嘉宾

王俊杰

目录

用户界面测试现状和挑战

面临挑战

相关成果

ICSE 2023

ICSE 2024

ICSE 2024

1INTRODUCTION

Under submission

Vision-driven AutomatedMobile GUI Testingvia MultimodalLargeLanguageModel

CHI2024最佳论文提名奖

Unblind TextInputs:Predicting Hint-textofText Input inMobile AppsviaLLM

ABSTRACT

1INTRODUCTION

ICSE 2024

TSE 2024

Software Testing with Large Language Model: Survey，Landscape,and Vision

FSE2024-SE2030

ABSTRACT

提纲

Text Input Generation

Ask LLM to fillin the blank according to the generated prompts

Text input generation

Text input generation for mobile app testing

Examples

Evaluation

提纲

GPTDroid: Function-aware Automatic GUl testing

APK Testing App

A1:..Operation:"Click".Widget:"ADD..".

APK Testing App

A2:.Operation:"Click".Widget:"Enter".

Auto GUl testing with LLM

GPTDroid: Function-aware Automatic GUl testing

? GPTDroid: Function-aware Automatic GUl testing

GUl context extraction

? GPTDroid: Function-aware Automatic GUl testing

GUl prompting and executive command generation

? GPTDroid: Function-aware Automatic GUl testing

Functionality-aware memory prompting

Example

Evaluation

Evaluation

Capability of GPTDroid

提纲

VisionDroid: Vision-driven Automated Mobile GUI Testing CA L

Crash-bugs vs. Non-crash functional bugs

VisionDroid: Vision-driven Automated Mobile GUI Testing 中国科学税线 AS

VisionDroid: Vision-driven Automated Mobile GUI Testing □ Challenae 1: Alianina visual and text for MLLM inr Challenge 1: Aligning visual and text for MLLM input.

#

Challenge 2: Functionality-oriented exploration.

Challenge 3: Infering test oracle.

Explorer Agent

Explorer'sTextPromptTemplate

[GUl information]:

[Testing history]:

[Explore optimization]:

###[Legend of image]:

##[Query]:

Monitor Agent

Monitor'sTextPromptTemplate

###[Testing history]:

###[Legend of image]:

##[Query]:

Detector Agent

■ Enriching Detector Prompt with Example

Detector'sText Prompt Template

###[Function-driven COT]:

Let'sthink step by step.

##[Legend of image]:

##[Query]:

Evaluation

面向自然语言描述的测试用例迁移

提纲

面向文本输入的模糊测试

Intra-widget constraint:

2024AI+研发数字峰会 Al+ Development Digital summit AI驱动研发变革促进企业降本增效

AiDD-人工智能行业2024AI+研发数字峰会：基于多模态大模型的用户界面交互和测试-250320.pdf