2024AI+研发数字峰会 Al+ Development Digital summit AI驱动研发变革促进企业降本增效

北京站 08/16-17

多模态文档大模型mPLUG-DocOwl

徐海洋阿里巴巴通义实验室

演讲嘉宾

徐海洋

阿里巴巴通义实验室-高级算法专家

负责通义多模态大模型mPLUG系列工作，包括基础多模态模型mPLUG/mPLUG-2，多模态对话大模型mPLUG-Owl，多模态文档大模型mPLUG-DocOwl，多模态智能体Mobile-Agent等，其中 mPLUG 工作在VQA榜单首超人类的成绩。在国际顶级期刊和会议ICML/CVPR/ICCV/ACL/EMNLP/MM/TOIS/IJCAI/AAAI等发表论文40多篇，并担任多个顶级和会议AC/PC/Reviewer。主导参与开源项目mPLUG，X-PLUG，AliceMind， DELTA。

多模态文档大模型背景

PART01多模态文档大模型背景

THEROBERTA.WELCHFOUNDATION 2010Bank of the Southwest Building Houston,Texas 77002

YEARI May1,1966 M through A

1.Personnel s11,228.00s

2.Permanent Scientific Equipment.......... S 0$

3.Expendable Scientific Items& Services. $ 840.00$

4.Othet Expense.. s 975.00s

5.TOTAL Exclusive of Overhead 513.043.00s

6.Overhead s1,957.00$

7TYTAAMOKTDNAnd Dn 15n0nn a

BUDGETREQUEST SL

aboutcampaigns licensing membership resources community donate shop

Intern at the FSF by Free Software Foundation

This is an educational opportunity to work with the organization that sponsors the GNU Project,publishes the GNU General Public License (GPL),andfights forsoftware freedom.

As an intern, you work closely with FSF staffmembers in your area of interest, such as campaign and community organizing, free software licensing, systems and network administration, GNU Project support, or Web development.

All applicants must be:

·open-minded and tolerant of others, • able to work as part of a team, and • interested in the ethical ramifications of computing.

We strongly prefer applicants able to work in-person at the FSF headquarters in downtown Boston, but applicants based elsewhere may also be considered.

Unfortunately, due to the complexities of US immigration law, applicants who are not US citizens or permanent residents and do not have a visa allowing them to work in the US must work remotely from outside the US.

These positions are unpaid unless you are doing your internship through Outreachy, but in either case, the FSF will provide any appropriate documentation you might need to receive funding and school credit from outside sources. We place an emphasis on providing hands-on educational opportunities for intems in which they work closely with staff mentors. Take a look at our past interns and some of their projects, including promoting GNU, launching the free Javascript campaign,contributing to Defective by Design,and updating the Email Self-Defense project.Youcan also see dozens of bloas from oast interns who worked for our tech team here. A smal number of bast interns have even become

Take Action!

Enter your email address to
receive our monthly
newsletter,the
FreeSoftwareSupporter

富含文字的图片（简称“文档图片”）在生活中无处不在！

Death rates from substance use disorders, Substance use disorders refers to direct deaths from overdoses of alcohol cocaine, amphetamines and cannabis). Death rates are measured as the I population.

industrydocuments.ucsf.edu/docs/zxfk0226

spunses veng suuicu as A，ucunsuauig thestrongerunderstandingability of mPLUG-DocOwl in diverse document scenarios.Besides,it's worth noting that all models have someresponses scored as“C”or“D”,showingthat instruction understandingperformance in the document domain is still far from promisingand needsmore endeavor.

4.2Benchmark Evaluation

Besides human evaluation,we also compare ourmPLUG-DocOwl with ocr-free state-ofthe-art document understanding models on public datasets.Table1 shows the comparison withDessurt[Davis et al., 2022], Donut [Kim

Figure 3:Human evaluation of mPLUG-DocOwl, mPLUG-Owl and MiniGPT-4on LLMDoc.

多模态文档大模型背景

Revolutionizing Document Al with Multimodal Document Foundation Models

》多模态文档大模型背景

一个通用的多模态大模型应该具备文档图片理解的能力，例如GPT4V

Prompt:

Howmany familiesare earning more than 13000 and ownsmore than2 cars?

Monthlyincome(inRs.)	Vehicles per frfamily
Monthlyincome(inRs.)	0	1	2	Above 2
Less than77000	10	160	25	0
7000-1-10000	0	305	27	2
10000-0-13000	1	535	29	1
13000-16000	2	469	59	25
16000ormore	1	579	82	88

GPT-4V:

Fromthe given table,wecanclearlysee thatthere area total of 113families thatearnmore thanRs.13o00 and own more than 2 cars (25 families in the 13000-16000 range and 88 families in the 16000 or more range).

Prompt: Describe the image.

GPT-4V:

This imageisa floor plan foranapartment or condominium.Itshows the layout of the roomsand the dimensions.Thereisaterrace,diningarea,living room, foyer,twobedrooms,amasterbathroom,andaguest bathroom.Thefloorplan isblackandwhiteand isa two-dimensional drawing.

多模态文档大模型背景

多模态大模型模型（MLLM）主要关注于一般图片的对话和问答

InstructBLIP

mPLUG-Owl

What is unusual about this image?

Score: A

What makes this scene unusual is the presence of a man ironing clothes ona busy New York City street whilea large yellow taxi drives by in the background. Usually,people would not iron clothes in such an urban environment,as it ismore common to do so in a more private or quieter location.The combination of a street scene and a man ironing his clothes in public can create a unique visual impression and draw attention to the man's unusual activity.

What is funny about this image?Describe it panel by panel.

Score:

This image is funny because it shows a person holding a cell phone and a USB cord,with asmall batteryand a USB charger plug.The person is using the USB charger to charge the phone using the battery,which is an unusual and comedic way to charge a cell phone.It is humorous because it shows the person's resourcefulness in finding ways to charge their phone without the usual charging cable or plug-in.Additionally, theperson's hand holding the phone and the battery ispositioned in a way that makes it appear as if they are trying to keep the battery and the phone together,adding to the comedic element. Overall, the image is funny due to the unusual charging method, the positioning of the hand holding the phone and the battery,and the small battery and USB charger being combined.

LLaVA

Visual input example, Chicken Nugget Map:

Sometimes ljust look at pictures of the earth from space andl marvel at how beautiful it allis.

Can you explain this meme in detail?

The meme in the image is a creative and humorous take on food, with a focus on chicken nuggets,as the center of the universe.The meme begins with the phrase "Sometimes I just look at pictures of the Earth from space and Imarvel at how beautiful it all is.." The main image features a tray of cooked chicken nuggets,with the chicken pieces positioned to resemble the continents and islands on a world map.The meme playfully suggests that the chicken nuggets represent the Earth,and the various locations depicted in the photo are actually chicken nugget versions of different places.The punchline of the meme appears in smaller text at the bottom,reading "Imean,it's not the real Earth,but how beautiful it is all is."This lighthearted and entertaining meme combines food and humor,using the image of chicken nuggets to create a funny and imaginative representation of the world.

What could have happened based on the current scene？

+ sedee caused significant damage to the buildings and infrastructure in the area. The person walking through the debris-covered area suggests that they are inspecting the damage caused by the disaster. The presence of palm trees indicates that the location may be in a tropical or sub-tropical region, which can be more susceptible to hurricanes and other natural disasters.

Can yon tel me about this image in detail?

tomatoes, and nuts arranged on a stone surface. There are several jars filled with different types of ingredients,such as peanuts,cashews,sunflower seeds, and pumpkin seeds. These ingredients are likely to be part of a healthy meal or snack.

Can you give me an example of such meal?

An example of a healthy meal or snack that incorporates some of the ingredients in the image could be a salad made with mixed greens, carrots, cucumbers, tomatoes, and nuts. The salad can be dressed with a homemade vinaigrette made with olive oil,jice,t Additionally, the salad can be topped with pumpkin seeds or sunflower seeds for added crunch and nutritional value.

Sounds good! Show me steps of making this salad.

Sure! Here are the steps to make this healthy salad: 1.Wash and chop the mixed greens, carrots,
cucumbers, and tomatoes into bite-sized pieces.
2. In a small bowl, whisk together olive oil, lemon juice, salt, and pepper to create a homemade
vinaigrette dressing.
3.Place the chopped vegetables in a large bowland tossthemwiththe vinaigretedressing.
4.Sprinkle pumpkin seeds or sunflower seeds on top of the salad for added crunch and nutritional value. 5.Serve the salad immediately or store it in the refrigeratoruntilreadytoenjoy

Itroduce me ti rsigia

Thepaintingdepictsayoung girl withapearlearrng, also known as "The Girl with the Pearl Earring or "Girl withaPearl Earring.Sheis wearing a blue dress and has a pearl earring on her left ear.Her hair is pulled back intoa bun,and sheis looking directly at the viewer. This painting is a famous work of art by Dutch painter Johannes Vermeer, created in 1665. It is considered one ofhis masterpieces and is widely recognized for its intricate detailsand realistic portrayal of the young girl's facial expression.

Canyoudeseribe tisae

Te wearing a jacket and appears to be contemplating the vastness of the universe beyond the doorway. In the background,there are multiple stars visible,creating a stunning celestial scene. The person's gaze is directed towards the center of the image, where a bright star can be seen shining brightly. There are also smaller stars scattered throughout the scene, adding to the overall cosmic atmosphere.

What if they jump over the doorway?

If the person were to jump over the doorway, they would be leaving the confines of their current location and venturing into the vastness of space. This could be a metaphorical representation of stepping out of one's comfort zone or embarking on a new adventure. However, it is important to note that such an action would be extremely dangerous and potentially lifethreateningdue to the harsh conditions of space.

What makes this dog special?

The dog in the image is wearing futuristic armor and has a glowing green eye, giving it a unique and futuristic appearance. This combination of elements makes the dog stand out from other dogs and adds an element of fantasy or sci-fi to the image. Additionally, the dog's posture and facial expression convey a sense of confidence and determination, further emphasizing its unique features.

多模态文档大模型背景

由于预训练中存在富含文字的图片，他们也初显一定的图片文字理解能力MLLM具备实现通用文档图片理解的潜力

Los Angeles Cimes

Score:

Explain why this meme is funny.

The humor in this meme comes from its playful use of words and the visual depiction of a cute,tired French bulldog lying down on its side, possibly after a long day.The phrase "just..Monday,"is commonly used to express a feeling of weariness or frustration with the start of a new week,and the dog's expression of exhaustion perfectly captures this mood.The image is visually appealingand relatable,making it a funny and entertainingway to express acommon sentiment.

Write a newscast based on the picture.

The Los Angeles Times is reporting that O.J.Simpson,the infamous American former footballplayer and actor,has been paroled from the Nevada State Prison after spending nearly nine years incarcerated fora 2oo7 armed robbery. Simpson was granted parole on Thursday by Nevada's Parole Board, which cited his positive behavior and good conduct during his time in prison.Simpson's release comes as President Donald Trump recently announced a new list of potential pardons,which included Simpson.

In the wake of the news,Simpson's longtime friend,Al Sharpton,issued a statement expressing support for the parole decision,saying that Simpson has been "rehabilitated"and deserves a second chance in life.Sharpton added that Simpson's release is a victory for "all Americans who believe in redemption and second chances."

However,not everyone is celebrating Simpson's release.Many people,including some of thevictimsofhiscrimes,have voicedopposition tohisrelease,saying that it undermines the rule of law and sends a negative message to society. Asa result of his parole,Simpson willhave several restrictions placed on his freedom,including a curfew,a ban on alcohol consumption,and the requirement to wear an electronic monitoring bracelet.Simpson willalso be required to stay in Nevada for the first year of his parole and willnot beallowed to return to California,where the murders of Nicole Brown Simpson and Ron Goldman took place in 1994.Simpson will also be required to participate in a mental health program and complete community service work. Overall, the release of O.J.Simpson has sparked controversy and divided public opinion.While some believe that Simpson has been rehabilitated and deserves a second chance,others believe that his release sends the wrong message about the importance of accountability for crimes committed.

多模态文档大模型挑战

利用MLLM进行文档图片理解的五大挑战：

·文字相关的图片在类型、形状、大小三个方面都十分多样化，已有的MLLM的视觉编码器难以编码这些图片2023.7mPLUG-DocOwl/UReader：无参数的形状适应的切图模块，EMNLP2023

·文档相关的任务形式很多样，包括信息抽取，问答和自然语言推理等，涉及文字识别和语义理解等不同层次；2023.7mPLUG-DocOwl/UReader:多任务联合学习，EMNLP2023

·文档图片在结构布局方面十分复杂多样化，其对于文档语义理解十分关键；2024.3mPLUG-DocOwl1.5：统一的结构学习

·作为文档图片的一类，Chart问答更考验模型的多步推理和数学计算的能力；2024.4TinyChart-3B：通过Program-of-Thought解决Chart领域多步推理和数学计算问题

·特殊场景（例如论文理解）下，文档图片的理解需要外部知识的支持；2023.12mPLUG-PaperOwl：基于上下文的论文图表分析，MM2024

PART01多模态文档大模型背景

mPLUG-DocOwl

DocOwl:通过轻量化训练，将已有的MLLM拓展到文档图片领域

1.文字相关的图片在类型、形状、大小三个方面都十分多样化，已有的MLLM的视觉编码器难以编码这些图片；

Ours:无参数的“形状适应的切图模块"

2.文档相关的任务形式很多样，包括信息抽取，问答和自然语言推理等，涉及文字识别和语义理解等不同层次；

Ours:多任务联合学习+辅助学习任务

YeJ,HuA,XuH,etal.mplug-docowl: Modularizedmultimodal large language model fordocumentunderstanding[J].arXivpreprintarXiv:23o7.02499,2023.

mPLUG-DocOwl

模型结构

形状适应的切图模块 1.Grids = {1x1,1x2, ... 1×20,

20×1}2.同时考虑形状和大小进行grid选择)

Large Language Model

Crop Position Encoding

Visual Abstractor

Visual Encoder

Shape-Adaptive Cropping Module

每个子图在原图中第几行第几列

Who has more active users, Pinterest or Twitter?

mPLUG-DocOwl

模型训练：训练任务

下游住务

Visual Question Answering (问答)

Information Extraction (信息抽取)

Natural Language Inference (自然语言推理)

Image Captioning (图像描述生成)

辅助任务

Text Reading (文字阅读)

Key Points Generation (要点生成)

Human:What will the presentation aim to encourageadiscussionabout AI: Itwill aim to encourage a discussion about open data ownership rights and the differing social

Human:whatis thevalue mentioned in the CID#: AI:7360000

Human:juanromanriquelme have the same score on july11,2007 november172007,YesorNo? AI: Yes

Human：Create a compact narrative representingtheimagepresented. AI:Sign that says Calea Dorobantilor inblueand white

mPLUG-DocOwl

模型训练：训练任务

辅助任务

Text Reading:增强模型基础的文字识别能力

·整个图片的文字序列按照从上到下，从左到右的形式组织，

·随机选取序列中{0,,,..,的一个位置（概率分布为{0.5,0.1,0.1,.,0.1）作为断点，

要求模型根据图片和断点前面的文字，生成断点后面的文字

·相比所有的样例都读全部文字，这种方法可以让模型更多地关注图片上不同区域的文字

KeyPointsGeneration：增强模型更高层次的语义理解能力

·对于每张图片的question-answer对，使用Vicuna将其转为陈述句的形式，即key point;

要求模型根据图片，生成多个keypoints

mPLUG-DocOwl

模型训练：训练数据

Document

DocVQA

InfoVQA

DeepForm

KLC

Table

WTQ

TabFact

WebPage VisualMRC

Chart ChartQA

Natural Image
TextVQA
TextCaps

mPLUG-DocOwl

实验结果

Table 1: Comparison with ocr-free methods on various types of visually-situated language understanding tasks. ‘TSFT’ means task-spcific fine-tuning on the downstream dataset.‘underline' means achieving 80% SOTA performance.

Model	Train Param	TS FT	Doc VQA	Info VQA	Deep Form	KLC	WTQ TabFact		ChartQA	TextVQA TextCaps	Visual MRC
Dessurt	127M	！	63.2	-	1	-				-
Donut	176M	<	67.5	11.6	61.6	30.0	18.85	54.6	41.8	43.5 74.4	93.91
Pix2Structbase	282M	<	72.1	38.2	1		-		56.0	88.0	-
Pix2Structarge	1.3B	√	76.6	40.0	-		- -	58.6	-	95.5	-
DocOw1	86M	×	65.4	42.2	49.5	32.8	29.4	67.6 59.3	57.6	118.4	221.7

mPLUG-DocOwl1.5

结构布局理解对于多模态文档理解十分关键

Figure2:Illustrations of theimportanceof structure informationin Visual Document Understanding ondocuments(a),tables (b),webpages(c),infographics (d),and charts (e-f).

mPLUG-DocOwl1.5

DocOwl1.5：通过统一的结构学习，显著提升多模态文档理解效果。

模型结构：视觉语言对齐结构HReducer

通过卷积更好的保持视觉特征的空间位置关系；

降低高分辨率视觉特征的数量；

模型训练:Unified Structure Learning

·结构感知的文字解析

多粒度文字识别和定位；

mPLUG-DocOwl1.5

延续DocOwl,采用Shape-AdaptiveCroppingModule处理高分辨率图片

mPLUG-DocOwl1.5

·不同于DocOwl,采用基于卷积的HReducer将视觉特征对齐到语言特征

考虑到文字大部分为横向排布，采用1x4的卷积

mPLUG-DocOwl1.5

不同于DocOwl采用学习的位置编码，DocOwl1.5直接采用特殊的文本token表示当前子图的位置

mPLUG-DocOwl1.5

模型训练

Stagel: Unified Structure Learning

Stage2:Multi-task Tuning

mPLUG-DocOwl1.5

模型训练

Struct-aware Parsing

·Doc/Webpage Parsing

·Table Parsing

·Chart Parsing

·Natural Image Parsing

Multi-grained Text Localization

·Text Recognition

·Text Grounding

Figure4:The ilustration of Unified Structure Learning of DocOwl1.5.

mPLUG-DocOwl1.5

模型训练

Task	\|Dataset	Num
Document Parsing	CCpdf RVL-CDIP	938,058
		159,418
	DUE VisualMRC	56,809
Table Parsing	TURL	7,016
Table Parsing	PubTabNet	200,000 200,000
Chart Parsing	ChartQA	18,317
	FigureQA	99,714
	PlotQA	157,070
Natural Image Parsing	DQA	200,000
Multi-Grained Text Localization	\|OCR-CC	1,000,000
	Word	114,344
	Phrase Line	113,130 378,432
Block Total		394,094 4,036,402

Figure 5:Detailed statistics of DocStruct4M.

mPLUG-DocOwl1.5

Table1:Thedetailed statistics of DocReason25K.The‘Avg Length'refers to the average token length of the answer.

	DocVQA	InfoVQA	WTQ	VisuaIMRC	ChartQA	TextVQA	ALL
Image	1,491	1,614	850	1,927	1,252	1,612	8,746
Sample	5,119	5,421	5,994	5,263	1,827	2,253	25,877
AvgLength	79.2	95.4	77.7	103.4	106.9	88.0	89.9

已有下游任务只提供简单的词组作为答案，并不能给出详细的解释

DocReason25K：基于GPT3.5/GPT4V构建的多模态文档领域关于详细解释的微调数据集DocOwl1.5-Chat:联合已有下游任务和DocReason25K进行第二阶段训练。

mPLUG-DocOwl1.5

Table2: Different settings of OCR-free Visual Document Understanding models.‘Open’refers to whether all OCR learning data is open-source.

Model	Init	Resolution	OCRLearning				Open
Model	Init	Resolution	Text	Bbox	Size	Domain	Open
Donut 四		2560x1920	√	×	13M	Synthetic,Doc	√
Pix2Struct24]		219(shape variable)		×	80M	Web	×
QwenVL[]		448x448		×	24.8M	Synthetic,Doc,Web	×
Monkey 四	QwenVL □	896x896		×			=
UReader[57]	Owl [59]	224x224（x20 crops）	>>x>	×	0.1M	Doc,Table,Chart,Web,Natural	√
DocPedia[]	=	2560×2560	√	√	0.9M	Doc	X
CogAgent [16]	CogVLM 四	1120×1120	√	√	107M	Synthetic,Nature,Doc,Web	×
DocOwl 1.5	Owl2 [59]	448x448(x9 crops)	√	√	4M	Doc,Table,Chart,Web,Natural	√

相比于当前SOTA文档理解大模型：

·DocOwl1.5同时进行了文字、结构以及位置的学习

·DocOwl1.5覆盖了更多的图片类型

·DocOwl1.5只利用开源数据，没有利用私有化数据，易复现

mPLUG-DocOwl1.5

Table 3: Comparison with OCR-free methods on various types of text-rich image understanding tasks.The superscript‘*’refers to models separately fine-tuned on each downstream task,rather than generalists. The underline means the best performance among models with <1OB parameters.

Model	Size	Doc VQA	Info VQA	Deep Form	KLC	WTQ	Tab Fact	Chart QA	Text VQA	Text Caps	Visual MRC
Model	Size	Doc VQA	Info VQA	Deep Form	KLC	WTQ	Tab Fact	Chart QA	Text VQA	Text Caps	Visual MRC
Dessurt	<1B	63.2	-	-	1	-	-	-	-	-	-
Donut	<1B	67.5	11.6	61.6	30.0	18.8	54.6	41.8	43.5	74.4	93.91
Pix2Structbase	<1B	72.1	38.2	-	-	-	-	56.0	-	88.0	-
Pix2Structarge	1.3B	76.6	40.0	-	-	-	-	58.6	-	95.5	-
DocPeida	7.0B	47.1	15.2	-	-	-	-	46.9	60.2
DocOw1	7.1B	62.2	38.2	42.6	30.3	26.9	60.2	57.4	52.6	111.9	188.8
QwenVL	9.6B	65.1	35.4	-	-	-	-	65.7	63.8	-	-
UReader	7.1B	65.4	42.2	49.5	32.8	29.4	67.6	59.3	57.6	118.4	221.7
Monkey	9.8B	66.5	36.1	40.6	32.8	25.3	-	-	67.6	93.2	-
CogAgent	17.3B	81.6	44.5	-	-	-	-	68.4	76.1	-	-
DocOwl-1.5	8.1B	81.6	50.4	68.8	37.9	39.8	80.4	70.5	68.8	132.0	239.5
DocOwl-1.5-Chat	8.1B	82.2	50.7	68.8	38.7	40.6	80.2	70.2	68.6	131.6	246.4

mPLUG-DocOwl1.5

样例：文档解析

Tropical Birding Trip Report NAMBIA NOY DEC 305
Tg	rat
Seen near Kayova. Chiping Otic	Ol ppint
Sen atKayoe Piping Csticola	Osticnle fulvicapille
Sen sear Kayoa 2ng Cstioola	Ostle juncdis
Seenat EDtosha & Windhoek. Desert Citicea	Osticole arilves
Seen at Etos.
ed WabsdAltescocepd) icterine Warbier	Hippolais icterine
Sn t os Windhoek Sedge Warbler	Acraorphores schoetoboret
Seen t ayu can Reed Warber	Acrarphelus berticatui
Seent WasBay& Winhoek
Lser SwanpWarbler Seen at Kayose& Windhoek.	Acrorphs grs
Sealows (Hirndinidae) Pain Martin	Riparia paludicole
Seen at Kayos.

	Aiporta rperie

Bank Swalow




Seen at Cap

































	+1-409-515-9110

DocOwl 1.5:

Extract words from the graphic.

Tropical BirdingTripReport NAMIBIA NOV DEC2021
TinklingCisticola Cisticola rufilatus
Seen near Kayova.
Chirping Cisticola Cisticola pipiens
SeenatKayova.
Piping Cisticola Cisticola fulvicapilla
Seen near Kayova.
Zitting Cisticola Cisticolajuncidis
Seenat Etosha&Windhoek.
Desert Cisticola Cistcola aridulus
Seen at Etosha.
ReedWarblers and Allies (Acrocephalidac)
Icterine Warbler Hippolaisicterina
Seen at Etosha & Windhoek.
Sedge Warbler Acrocephalus schoenobaenus
Seenat Kayova.
African Reed Warbler Acrocephalus baeticus
Seen at Walvis Bay& Windhoek.
LesserSwamp Warbler Acrocephalus gracilirostris
Seenat Kayova& Windhoek.
Swallows(Hirundinidae)
Plain Martin Riparia paludicola
Seen atKayova.
Bank Swallow Ripariariparia
Seen at Caprivi.
www.tropicalbirding.com +1-409-515-9110 info@tropicalbirding.com

mPLUG-DocOwl1.5

样例：图表解析

	N(%)*		Crude results (n= 61,129)		Adjusted results** (n = 47,817)
LANexposure	Cases	Referents	Odds ratios	95% CI	Odds ratios	95% C
Low	27,121 (63)	10,970 (60)	Ref.		Ref
Medium	5,974 (14)	2,623 (14)	1.13	(1.07-1.19)	1.06	(0.97-1.16)
High	9,659 (23)	4,782 (26)	1.22	(1,18-1.28)	1.12	(1.04-1.20)”

DocOwl 1.5:

Convertthefigureinto Markdownlanguage

||N(%)**|Cruderesults(n=61,129)|Adjusted results**（n=47,817)|

LAN exposure	Cases	Referents	Odds ratios	95% CI	Odds ratios	95%CI
Low	27,121（63）	10,970（60）		Ref.		Ref.
Medium	5,974(14)	2,623(14)	1.13	(1.07-1.19)	1.06	(0.97-1.16)
High	9,659(23)	4,782(26)	1.22	(1.18-1.28)	1.12	(1.04-1.20)t

Render the figure in Markdown format.

DocOwl 1.5:

mPLUG-DocOwl1.5

样例：自然图解析

(NIMARESTAURNT&BARPAROTOWN Specialistin SHUTNISDI SMPSTAIRS)

(PEMA DORA GENERAL SHOP)

alamy stock photo

(a alamy stock photo E7BDR1 www.alamy.com)

Detallthe Informatlon and textual components depictedin the image.

DocOwl 1.5:

the main strect in the toWn a a a alamy NIMA RESTAURANT & BAR SpeCial TOURS & DEL PSTAIRS PERIA DORA GENERAL SHOP SONAR ENTERPRISE a a a a aa alamy photo stock E7BDR1 a www.alamy.com

mPLUG-DocOwl1.5

样例：多粒度文字识别

Draw	Artist	Song	Points	Place
1	The Memerles	MeanLosing You*	57	Sth
2	AmBrees	"Oh, Darting*	80	4th
3	Fran Meen	“Say That You Love Me*	66	6th
4	Dreams	Si Sin(Tat's Tat)*	73	5th
5	Connor Sevens	"Count OnMe'	88	3re
6	Unda Martin and Friends	“All Te People in Te Worle*	105	2n6
7	Maggle Toal	“Teed Hirs Wit Love”	61	7th
8	UamReilly	"Somewhere in Eurooe”	130	1st

mPLUG-DocOwl1.5

样例：多粒度文字定位

3 cE 0t x王04.9SHOE 2	38注:1:183	a2					1+r+r
LSRM		zeu	生
241	m	Ye	生金	E2	EMMOHN	EIEF

Ludwig Witgenstein (edt]

Bo 1889, died 1951.An Austan bon Cambridgescholar, Wigenstein wasinvoved in his eary yes with theLogicalPostivists ofthe Vienna Cicle WigensteinlPosophyoa ilebtaspsadedtoreuinthosgeeinsetpromtedadetayofookngatiios lngage,heolongersoughtto provetherlangagereesbuookedateyin wichlanguagerefestohings He decidedthatthe best way to determinea words meaning was tolook athowitis used,andnot to come up witha theoretical definition. Witgenstein said this wasabout: Geting back to the rough ground.

Detect the text in the bounding box 14,532,981,850

DocOwl 1.5:

Ludwig Wittgenstein [edit]
Bo189dd195.AAusraboCmbdgesolWigesesivdinsarlyyaswithhgical
Positivists of the Vienna Circle.n
Witgesteilefsofoailetssadtoint92sWitsseppedty
oflooking at religious
language,heologersoughtoprove"whetherangagerefersbutlookedatthewayinichlanguagereferstotings
Hedecidedthat the best wayto determinea word'smeaning wastolook athowitisused,andnotto comeupwithatheoretical
definition. Wittgenstein
saidthis wasabout:"Getingback to therough ground".

mPLUG-DocOwl1.5

样例：详细解释的问答

Whats the vale of the Very well bar in the 65+ age group?Answer the question with detailed explanation

Itheimage,whichappearstobeachartfromthePew ResearchCenterthe bar wellfortheagegrop65andelderismaked 68%.This 68%efRepubsdebicalasseoup
believethatTpisfightingortherbeliefsVrywel.

what is written on the blue shirt on the left? Answer the qestion wih detaied explanation.

DocOw1 1.5-Chat:

Fly Emirates. The blue shirton the lefthas the word “Fly Emirates" esheetTing against a blue background,which is the color scheme typicallyusedfor the airline's branding.

What is the SIS #efL. Answer the tion with detaled

DocOwl 1.5-Chat:

SIS#158120. The SIS # ofL.MacWlliams is 15820,ssd i the text under the section L. MacWiiams SIS # DireetAeeout NamelnfoationRequredPO

What percentage of teagars fro the age group12-17 didn't use the Ime? Aser the qeston with detailed explanation

DecOwl 1.5-Chat

Accoding te the text,95% ofteegers ages12-17e teletThereore he prnge of eea othe egoup 12-17 who didn'ehe eet would be 100%-95%=5%

Bute	TH	Opperest"	F	Hine	TV	Rest	anendancs
Septander 1	2.30PM	FC	#12	Camp dal Stadue·Masan.W	ITN	W-21	79.560
Septanteri	3:60PM	OegenStt	#13	C R	Fx	L7-10	40.189
Septenter 11	7:00PH	Uiah Sute"	#22	Cag dal ad-ds,W	T	W1624	70,332
Sepater22	11:00AM	D	*24	Camg adal Stadum Madse,W	(SPN2	W17-26	79.806
Septanter 24		7:00PM #2 b	*23	d LM	业	L21-30	85.962
Ceiober6	2:30PM	llinos		Camg ardal Sadum + Madson,W	ABOEUNT	W31-14	80.096
ber13		11AN		osde SeLe	WTN	W38-34	46.007
90r29		11O0AN Heda		Carg dal adum -adsn,W aul yas et		W13	80,387
Ctober27	2:30 PM	Hitigan Staet	5	Cag del Sdu daW	AIOISPN2	L13-2601	80,538
Nonber15		11.00AM		Saun·ongoN	ISPS:	W-24	41.240
Novinter17	2.30PM	Oheate		Ca dal SdudW	AICESPNZ	L14-21O7	312
ne24	2.30PM			sy k	ESPN2	L21-24T	91.345
Oeonber1	7.00PM	V4b		gT	FOX	W70-31	41.290
1.20401s			t e Ai oe ee e	2e C	5	L14-20	93.259

whatithetdteadtmecetabe?Aswerteestiowhdetaildepion

DecOwl 1.5-Chat:

Thelstdateandtimeothechatisuary12013at40PThechtlistsallthegmesplayedbyerinteanalong with 2013at4:10PTrefothelst daeandieon thechtisry1,2013a4:10PM.

TinyChart

图表（Chart）理解的主要挑战：

问答经常涉及多步数学推理

现有最优Chart多模态模型参数太大，推理慢(ChartLLama-13B,ChartAssistant-13B)

College completion by race and ethnicity

Question:What is the difference between
Asiansand Whites degree distribution?
GTAnswer:22
ChartLlama:21
TinyChart Direct:22

TinyChart PoT:22

Get the values of all

'Values'inthechart，set
toY
Y=[15,22,41,63]

Getthe difference

between the value of
Asiansand Whites,setto
Subtract
Subtract=np.subtract(Y[3],
Y[2])

Calculate the absolute

valueof Subtract，set to
Answer
Answer=np.abs(Subtract)

ZhangL，HuAXuH，etal.TinyChartEficientChartUnderstandingwithVisualTokenMergingandProgram-of-ThoughtsLearning[J].arXivpreprint arXiv:2404.16635，2024.

TinyChart

Figure2:Overview of TinyChart.

ZhangL,HuAXu H,etal.TinyChartEficient Chart Understanding withVisual Token MergingandProgram-of-ThoughtsLearning[J].arXivpreprint arXiv:2404.16635，2024.

TinyChart

1	4	7
2	5	8
3	6	9

(a)

(b)
Figure3:(a)Visiontransformerlayerwith VisualTokenMerging.(b)Processof theVisualTokenMerging.

TinyChart

Figure4:ThedemonstrationofconstructingTemplate-basedPoT(upper half)andGPT-basedPoT(lowerhalf)intheChartQA-PoTdataset.

TinyChart

Table1:Datasetsused fortrainingTinyChart.Thebenchmarkdatasetsconsist ofbasicchart understandingevaluationsincludingQA,summary,andchart-to-tablegeneration. Notethatinablationstudies,weonlyusethebenchmark datasetsfortrainingduetolimitedcomputationalresources.

Dataset	Benchmark	Samples
Chartquestionanswer
ChartQA [32]	>	28,299
ChartQA-PoT		140,584
PlotQA[36]		157,070
DVQA[19]		200,000
OpenCQA[20]		5,407
Chart-to-text generation
Pew[21]	>	7,892
Statista [21]		29,589
OpenCQA[20]		5,407
Vistext [45]		11,171
ChartSumm [42]		75,255
Chart2Text-8k [37]		7,862
Chart-to-tablegeneration
ChartQA[32]	/	19,373
PlotQA [36]		190,720
Chart2Text-8k		8,305
DVQA[19]		300,000
Statista [21]		29,589
Chartinstructionfollowing
ChartLlama [10]		148,398
Total		1,364,921

TinyChart

Table2:Main resultsonchart-related benchmarks.The inference throughput is evaluated on theChartQAtest withabatch size of 1on V10032GB.

Model	#Parameters	Resolution	Inference Throughput	ChartQA			Chart-to-Text	Chart-to-Table	OpenCQA
Model	#Parameters	Resolution	Inference Throughput	Aug.	Hum.	Avg.	BLEU4	RMSF1	BLEU4
Close source models
GPT-4V[39]						78.50
Gemini-Ultra [46]						80.80
Qwen-VL-Max [1]	-			-	=	79.80		-
Deplot+Codex[26]	1.3B+175B			91.00	67.60	79.30		87.22
Open source models
Llava1.5[29]	13B	336×336	1.94it/s	72.96	37.68	55.32	7.16	48.95	=
Qwen-VL[1]	9.6B	448×448	1.65it/s	78.90	44.30	61.60	=	-
UReader[52]	7B	224×224（×20)	1.67it/s	79.42	39.12	59.30	-	-	=
DocOwl1.5[15]	8B	448×448（×9)	1.56it/s	91.38	49.62	70.50	-	-	-
ChartInstruct [34]	7B	=	-	87.76	45.52	66.64	13.83	-	15.59
ChartLlama[10]	13B	336×336	1.94 it/s	90.36	48.96	69.66	14.23	90.00	=
ChartAst [35]	13B	448×448	1.47 it/s	93.90	65.90	79.90	15.50	91.60	15.50
TinyChart@512	3B	512×512	3.65it/s	93.60	72.16	82.88	17.93	92.93	19.62
TinyChart@768	3B	768×768	3.14it/s	93.86	73.34	83.60	17.18	93.78	20.39

TinyChart

Question: What is land use per 10 kilocalories by Mutton& Goat Meat production? TinyChart Direct: 20.98 (a)

Input: Generate underlying data table for the chart.TinyChart:

Characteristic Male | Female September 2018 | 1.6% | 2.2% August 2018 |1.5% ↓ 1.8% July 2018 |1.2%|1.7% June 2018 |1.5%|2.1% May 2018 |1.3% |2.4% April 2018 | 1.7%|2.3% March 2018 |1.9% |2.5% February 2018 | 2.3% | 3.2% Janary 2018 | 2.6% |3.6% December 2017 | 1.9% | 2.8% November 2017|1.4%2.3% October 2017 | 1.4%|2.1%

College completion by raceand ethnicity

Question: What is the diference between Asians and Whites degree distribution? TinyChart Direet:22

TinyChart PoT: 22

Get the values of all

'Values'in the chart,set to
Y=[15,22,41,63]

Get the difference between

the value of Asians and Whites,
set to Subtract
Subtract=np.subtract(Y[3],
Y[2]）
Calculate the absolute value
of Subtract,set to Answer
Answer=np.abs(Subtract)

Input:Generate underlying data table for the chart. TinyChart:

ZhangLHuAXuHetal.TnyChartEicientChartUnderstandingwithVisualTokenMergingandProgramof-ToughtsLearing[J].arXvpreprint arXiv:2404.16635，2024.

TinyChart

Input: Create a briefsummarization or extract key insights based on the chart image.

TinyChart: The chart presents the popularity of various sports in the United States in 2023. Footballis the most popular sport with a 30% popularity rating, followed by Basketball at 25%.Baseballand Hockey have a 15% and 10% popularity respectively. Meanwhile, Golf and Tennis have a 5% and 10% popularity rating respectively. The least popular sport is “Other" witha 5% rating.

Input: Create a briefsummarization or extract key insights based on the chart image.

TinyChart: The chart illustrates the sales trends for online and store sales from 2018 to 2022. It shows that online sales steadily increased from 500 in 2018 to 1200 in 2022, while store sales also experienced an increase from 400 in 2018 to 1000 in 2022. The data indicates that while online sales growth has been consistently positive, store sales growth has been relatively slower. This could suggest a shift in consumer preference or a need for amore extensive rctail network.

Input: Redraw the chart with Python code. TinyChart:

Input: Redraw the chart with Python code.

TinyChart:

ZhangL,HuAXuHetal.TnyChartEficientChartUnderstandingwith VisualTokenMergingandProgram-of-ThoughtsLearning[J].arXivpreprint arXiv:2404.16635,2024.

mPLUG-PaperOwl

论文图表理解的主要挑战：

1.需要专业领域知识的辅助

2.需要具备多图表关联分析能力

Figure 4: Tap-10 vbead cd nos d y eeti lge del

Table1:Top-10 verbegmeedbyLLaMA-baed odl that doot everap with GPT-4 and ChatGPT.

Recommend somekey pints to discussnext about Figure4 and Tablc1

How about:

1.The word preference ofdifferent models
2.SAIL-7B generatesmore verbs

Write a analvsis about Figure4 and Table 1 following the outline: 1.Compare the word preference of different models on unseen instructions. 2.search augmentation helps generates more verbs not overlap with GPT. 3.Theinfluence of grounding search results.

Wefirst show the word preference of diferent models on the 80 unseen instructions.The resultsare shown in Figure4.We compare the distributionsof top-10 verbs generated by GPT4, GPT-3.5-Turbo (ChatGPT), Vicuna-7B-v1.1,and SAIL-7B models.With search augmentation,SAIL-7B generates significantlymore verbs that do not overlap with GPT's generations,as shownin Table1.Only two top-10 verbs generated by Vicuna are not covered by GPT-4 and ChatGPT, while six out of ten verbs generated by SAIL-7b are not highfrequencyverbs by theGPTmodels.This indicatesthatthe grounding search results can shift the generation preference of the language models.

AHu,YShi,HXuetal.mplug-paperowlScientificdiagramanalysiswiththemultimodallargelanguagemodel[J]arXivreprint arXiv:2311.1824862023

mPLUG-PaperOwl

Figure2.The pipeline ofM-Paper construction and definition of our three tasks.

·提供上下文(context)作为专业知识来源，

*基于GPT3.5,构建“要点"(outline)作为图表分析的控制信号

M-Paper:48k papers,447k diagrams,702k training samples

text-only1diagram2diagrams3diagrams4diagrams

mPLUG-PaperOwl

Figure 6: The overall architecture of PaperOwl.

mPLUG-PaperOwl

[Diagrams]

figure 8 poposed SC-GNN Model

GT: In contrast, tbe baseline TISER-GCN model (Fig, 8) presents a mean difference of 0.67 and limits of agreement of [-1.09, 2.43]. The increased mean difference compared to the SC-GNN model suggests that the TISER-GCN model predictions are lessaccurate.Additionally, the widerlimits ofagreement indicate ahigherlevelof variabilityinthemodel’s performance

PaperOwl The TISER-GCN model (Fig.8)shows a mean difference of 0.60 and limits of agreement of[-1.10, 2.10]. This indicates that theTISER-GCNmodel predictionsare,onaverage,less accurate thanthe SC-GNN model predictions.The widerange of the limits of agreement suggests that the model’s performance is inconsistent across the range of earthquake intensities.

[Diagrams]

Method	Backbone	mloU	VC8	VC16
Mask2Former	Swin-L	0.5709	0.8876	0.8596
Mask2Former	BEiT-L	0.5854	0.8964	0.8611
Mask2Former	VIT-Adapter-L	0.6140	0.9007	0.8638

Table 1

GT: Experimenfs of diferent buckbones on PVUW2023 challenge test part 1

PaperOwl

: The comperison cfdiferent beckbones on the PVUw20z3 validetion set

[Diagrems]

\begin(toole)itj
lcentering oegintobutovilee) ltoprute 1of Model & FD \$]dowsarrow\$ & Chor-F2SlaporrowS & F-ACC\$(uporrcw5 V |mldrule ACM-V5G & 15.36 & 45.71 & 22.62 (1 qued - Gu/dance & 15.95 & 44.56 & 22.13 } lquod - Attent/on & 16.88 & 44.27 & 20.25 1) loottomrule lendjtobular)
Vcopt/anG
endjtable) Table 4

PART03mPLUG-DocOwl开源实战

mPLUG-DocOwl开源

数据和模型在ModelScope和HugginaFace上均可下载

https://github.com/X-PLUG/mPLUG-DocOwl

PLUG-DocOtModued Multimodal Large Languege Model for Document Understanding
de che-2.0 Aevi 7 2 Y leport/eposiory
Releases

Packages e y a

The Powerful Multi-modal LLM Family forOCR-free Document Understanding

mPLUG-DocOwl Demo

https://huggingface.co/spaces/mPLUG/DocOwl

https://modelscope.cn/studios/iic/mPLuG-DocOwl

https://hugginaface.co/spaces/mPLUG/TinyChart-3B

TinyChart:EficientChartUnderstandingwithVisualToken MergingandProgram-of-ThoughtsLearning dde

1.etepe
2.Teofoeepdit

通义mPLUG多模态体系

ModelScopeGithub开源

30+多/单模态任务 SOTA

平台化业务应用

图文预训练模型(mPLUG)
提出高效的模态对齐融合新范式Skip-connectedNetwork进行跨层/跨语义融合，CVPR2021VQAChallenge排名第一，并以81.26分的成绩首次超越人类结果，mPLUG，E2E-VLP相关论文被ACL2021Oral，EMNLP2022，TIOS接收。

多模态对话大模型（mPLUG-Owl)将模块化轻量化将视觉表征与大语言模型结合，解锁大语言模型的多模态能力，多个，作为多模态LLM通用对比基准；提出mPLUG-Owl2，通过模态协同同时提升纯文本和多模态性能，超过LLaVA1.5，MiniGPT4等模型，取得多种模态多种任务SOTA，CVPR2024接收。

ALImPLUG多模态深度语言模型体系模块化大一统模型 (mPLUG-2)层次模块化提升单模态和多模态任务水平，30+多/单模态图文视频任务，同等数据量和模型规模SOTA效果，在VideoQA和VideoCaption上超越Flamingo、VideoCoca、GITv2等超大模型取得绝对SOTA，ICML2023接收；

多模态文档大模型（mPLUG-DocOwl)模块化轻量化提升各类文档图表理解效果，取得多个Doc任务OCR-FreeSOTA，论文EMNLP2023接收，提出首个论文大模型mPLUG-PaperOwl，提出文档结构增强的mPLUG-DocOwl1.5，在10个文档理解benchmark上达到最优效果，5个数据集上提升超过10个点；

个性化多模态模型 (Character-Owl)多模态个性化理解生成统一大模型，可以进行轻量化的深度的个性化定制，具备深度个性化情感多模态能力，CCLDemo演示，mPLUG-Octopus多模态全能助手ACMMMdemo，提名BestDemo，应用到星尘上的多模态用户发图和角色发图业务，云栖大会发布。

多模态智能体 (mPLUG-Agent)通过纯视觉感知方案操作手机的智能体Mobile-Agent，高效的进行Self-Planing和Self-Reflection，不需要依赖XML、系统metadata、系统适配，ICLR2024AgentWorkshop接收。

端到端图文模型-E2E-VLP 结构化模型StructuralLM ACL2021OraI

多模态预训练加速mPLUG-TRIPSEMNLP2022

多模态大一统模型-mPLUG-2ICML2023

中文最大视频数据集-Youku-mPLUG视频对话大模型-mPLUG-video

2021.08

2022.12

2023.04

2023.07

2024.01

2021.01

2022.05

2023.01

2023.06

2023.11

2024.03

理解生成统一图文模型-mPLUG CVPR2021VQAChallenge The First Prize Achieving Human Parity on VQA TOIS 2022

理解生成统一视频文本模型 mPLUG-HiTeA，TWBERT ICCV2023

多模态对话大模型-mPLUG-Owl多模态Denoise-mPLUG-XXM,ACL2023Oral

多模态文档大模型-mPLUG-DocOwl图文预训练加速BUSICCV2023

多模态智能体Mobile-Agent

通义mPLUG多模态体系

MobileAgent Public Mobile-Agent:The Powerful Mobile Device Operation Assistant Family android agent harmony ios app gui automation mobile copilot multimodal +5 Python ·亚 MiT License ·179·☆ 2.2k ·16·80 · Updated 15 minutes ago

mPLUG-DocOwl Public

mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding
multimodal table-understanding document-understanding mllm multimodal-large-language-models chart-understanding
● Python · Apache License 2.0 · 64 ·☆ 1.1k · 27 · 1 · Updated 2 weeks ago

Rolelnteract Public

Rolelnteract: Evaluating the Social Interaction of Role-Playing Agents Python ·4·☆34·1· 0· Updated 3weeks ago

Multi-LLM-Agent Public Python · 20 ·☆156·4·0·Updated on Apr 23

mPLUG-Owl Public

mPLUG-Owl & mPLUG-Owl2: Modularized Multimodal Large Language Model

video chatbot dialogue pytorch transformer llama gpt alpaca visual-recognition multimodal +10
●Python ·4 MIT License· 158 ·☆ 2k ·77 · &1 · Updated on Apr6

SocialBench Public
SocialBench
亚Apache License 2.0· 0·☆1 ·O0 ·0 · Updated on Feb 15

mPLUG-HalOwl Public

mPLUG-HalOwl: Multimodal Hallucination Evaluationand Mitigating

benchmark contrastive-learning hallucinations mlm multimodal-large-language-modelsmultimodal-hallucination

Python·4& MIT License·1·☆ 59·3·0·Updated on Jan 29

#

https://github.com/orgs/X-PLUG/repositories

通义mPLUG多模态体系

Modelscope

首页

数据集

创空间

文档中心

社区

GitHub

司南评测 NEW

Qmplug

Q搜索你感兴趣的内容

综合（54）

创空间（4）

模型库（16)

数据集（1)

创空间（4个搜索结果）查看更多>

动态（2)

文档（3）

帖子（28）

通义实验室
通义实验室

DocOwI1.5:不依赖OCR的8B最优多模态文档理解大模型。可以识别包含图表、表格、海报、扫描件、网页截图、自然.

mPLUG-Owl2是达摩院研发的多模态大规模语言模型(Multi-modal Large Language Model)， mPLUG-Owl2可以以图像...

wangjunyang

Mobile-Agent是一个自动化移动设备操作智能体。只需根据设备屏幕截图和用户指令，就能自动完成一系列操作，帮..

Mobile-AgentMobile-Agent是一个自动化移动设备操作智能体。只需根据设备屏幕截图和用户指令，就能自动完成一系列操作，..

多模态

mPLUG

mPLUG-DocOwl 1.5

多模态

2024.06.12更新

2.6w84

2024.06.12更新

5.7k25

计算机视觉

2024.05.15更新

多模态

2.6k12

2024.02.26更新

790

模型库（16个搜索结果）查看更多>

已认证

mplug_visual-question-answering_coco_large_en

mPLUG预训练模型-英文-large已认证 mplug_backbone_large_en

视觉问答

PyTorch

mplug

开源协议：Apache License 2.0

multi-modal

accuracy

视觉问答

PyTorch

mplug

开源协议：Apache License 2.0

multi-modal

CIDEr

本任务是mPLUG，在英文VQA数据集进行finetune的视觉问答下游任务。给定一个问题和图片，通过图片信息来给出答案。

mPLUG英文预训练Large模型

通义实验室

2022.12.12更新

11.5w

通义实验室

2022.12.10更新

302

mPLUG图像描述模型-英文-large&已认证

mplug_image-captioning_coco_large_en

已认证

图像描述

PyTorch

mplug

开源协议：Apache License 2.0

multi-modal

CIDEr

达摩MPLUG英文图像描述large模型

图像描述

通义实验室丨2022.12.12更新丨6.3w|18

PyTorch

mplug

开源协议：Apache License 2.0

multi-modal

达摩MPLUG英文图像描述base模型

CIDEr

通义实验室丨2022.12.12更新丨6.3w丨15

免费领算登录/注册

https://modelscope.cn/search?search=mplug

PART 04 总结与展望

总结与展望

多模态文档大模型展示了Promising效果，但仍然存在通用性、鲁棒性、效率等问题，仍然还有很多提升的空间；

针对高分辨率，多页，探索更高效的视觉编码，图文融合结构；

增加更多Domain数据，探索通用的、鲁棒的多模态文档大模型；

·大小模型协同，能够结合OCR、Chart等专业小模型；

2024AI+研发数字峰会 Al+ Development Digital summit AI驱动研发变革 促进企业降本增效

多模态文档大模型mPLUG-DocOwl

演讲嘉宾

徐海洋

目录

PART01多模态文档大模型背景

BUDGETREQUEST SL

aboutcampaigns licensing membership resources community donate shop

Intern at the FSF by Free Software Foundation

All applicants must be:

富含文字的图片 （简称“文档图片”）在生活中无处不在！

4.2Benchmark Evaluation

多模态文档大模型背景

》多模态文档大模型背景

一个通用的多模态大模型应该具备文档图片理解的能力，例如GPT4V

Prompt:

GPT-4V:

GPT-4V:

多模态文档大模型背景

多模态大模型模型（MLLM）主要关注于一般图片的对话和问答

InstructBLIP

mPLUG-Owl

Score: A

Score:

LLaVA

Visual input example, Chicken Nugget Map:

Can you explain this meme in detail?

What could have happened based on the current scene？

Can yon tel me about this image in detail?

Can you give me an example of such meal?

Sounds good! Show me steps of making this salad.

Itroduce me ti rsigia

Canyoudeseribe tisae

What if they jump over the doorway?

What makes this dog special?

多模态文档大模型背景

由于预训练中存在富含文字的图片，他们也初显一定的图片文字理解能力MLLM具备实现通用文档图片理解的潜力

Los Angeles Cimes

Score:

多模态文档大模型挑战

利用MLLM进行文档图片理解的五大挑战：

PART01多模态文档大模型背景

mPLUG-DocOwl

DocOwl:通过轻量化训练，将已有的MLLM拓展到文档图片领域

mPLUG-DocOwl

模型结构

mPLUG-DocOwl

模型训练：训练任务

下游住务

辅助任务

mPLUG-DocOwl

模型训练：训练任务

辅助任务

Text Reading:增强模型基础的文字识别能力

KeyPointsGeneration：增强模型更高层次的语义理解能力

mPLUG-DocOwl

mPLUG-DocOwl

实验结果

mPLUG-DocOwl1.5

结构布局理解对于多模态文档理解十分关键

mPLUG-DocOwl1.5

DocOwl1.5：通过统一的结构学习，显著提升多模态文档理解效果。

模型结构：视觉语言对齐结构HReducer

模型训练:Unified Structure Learning

mPLUG-DocOwl1.5

mPLUG-DocOwl1.5

mPLUG-DocOwl1.5

mPLUG-DocOwl1.5

mPLUG-DocOwl1.5

模型训练

mPLUG-DocOwl1.5

模型训练

Struct-aware Parsing

Multi-grained Text Localization

mPLUG-DocOwl1.5

模型训练

mPLUG-DocOwl1.5

mPLUG-DocOwl1.5

相比于当前SOTA文档理解大模型：

mPLUG-DocOwl1.5

2024AI+研发数字峰会 Al+ Development Digital summit AI驱动研发变革促进企业降本增效

富含文字的图片（简称“文档图片”）在生活中无处不在！

创空间（4个搜索结果）查看更多>

模型库（16个搜索结果）查看更多>

AiDD-人工智能行业2024AI+研发数字峰会：多模态文档大模型mPLUG-DocOwl-250320.pdf