2024AI+研发数字峰会 Al+ Development Digital summit AI驱动研发变革促进企业降本增效

北京站 08/16-17

基于 eBPF 和 Agent 构建LLM 训练推理优化体系

向阳云杉网络

演讲嘉宾

向阳

清华大学博士 / 云杉网络研发 VP

清华大学博士，云杉网络研发 VP，DeepFlow 开源社区负责人。曾在国际顶级学术会议 ACM SIGCOMM、ACM IMC 上发表可观测性方向的学术论文，现负责可观测性产品 DeepFlow，致力于打造一款为云原生和AI 应用而生的零侵扰可观测性产品。

目录 CONTENTS

背景：训练和推理的效率挑战

现状：传统解决方案和工具的问题
方法：eBPF 构建零侵扰可观测性
实践：PyTorch 全栈剖析和追踪
探索：Agent 自动优化 ML 代码

PART 01背景：训练和推理的效率挑战

LLM 训练开销大、效率低

GPT-4	Llama-3.1
参数 1.8T	405B
GPU 25K A100	16K H100
时长 90~100天	54天
MFU 32%~36%	38%~43%

Everything We Know About GPT-4 - Klu.ai

GPT4- All Details Leaked

The Llama 3 Herd of Models

训练时间长：数月

GPU 数量多：数万

模型参数大：万亿

GPU 利用率低：40%

Component	Category	Interruption Count	% of Interruptions
Faulty GPU	GPU	148	30.1%
GPU HBM3 Memory	GPU	72	17.2%
Software Bug	Dependency	54	12.9%
Network Switch/Cable	Network	35	8.4%
Host Maintenance	Unplanned Maintenance	32	7.6%
GPU SRAM Memory	GPU	19	4.5%
GPU: System Processor	GPU	17	4.1%
NIC	Host	7	1.7%
NCCL Watchdog Timeouts	Unknown	7	1.7%
Silent Data Corruption	GPU	6	1.4%
GPU Thermal Interface + Sensor	GPU	6	1.4%
SSD	Host	3	0.7%
Power Supply	Host	3	0.7%
Server Chassis	Host		0.5%
IO Expansion Board	Host		0.5%
Dependency	Dependency	２22	0.5%
CPU	Host	２	0.5%
System Memory	Host	2	0.5%

Table 5 Root-cause categorization of unexpected interruptions during a 54-day period of Llama 3 405B pre-training. About 78% of unexpected interruptions were atibuted to confirmed or suspected hardware issues.

GPU 年化故障高：6%\~11%

代码层面训练低效的主要原因

Classification of the 706 low-GPU-utilization issues discovered across 40o deep learning jobs.

Category	Description	No.	Ratio
Interactive Job	The execution of ajob entails regular interaction with its owner.	15	2.12%
GPU Oversubscription	A job requests more GPUs than it actually utilizes.	6	0.85%
Unreleased Job	A job does not terminate promptly after completing its computation.	9	1.27%
Non-DL Job	A job is unrelated to deep learning and does not utilize GPUs at all. For example, the job performs data analysis using CPUs solely.	4	0.57%

Improper Batch Size	Improper values of thebatch size areused,which decrease the GPU computation ofdeep learning operators.	181	25.64%
Insufficient GPU Memory	The GPU memory is not sufficient to support more GPU computation.	22	3.12%
Model Checkpointing	A job saves model checkpoints synchronously to the distributed data store.	116	16.43%

Inefficient Host-GPU DataTransfer	Thedata transfer betweenmainmemory and GPU memory is notefficient.	197	27.90%
Data Preprocessing	Raw input data is preprocessed using CPUs before model training.	28	3.97%
Remote Data Read	A job opens and reads input data directly from the distributed data store.	18	2.55%
External Data Usage	A jobaccesses input data or model files directly from external sites.	18	2.55%
Intermediate Result Upload	A job saves intermediate training results synchronously to the distributed data store.	14	1.98%
Data Exchange	The GPUs ofa distributed job continually exchange data,such as gradients and outputtensors, among one another.	50	7.08%

Long Library Installation	The installation of dependent libraries takes too much time (at least 10 minutes).	12	1.70%
API Misuse	API usage violates assumptions.	16	2.27%

• Yanjie Gao (Microsoft Research) et al, ACM ICSE 2024, An Empirical Study on Low GPU Utilization of Deep Learning Jobs .

• Yanjie Gao (Microsoft Research) et al, ACM ICSE 2023, An Empirical Study on Quality Issues of Deep Learning Platform.

如何知晓你的训练任务是否存在这些问题？

计算效率

Figure 4:A simplified example of Inefficient Host-GPU Data Transfer issues.The fix is to enable automatic memory pinning by seting the pin_memory parameter to True (lines 3-4). We further set the non_blocking parameter to True (lines 11- 12),which tries asynchronous data transfer if possible.
Figure 3: A simplified example of Model Checkpointing issues.The fix is to save the checkpoint to alocal temporary file (lines 25-26),followed by anasynchronous copyto the remote data store in a separate thread (lines 28-32).

In a distributed job, GPUs consistently share data such as gradients and output tensors.This data exchange between GPUs, whether through the network or PCI Express bus,frequently interrupts their designated tasks and causes a sudden drop in GPU utilization to zero.Within the Data Exchange category,there are 50 (7.08%) issues.A common fix is to enhance communication effciency by minimizing the frequency of data exchange (e.g., using large batch sizes)and enabling compressed communication [34,39].For users of Horovod [67],enlarging the backward_passes_per_step parameter can help accumulate more local gradient updates and transmit them simultaneously.Developers can also leverage Microsoft DeepSpeed's 3D(data,model,and pipeline) parallelism, whose 1-bit Adam and 0/1 Adam [39] optimizers demonstrate sig nificant reductions in communication volume.

网络传输

LLM 推理开销大、时延高

Llama	8B	70B 405B
FP32	36GB 267GB	1.48TB
FP16	20GB	135GB 758GB
INT8	12GB	70GB 382GB
INT4	8GB	37GB 193GB

LLM Memory Requirements

LLM Inference Performance Engineering: Best Practices

80GB: 1 GPU
640GB: 1 Node x 8 GPU
1.28TB：2 Node x 8 GPU

Time To First Token (TTFT) Time Per Output Token (TPOT) Model Bandwidth Utilization

没有银弹，唯有持续观测 & 优化。

GPU 并非越少越好

GPU 越少，则每个GPU 需要加载更多的模型参数。

GPU 并非越多越好

GPU 越多，则通信越复杂，内存碎块越多。

排查 LLM 推理显存消耗的挑战

PyTorch

Help Needed from vLLM team on profiling pytorch cuda memory

Biz

vLLM

youkaichao1

一

Hi, I'm working on GitHub - vllm-project/vlIm: A high-throughput and memory-efficient inference and serving engine for LLMs 7 ,and the recently release of pytorch 2.2.0 caused some trouble to me.I came here for help in profiling cuda memory usage.

PyTorch

Python /

The basic story: vLLM tries to allocate as much memory as possible for KV Cache to accelerate LLM inference.In order to do so, it first profiles the memory usage, guess the maximum size of memory available for KV Cache,and also leave some for storing activation during inference.

What's more,vLLM uses cuda graph to reduce Python overhead.

When PyTorch upgrades from 2.1.2 to 2.2.0, there seems to be some internal change of memory allocator, and the memory that can be used is decreased. It can cause OOM error during inference.

Here are the diagnoses data (produced by torch.cuda.memory_stats and torch.cuda.memory._dump_snapshot), collected from a server with 2 L4 GPUs:

从推理应用到在线 LLM 推理服务

云 - 在线推理服务是一个复杂的分布式服务TTFT、TPOT、时延、吞吐

Perception-Planning-Control BEV-OCC-Transformer

Robot Operating System 2 (ROS2)-Based Frameworks for Increasing Robot Autonomy: A Survey

端 - 自动驾驶、具身智能（ROS2）的端到端低时延和高稳定性要求

从大模型到小模型：消费级 GPU、CPU 协同

Accelerating Model Training in Multi-cluster Environments with Consumer-grade GPUs

Hwijoon Lim
KAIST
Daejeon,Republic of Korea
hwijoon.lim@kaist.ac.kr
Sangeetha Abdu Jyothi
UCIrvine&VMware Research Irvine, California,USA sangeetha.aj@uci.edu
Juncheol Ye
KAIST
Daejeon,Republic of Korea
juncheol@kaist.ac.kr
Dongsu Han
KAIST
Daejeon, Republic of Korea
dhan.ee@kaist.ac.kr

Accelerating Model Training in Multi-cluster Environments with Consumer-grade GPUs, SIGCOMM 2024.

Figure 1: A multi-cluster environment with two on-premises lab clusters and a cloud cluster.

(a)GPU-based Compression (e.g.,BytePS-Compress [5])

Figure 4: Comparison of training pipelines.

AI 训练和推理的可观测性需求

训练 & 推理

在线、端侧推理服务

PART 02现状：传统解决方案和工具的问题

DCGM Prometheus Exporter

发现故障 √优化性能 X

GPU：Nvidia Nsight、PyTorch Profiler

Nsight packages

Nsight 的问题：

需要重启进程、缺少 CPU Context。

PyTorch Profiler 的问题：

只能用于 PyTorch；性能影响大；需要改代码、重启进程。

RDMA 网络：网卡/交换机指标、拨测

RDMA network behind Al Training Cluster Enables two networked hosts to exchange data in main memory without relying on the processor(CPU)

SCALE AI
TRAINING
WORKLOADS

THE PICK TO KEEP GPUS BUSY

CAN AFFECT TRAINING EFFICIENCY

Meta: Network
Observability
for AI/HPC
Training
Workflows
私有基础设施
网卡/交换机指标
粒度粗

公有基础设施RDMA 网络是性能黑盒

公有云RDMA 网络

AI/ML 应用（K8s Pod）

R-Pingmesh: A Service-Aware RoCE Network Monitoring and Diagnostic System

Hostping: Diagnosing Intra-host Network Bottlenecks in RDMA Servers

Kefei Liu, Zhuo Jiang, Jiao Zhang, Shixian Guo, Xuan Zhang, Yangyang Bai, Yongbin Dong,Feng Luo,Zhang Zhang,Lei Wang, Xiang Shi,Haohan Xu, Yang Bai,Dongyang Song,Haoran Wei,Bo Li, Yongchen Pan, Tian Pan,and Tao Huang

字节&北邮聚焦 RDMA网络主动拨测

SIGCOMM

Kefei Liu,Zhuo Jiang, JiaoZhang,Haoran Wei, Xiaolong Zhong Lizhuang Tan, Tian Pan and Tao Huang

NSDI 2023

在线推理服务的可观测性：分布式追踪

OpenLIT

Python TypeScript
import openai
from langsmith.wrappers import wrap_openai
from langsmith import traceable
client = wrap_openai(openai.Client())
@traceable # Auto-trace this function
def pipeline(user_input: str):
result = client.chat.completions.create(
messages=[{"role":"user","content":user_input}],
model="gpt-3.5-turbo"
return result.choices[0].message.content
pipeline("Hello,world!")
LangSmith
支持的语言有限、需要修改代码

PART 03方法：eBPF 构建零侵扰可观测性

eBPF 的可观测性能力

使用 eBPF 实现可观测性的优势

业内探索：eBPF Profiling & Tracing

Meta：eBPF GPU Profiling

Trace-enabled Timing Model Synthesis for ROS2-based Autonomous Applications

Hazem Abaza*†, Debayan Roy*, Shiqing Fan‡, Selma Saidit and Antonios Motakis* tTechnische Universitat Dortmund, *Huawei Dresden Research Center, ?Huawei Munich Research Center {hazem.abaza, selma.saidi} @ tu-dortmund.de, {debayan.roy6, shiqing.fan, antonios.motakis} @huawei.com

Fig. 1: Proposed trace-enabled timing model synthesis framework.
华为：eBPF Tracing + ROS2

使用 eBPF 实现持续剖析的技术挑战

数据采集

观测对象

数据关联

数据分析

如何合并 Python Stack 和 C/C++ Stack

栈合并目的：全栈剖析

0 select_nocancel ()

1 pysleep

2 time sleep

3 call_function

4 PyEval EvalFrameEx

5 fast function

6 call function

7 PyEval _EvalFrameEx

8 fast function

9 call function

10 PyEval EvalFrameEx

11 fast function

12 call function

13 PyEval _EvalFrameEx

14 fast function

15 call function

16 PyEval EvalFrameEx

17 PyEval_EvalCodeWithName

18 PyEval EvalCodeEx

19 PyEval EvalCode

https://github.com/deepflwio/deepflo

例：剖析显存申请和使用量

① eBPF uprobe Hook cuda_malloc 获取显存申请调用栈

③ eBPF uprobe Hook cuda_free 获取释放的显存地址

④ 计算当前显存消耗

SEC("uprobe/cuda_malloc")
int uprobe_cuda_malloc(struct pt_regs *ctx){
__u32 tgid =id >> 32;
void *address = (void *) PT_REGS_PARM1(ctx);
malloc_data_t *data = cuda_malloc_info__lookup(&tgid);
if(data){
data->address = address;
data->call_time = call_time;
data->rip = PT_REGS_IP(ctx);
cuda_malloc_info__update(&tgid,&newdata);
}
广

SEC("uprobe/cuda_free")
int uprobe_cuda_free(struct pt_regs *ctx){
__u64 id = bpf_get_current_pid_tgid();
void *addr = (void *) PT_REGS_PARM1(ctx);
__u32 zero =0;
struct stack_trace_key_t *key = &state->key;
key->mem_addr = (__u64)addr;
bpf_perf_event_output(ctx，&NAME(cuda_memory_output)，BPF_F_CURRENT_CPU，&state->key，sizeof(state->key));
}

SEC("uretprobe/cuda_malloc")
int uretprobe_cuda_malloc(struct pt_regs *ctx)
_u64id= bpf_get_current_pid_tgid();
__u32 tgid = id >> 32;
malloc_data_t *data = cuda_malloc_info__lookup(&tgid);
cuda_malloc_info_delete(&tgid);
__u32 zero =0;
unwind_state_t *state = heap_lookup(&zero);
struct stack_trace_key_t *key = &state->key;
key->pid = (_u32)id;
key->cpu = bpf_get_smp_processor_id();
bpf_probe_read_user(&key->mem_addr， sizeof(__u64)， (void *)data->address);
key->mem_size = (__u64) data->size;
add_frame(&state->stack,data->rip);
add_frame(&state->stack, state->regs.ip);
bpf_tail_call(ctx，&NAME(progs_jmp_uprobe_map)，PROG_DWARF_UNWIND_IDX);

② eBPF uretprobe Hook cuda_malloc 获取申请的显存地址

Network-Centric Distributed Tracing with DeepFlow: Troubleshooting Your Microservices in Zero Code

使用 eBPF 实现分布式追踪的技术挑战

JunxianShen12,Han Zhang14·,Yang Xiang2*,Xingang Shi14,XinruiLi,Yunxi Shen,Zijian Zhang12,YongxiangWu3,XiaYin34,JilongWang14,MingweiXu14,YahuiLi,JipingYin, Jianchang Song²,ZhuofengLi²,Runjie Nie² 1InstituteforNetworkiencesandCyberspacesinguaUniersity 2Yunshan Networks 3Department of Computer Science and Technology,Tsinghua University P4ZhongguancunLaboratory shenjx22@mails.tsinghua.edu.cn‘zhhan@tsinghua.edu.cn*xiangyang@yunshan.net

ABSTRACT

Microservices are becoming more complicated, posing new challenges for traditional performance monitoring solutions. On the one hand, the rapid evolution of microservices places a significant L.

MingweiXu14YuiLi,JingY2,Jahangong²,ZuofengL jieNie22023.Network-CentricDistributed Tracing with DeepFlow:TroubleshootingYour Microservices in Zero Code.InACM SIGCOMM 2023Conference(ACMSIGCOMM‘23),September 10,2023,New York,NY,USA.ACM, New York, NY, USA, 18 pages.https://doi.org/10.1145/3603269.3604823

Figure 7: Intra-component causal association.

Network-Centric Distributed Tracing with DeepFlow: Troubleshooting Your Microservices in Zero Code

PART 04实践：PyTorch 全栈剖析和追踪

Business Function

Lib/Framework Function

Runtime Function

Shared Lib Function

Kernel Function (Syscall)

CUDA Function

Shared Lib Function

GNU

CUDA Function

Kernel Function

Android

Kernel Function

CUDA Function

Linux

NVIDIA

1. Compute Profiling

PyTorch + nanoGPT CPU & GPU 火焰图

您正在使用函数监控的临时授权

到底哪个函数调用栈在 CPU、GPU 上花费的时间最长？到底一个 Task 的哪个 Worker 拖累了整体的速度？零侵扰、全栈回答上述问题

CPU & GPU 全栈剖析：√ Python 业务函数√ Python vLLM 函数√ Python PyTorch 函数√ C/C++ Lib 函数√ CUDA 入口函数

自身消耗	总消耗
4.431m	4.574m
1.31414m	1.314m
53.758s	2.627m
40.677s	1.155m
33.909s	1.235m
23.727s	15.511m
14.141s	41.717s
12.848s	1.588m
8.061s	9.301m
7.788s	3.99m
7.515s	23.061s
4.99s	15.667s
3.717s	3.717s
2.717s	2.094m
2.465s	55.01s
2.101s	2.101s
2.051s	5.081s
1.99s	1.09m
1.879s	19.414s
1.869s	1.869s
1.859s	24.323s
1.626s	1.626s
1.566s	1.566s
1.465s	17.071s
1.424s	2.859s
1.424s	1.424s

PyTorch + nanoGPT CPU & GPU 火焰图

	名称	自身消耗	总消耗
D	[1] cudaMemcpyAsync	4.431m	4.574m
D	■[1]cudaLaunchKernel	14.141s	41.717s
D	■[1] cublasLtHSHMatmul	7.515s	23.061s
D	[1] cudaGetDevice	1.424s	2.859s
D	■[1] cublasGemmEx	1.414s	25.121s
D	[1]cublasLtHSHMatmulAlgoGetHeuristic	595.959ms	646.464ms
D	[1] cuDevicePrimaryCtxGetState	343.434ms	343.434ms
D	[1]cudaEventQuery	30.303ms	30.303ms
D	[1]custom_eval_frame_shim	eus	45.03s

CUDA 函数耗时

2. HBM Profiling

CUDA mem-alloc 显存申请火焰图

CompiledFunctionBackward:aply (1420%，47run (85.80%，2.9TB)backward (14.20%，479.93GB) g(85.75%，2.9TB)call_compiled_backward (14.20%，479.93GB) wrapper (85.75%，2.9TB)call_func_at_runtime_withargs (14.20%，479run (85.75%，2.9TB)




run (14.20%，479.93GB)	call_func_at_runtime_with_args (85.75%，2.9TB)
_run_from_cache (14.20%，479.93GB)	rng_functionalization_wrapper (85.75%，2.9TB)
CompiledFxGraph
run (14.20%，479.93GB)
run (14.20%，479.93GB)	_call..(14.20%，479.93GBCompiledFxGraph:_ca1l.(85.75%，2.9TB)run (85.75%， 2.9TB)
_run_from_cache (14.20%，479.93GB)call (14.20%，479.93GB)	_run_from_cache (85.75%，2.9TB)call (85.75%，2.9TB)
_Pyobject_MakeTpCall (14.20%，479.93GB)	g(85.75%，2.9TB)apper (85.75%，2.9TB)
[/usr/bin/python3.10] (14.20%,479.93GB)	g(85.75%，2.9TB)apper (85.75%，2.9TB)


c10::impl:wrap_kernel_functor_unboxed_<c10	CompiledFunction:apply (85.75%，2.9TB)forward (85.75%，
	CompiledFunction:apply (85.75%，2.9TB)forward (85.75%，
c10:impl::wrap_kernel_functor_unboxed_c10	call_func_at_runtime_withargs (85.75%2.9TB)
at::(anony	_call_. (85.75%，2.9TB)
at:native:empty_cuda(c10::ArrayRef<long>,	run (85.75%，2.9TB)
at::detail::empty_cuda(c10:ArrayRef<long>,	_run_from_cache (85.75%，2.9TB)
at:detail:empty_cuda(c10:ArrayRef<long>	call (85.75%，2.9TB)_PyObject_MakeTpCall (80.93%，2.73TB)(lost]incompl[/usr/bin/python3.10] (80.93%,2.73TB)/usr/bin/pyth :_ops::emptymemoryformat:cal(c10::Arrayefc10:Snt>std:optional<c10::Scalarype>std:otionalc10:ayout>std:optional<c10::Device>std:optionabool>std:optionalc10::emoryFormat>)(80.84%2.3B)ustom_eval_fr 0:impl:wrap_kernel_functorboeed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunction std::optional<c10:Device>, std::0_PyEval_EvalFr redispatch(c10::DispatchKeySet， emoryFormat>)(80.84%，2.73T_PyFunction_Vec10:impl::wrap_kernel_functor_unboxed_<c10::impl:detail:WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at:Tensor(c10:ArrayRef<c10::SymInt>,std::optional<c10::ScalarType>, std::optional<c10::Layout>, std::optional<c10::Device>, std::custom_eval_frat::(anonymous namespace)::(anonymous namespace):wrapper_CUDA_memory_format_empty(c10:ArrayRef<c10:SymInt>， std:optional<c10:ScalarType>std::optional<c10::Layout>std::optional<c10::Device>std::optional<bool>std::optional<c10::eoor_PyEval_EvalFrat::native:empty_cuda(c10:ArrayRef<long>，std:optional<c10:ScalarType>,std:optional<c10:Layout>， std:optional<c10::Device>,std::optional<bool>，std::optional<c10::MemoryFormat>) (80.84%，2.73B)_PyFunction_Veat:detail:tyua(c10:Araefog>std:oonac10:alarye>std:oioalc10:out>td:oalc0:Devce>st:ooalbool>st:oalc10:emoForat>)（0.84)-PyEval.EvaFrat::detail::empty_cuda(c10:ArrayRef<long>，c10::ScalarType，std::optional<c10::Device>，std:optional<c10:MemoryFormat>) (80.84%，2.73TB)_PyFunction_Veeat:detail:empty_generic(c10:ArrayRef<long>c10:Allocator*，c10:DispatchKeySet，c10:ScalarType，std:optional<c10:emoryFormat>) (80.84%，2.73B)_PyEval_EvalFrat:TensorBaseat:detail:emptygeneric<long>(c10:ArrayRef<long>c10:locatorc10:spatchKeyetc10:calarypestd:optional<c10::emoryFormat>)(80.84%2.7B)
at:detail:empty_generic(c10:ArrayRef<lon
at:TensorBaseat:detail:_emptygenericcl
c10:cuda：CUDCacgllocatoae
c10::cuda::CUDACachingAllocator::Native::Na
c10:cudaCUDCacngAlatoaic
c10:cuda:CUDACachingAllocator:Native:De
c10:cuda:CUDACachingAllocator::Native:(a
cudaMalloc (14.20%，479.93GB)

CUDA mem-inuse 显存实时用量火焰图

为什么显存不够了，OOM 了？哪个函数调用栈当前消耗的显存最大？

Host<->Device 拷贝耗时？

零侵扰、全栈回答上述问题

演进：Hook KV-Cache allocator ?

tick-ok (eBPF) 每个函数使用中的内存大小CompiledFunctionBackward:run (91.60%，21.17GB)backward(8.40%，1.94GB)_run_from_Cache (7.71%，g (83.8%，19.38GB)call_compiledbackward(8all(7.1B) runtime_wrapper(83.88%，19.38GB)call_func_at_runtime_withg(7.71%，1.78GB) run (83.88%，19.38GB)run (8.40%，1.94GB) runtime_wrapper (7.71%，call_func_at_runtime_with_args (83.88%，19.38GB)_run_from_cache (8.40%，run (7.71%，1.78GB) rng_functionaization_wrapper (83.88，19.38GB)
				pythonedFunctionack.1rd:



					Tun(7.71%，1.78GB)
CompiledFxGraph:__.al__	call_func_at_runtime_wi	CompiledFxGra(8）run (83.88%，19.38GB)run_from_cache (83.88%，19.38GB)call (83.88%，19.38GB)
run (8.40%，1.94GB)	rng_functionalization_w
_run_from_cache (8.40%，	CompiledFxGraph:__.al1
call (8.40%，1.94GB)	run (7.71%，1.78GB)
_Pyobject_MakeTpCa11 (8.		runtime_wrapper(83.88%，19.8GB)e_with_args (83.88%，19.38GB)

torch::autograd::THPVari	9 (7.71%，1.78GB)
at:_ops::empty_memoryf	runtime_wrapper (7.71%	，g(83.88%，19.38GB)
c10::impl::wrap_ker	call_func_at_runtime_wi	ompiledFunction:apply888%，19.B)Forward(83.88%，19.38GB)call_func_at_runtine_with_args (83.88%,19.38GB)
at::_ops::empty_memory_f	frng_functionalization_w
c10::impl::wrap_kernel_ff C	ompiledFxGraph:_.al1
at:(anonymous namespace	run (7.71%，1.78GB)	CompiledFxGaa.（3.881B)
at:detail:empty_cuda(c
	_run_from_cache (7.71%,run (83.88%，19.38GB)cal1 (7.71%，1.78GB) _run_from_cache (83.88%，19.38GB)at:detai1:emtycuda(cCoybect_MakeTCallcall(3.88%198B)[1[/usr/bin/python3.0 CompiledFxGraph::_Pyobject_MakeTpCall (77.90%，18GB)torc[lost] incomplete[/usr/bin/python3.10] (77.90%18GB)_Pat::_ops::empt at::_PyEval_EvalFrametorch::autograd:THPVariable_empty(_object，_object，_object*) (7707%，17.81GB)at:_ops:empty_memory_format:call(c10:ArrayRef<c10:SymInt>,std:optionalc0::calarType>,std:optional<c10:Layout>,std:optionalc10::Device>，std::optional<bool>,std:optional<c10:MemorFormat>) (77.07%，17.1B)torc custom_eval_frame :optional<c10::L std::optional<c10::Devc_PyEval_EvalFrameat:ptathsetceat:_PyFunction_Vecto std:optionalcc10:ScalarType>, std::optional<c10::Layout>, std:optionalcc10:DevcPat:native:ematcustom_evalframe :ScalarType>[/at:detail:emvoidPyEvalEvaaeat:native:etycuda(c10:AraRef<ong>std:oionalc10:calarye>std:tonalc10:ayout>std:otioalc10:Dvice>std:tioa<bol>std:tonalc10:emorFoat>) (77.07%171B)cu at:detail:em _PyFunction_Vectoat::detail::empty_cuda optional<bool>，std:optional<c10:MemoryFormat>)Pat:detail::eat：：_PyEval_EvalFrameat:detail:empty_cuda(c10:ArrayRef<long>c10::ScalarType，std::optional<c10::Device>，std:optional<c10:MemoryFormat>) (77.07%，17.81GB)_PyFunction_Vectoat:detail:etyenricc10:aeog> :Dispatchlcuc10::cuda::CUDc10：_PyEval_EvalFrame


at:detail::empty_generi

c10::cuda:CUDACachingA1c10::cuda:CUDACachingA1P
			Pc10::impl:wratorc
610:cuda:CUDACachingA]
c10::cuda::CUDACachingAl
c10:cuda:CUDACachingA1
1.9GB			Plat:native'emat.

3. COMM. Profiling

RDMA 网络性能剖析

UDP encapsulation			Original InfiniBand packet
Ethemet Header	IP Header	UDP Header	InfiniBand Base Transport Header		InfiniB and Payload	ICRC	FCS

Opcode	Pad Count	：	Dest QP	ACK Request	Reserved	PSN
8 bits	2 bits		24 bits	1 bits	7 bits	24 bits

RoCEv2 Packet

eBPF Hooks

Function	作用
ibv_modify_qp	Queue Pair 状态修改Q
ibv_reg_mr	Memory Region 注册
ibv_dereg_mrM	emory Region 注销
ibv_post_sendb	uffer 发送或 RDMA 读写
ibv_poll_cq	等待wr完成

libibverbs Hooks, NCCL Hooks, PyTorch Hooks, ...

关键标签：

Client 及其关联的 K8s Pod、标签

• Server 及其关联的 K8s Pod、标签

• Client Queue Pair

• Server Queue Pair

关键指标：

丢包率：即 NACK 的比例

时延：ACK 的时延

吞吐：通信对的 bps、pps 等

使用 eBPF Hook RDMA API

4. Distributed Tracing

在线推理服务、端侧 ROS2 推理服务

脚本*

内置协议识别能力 + 可编程协议识别能力

deepflow-server

框架开发

平台工程

.so

热插拔

内置解析器

LUA 插件

WASM插件

Function Call

Network Packet

协议识别

应用协议字段提取

业务属性标签提取

deepflow-agent

拓扑、指标、追踪、日志LLM 应用全栈可观测性

业务属性标签用户唯一标识、Prompt 唯一标识、

业务

Protobuf

JSON

XML

私有

RPC over TCP

常见

SOFASTACK

MQ

8 kafka APACHE LRabbitMQrPULSAR ACTIVEMQ OMQNATS SMQTT RocketMQ

PART 05探索：Agent 自动优化 ML 代码

如何快速高效的理解全栈函数

Business Code

GNU C Library

RAG Docs/Comments

The Linux Kernel

Prompt Engineering tag.commit_id

LLM/微调

利用 LLM Agent 自动优化 ML 代码

最热调用路径概述

整个性能分析显示，程序存在大量与定时器相关的操作，这些操作占据了很大一部分的CPU资源，特别是runtime.siftdownTimer和runtime.runOneTimer。同时，main.busyLoop表明业务逻辑中存在大量的忙等，可能需要优化逻辑以减少CPU消耗。调用链分析显示，time.Sleep和runtime.gopark的使用表明在循环中有意识的休眠或等待，可能需要重新评估这部分逻辑的必要性和效率。需要特别注意的是，频繁的定时器操作和channel通信可能是导致性能问题的关键所在，应当作为优化的重点。

查看runtime.siftdownTimer的具体信息

查看runtime.chansend的具体信息

查看-_vdso_clock_gettime的具体信息

查看runtime.runoneTimer的具体信息

Di纳量分价

利用 LLM Agent 自动优化 ML 代码 / 2

Fl&w* 析
续机智分析
能分析总的情况
节点

当体概述
作，大一，可，量
优化的点

Diff内容分析
对于文件main.go，commitid f52deb9494fff9db104028cbc3a341ade8d356d的diff内容如下：
@@-49,9+49,15@@ func busyLoop(id int）{
func tickerLeak(id int）{
fmt.Printf("Starting ticker leak thread id:%d\n",id)
var ticker *time.Ticker
time.Sleep(time.Second）
ticker=time.NewTicker(5*time.Millisecond)
//dosomething
break

代码分析

fmt.Printf("Starting ticker leak thread id:%d\n",id)
var ticker*time.Ticker
ticker=time.NewTicker(5* time.Millisecond)
deferticker.Stop（）//确保定时器被停止，避免泄漏
for range ticker.C{
//do something
//..执行定时任务..
}

DeepFlow：零侵扰实现 AI 应用的全栈可观测性

CPU+GPU 全栈持续剖析

Prometheus

OpenTelemetry

SKywalking

Pyroscope

AS SINK

集合通信网络性能监控

DeepFl W https://deepflow.io

在线推理服务分布式追踪

AS SOURCE

eBPF

Open Telemetry

WA AWasm

SKywalking

无需修改代码，无需重启进程

8kafka

2024AI+研发数字峰会 Al+ Development Digital summit AI驱动研发变革 促进企业降本增效

基于 eBPF 和 Agent 构建LLM 训练推理优化体系

演讲嘉宾

向阳

目 录 CONTENTS

PART 01背景：训练和推理的效率挑战

LLM 训练开销大、 效率低

代码层面训练低效的主要原因

网络传输

LLM 推理开销大、 时延高

排查 LLM 推理显存消耗的挑战

PyTorch

Help Needed from vLLM team on profiling pytorch cuda memory

Biz

youkaichao1

从推理应用到在线 LLM 推理服务

端 - 自动驾驶、具身智能（ROS2） 的端到端低时延和高稳定性要求

从大模型到小模型：消费级 GPU、CPU 协同

Accelerating Model Training in Multi-cluster Environments with Consumer-grade GPUs

AI 训练和推理的可观测性需求

PART 02现状：传统解决方案和工具的问题

DCGM Prometheus Exporter

GPU：Nvidia Nsight、PyTorch Profiler

Nsight packages

Nsight 的问题：

PyTorch Profiler 的问题：

RDMA 网络：网卡/交换机指标、拨测

在线推理服务的可观测性：分布式追踪

PART 03方法：eBPF 构建零侵扰可观测性

eBPF 的可观测性能力

使用 eBPF 实现可观测性的优势

业内探索：eBPF Profiling & Tracing

Trace-enabled Timing Model Synthesis for ROS2-based Autonomous Applications

使用 eBPF 实现持续剖析的技术挑战

如何合并 Python Stack 和 C/C++ Stack

0 select_nocancel ()

1 pysleep

2 time sleep

3 call_function

4 PyEval EvalFrameEx

5 fast function

6 call function

7 PyEval _EvalFrameEx

8 fast function

9 call function

10 PyEval EvalFrameEx

11 fast function

12 call function

13 PyEval _EvalFrameEx

14 fast function

15 call function

16 PyEval EvalFrameEx

17 PyEval_EvalCodeWithName

18 PyEval EvalCodeEx

19 PyEval EvalCode

例：剖析显存申请和使用量

① eBPF uprobe Hook cuda_malloc 获取显存申请调用栈

③ eBPF uprobe Hook cuda_free 获取释放的显存地址

④ 计算当前显存消耗

② eBPF uretprobe Hook cuda_malloc 获取申请的显存地址

使用 eBPF 实现分布式追踪的技术挑战

ABSTRACT

PART 04实践：PyTorch 全栈剖析和追踪

1. Compute Profiling

PyTorch + nanoGPT CPU & GPU 火焰图

PyTorch + nanoGPT CPU & GPU 火焰图

CUDA 函数耗时

2. HBM Profiling

CUDA mem-alloc 显存申请火焰图

CUDA mem-inuse 显存实时用量火焰图

3. COMM. Profiling

RDMA 网络性能剖析

eBPF Hooks

关键标签：

关键指标：

4. Distributed Tracing

在线推理服务、端侧 ROS2 推理服务

内置协议识别能力 + 可编程协议识别能力

拓扑、指标、追踪、日志LLM 应用全栈可观测性

MQ

2024AI+研发数字峰会 Al+ Development Digital summit AI驱动研发变革促进企业降本增效

目录 CONTENTS

LLM 训练开销大、效率低

LLM 推理开销大、时延高

端 - 自动驾驶、具身智能（ROS2）的端到端低时延和高稳定性要求

AiDD-计算机行业2024AI+研发数字峰会：基于eBPF和Agent构建LLM训练推理优化体系-250320.pdf