Let’s dive into the memory leak detection part of our eBPF
toolkit.
让我们进入 eBPF 工具集中内存泄漏检测的部分。
So, you’ve got a memory leak in production. What do you do? Your
first instinct might be to reach for Valgrind — but then you realize it
would slow your service down by twenty times. Your users would notice.
Your boss would notice. Not ideal.
假设你的生产服务有内存泄漏。你怎么办?第一反应可能是用
Valgrind——但随后你意识到它会让服务慢 20
倍。用户会发现,老板也会发现。不太理想。
That’s exactly why we chose eBPF. Look at the left side of this table
— zero code changes, production-safe, minimal overhead, and it can trace
both user-space and kernel-space allocations in real time. It’s like
having X-ray vision into your running process without the patient even
knowing they’re being examined.
这正是我们选择 eBPF
的原因。看表格左侧——零代码修改、生产安全、极低开销,还能实时追踪用户态和内核态的内存分配。就像给正在运行的进程做
X 光检查,而病人完全不知道自己在被检查。
Now, to be fair, traditional tools have superpowers that eBPF
doesn’t. Valgrind and ASan can catch use-after-free, buffer overflows,
and double-free with byte-level precision. But Valgrind costs you ten to
twenty X in performance, and ASan requires recompiling your code.
公平地说,传统工具有 eBPF 不具备的超能力。Valgrind 和 ASan 能捕获
use-after-free、缓冲区溢出和 double-free,精确到字节级别。但 Valgrind
要付出 10 到 20 倍的性能代价,ASan 需要重新编译代码。
So think of it this way — eBPF is your doctor on call in production;
Valgrind and ASan are the specialists you visit during development.
They’re complementary, not competing.
所以这样想——eBPF 是你生产环境的值班医生;Valgrind 和 ASan
是你开发阶段去看的专科医生。它们是互补关系,不是竞争。
The tradeoffs: eBPF only detects leaks — unfreed memory. It requires
Linux 4.9 or above, root privileges — or CAP_BPF on kernels 5.8 and
later. It won’t catch use-after-free or buffer overflows. But for
hunting down leaks in long-running production services? It’s exactly the
right tool for the job.
代价是:eBPF 只能检测泄漏——即未释放的内存。需要 Linux 4.9 以上、root
权限——或在 5.8 及更高版本内核上使用 CAP_BPF。它抓不到 use-after-free
或缓冲区溢出。但对于在长期运行的生产服务中找内存泄漏?它正是对症的工具。
Now let’s look at what data we actually capture. Next slide,
please.
现在来看我们实际捕获了哪些数据。下一页。
Alright, so what exactly are we capturing under the hood?
好,那我们底层到底在捕获什么?
In user-space, we attach uprobes to the usual suspects: malloc,
calloc, realloc, and free. Every single call gets intercepted — and your
application doesn’t even know it’s happening.
在用户态,我们把 uprobes
挂到那几个”常见嫌疑人”上:malloc、calloc、realloc 和
free。每次调用都被拦截——而你的应用程序完全不知道这事正在发生。
In kernel-space, we use kprobes to hook kmalloc, kfree, and
kmem_cache_alloc. This is where it gets interesting — we can see inside
the kernel’s memory behavior. Try doing that with Valgrind. Spoiler: you
can’t.
在内核态,我们用 kprobes 挂载 kmalloc、kfree 和
kmem_cache_alloc。这就是有趣的地方——我们能看到内核内部的内存行为。你试试用
Valgrind 做这个。剧透:做不到。
For each allocation, we record everything you’d want for debugging:
the address and size, timestamps for both alloc and free, the full stack
trace — so you know exactly who allocated this memory — and the PID and
TID so you can trace it back to a specific process and thread.
每次分配,我们记录调试所需的一切:地址和大小、分配和释放的时间戳、完整调用栈——这样你就能精确知道是谁分配了这块内存——以及
PID 和 TID,方便追溯到具体的进程和线程。
The output gives us two views: a timeline showing how unfreed memory
accumulates over time — think of it as watching a leak drip — and a
ranked list of the top leaking call stacks by total bytes. The biggest
offenders float right to the top.
输出给我们两个视图:一个时间线展示未释放内存如何随时间累积——想象看水龙头一滴一滴漏水——以及按泄漏总字节排名的调用栈列表。最大的”罪犯”直接浮到最上面。
Let’s see this in practice. Next slide, please.
来看看实际效果。下一页。
Let’s see this in action with a concrete example.
让我们用一个具体例子来看看实际效果。
Here’s a simple test program — intentionally buggy. There are memory
allocations at lines 13, 20, 25, and 30. Line 25 is the “good citizen” —
it frees its memory properly. The other three? They just allocate and
walk away. Classic leak pattern.
这是一个简单的测试程序——故意写的有 bug。第 13、20、25 和 30
行各有一次内存分配。第 25
行是”好公民”——它正确释放了内存。其他三个?分配完就走人了。经典的泄漏模式。
The question is: can memleak tell the difference between the
responsible allocation and the irresponsible ones? Let’s find out.
问题是:memleak 能区分负责任的分配和不负责任的分配吗?我们来看看。
Let’s look at the output. Next slide, please.
来看看输出结果。下一页。
And here’s the verdict.
结果出来了。
Memleak nails it — three leaked allocations reported, pointing
directly to lines 13, 20, and 30 in our source code. Full stack traces
included, so there’s zero ambiguity about where the leak
originates.
Memleak 精准命中——报告了三处泄漏分配,直接指向源代码的第 13、20 和 30
行。包含完整调用栈,泄漏来源毫无歧义。
And notice what’s missing — line 25. The one allocation that was
properly freed is completely absent from the report. Memleak isn’t just
dumping all allocations; it’s tracking the alloc-free pairs and only
flagging the ones where free never showed up. It’s like a bouncer who
only kicks out the people who didn’t pay their tab.
注意什么没出现——第 25 行。那个被正确释放的分配完全不在报告中。Memleak
不是简单地转储所有分配;它追踪分配-释放配对,只标记那些 free
从未出现的。就像酒吧保镖只赶走没买单的人。
That was user-space. Now let’s see what happens in kernel-space. Next
slide, please.
那是用户态的。现在来看内核态会怎样。下一页。
Now let’s go deeper — literally. Into the kernel.
现在让我们更深入——字面意义上的。进入内核。
With memleak, we can monitor kmalloc and kfree inside the kernel
without rebuilding anything or loading custom kernel modules. Just point
the tool at kernel mode, and you immediately start seeing kernel memory
allocations with full call chains.
用 memleak,我们可以监控内核内部的 kmalloc 和
kfree,无需重新编译任何东西或加载自定义内核模块。只要把工具指向内核模式,你就能立即看到带有完整调用链的内核内存分配。
This is a game-changer for debugging kernel module leaks, file system
driver issues, or network subsystem problems. These are scenarios where
Valgrind and ASan are completely blind — they simply don’t operate at
this level.
这对调试内核模块泄漏、文件系统驱动问题或网络子系统问题是颠覆性的。这些场景下
Valgrind 和 ASan 完全是瞎的——它们根本不在这个层面工作。
Now, what about Rust projects? Next slide, please.
那 Rust 项目呢?下一页。
Now here’s a question I get a lot: “But we use Rust — doesn’t the
borrow checker prevent memory leaks?”
这是我经常被问到的一个问题:“但我们用的是
Rust——借用检查器不是能防止内存泄漏吗?”
Well… not entirely. Rust prevents use-after-free and double-free at
compile time, yes. But memory leaks? Those are actually considered safe
in Rust. You can leak memory through reference cycles with Rc, through
mem::forget, or through circular Arc references. The compiler won’t stop
you.
嗯……不完全是。Rust 确实在编译期阻止了 use-after-free 和
double-free。但内存泄漏?在 Rust 中实际上被认为是”安全的”。你可以通过 Rc
引用循环、mem::forget 或循环 Arc 引用来泄漏内存。编译器不会阻止你。
On the left, you can see a leak we detected in a real Rust project.
On the right — the fix. And here’s the beauty: since Rust’s allocator
calls malloc and free under the hood, our uprobe-based approach works
out of the box. No Rust-specific setup needed.
左侧可以看到我们在一个真实 Rust
项目中检测到的泄漏。右侧——修复后的结果。精妙之处在于:由于 Rust
的分配器底层调用的是 malloc 和 free,我们基于 uprobe
的方法开箱即用。不需要任何 Rust 特定的配置。
Now let’s talk about how we visualize all this data. Next slide,
please.
现在来看我们如何把这些数据可视化。下一页。
Now, reading raw terminal output is fine when you’re debugging at two
AM with a coffee in hand. But when you need to share findings with your
team or write a post-mortem? You need something prettier.
读原始终端输出在凌晨两点端着咖啡调试的时候还行。但当你需要和团队分享发现或写事后分析报告时?你需要更漂亮的东西。
That’s why we built an HTML visual report. We parse the raw memleak
log and generate a self-contained HTML file — no server needed, no
dependencies, just open it in any browser.
所以我们构建了 HTML 可视化报告。解析 memleak 原始日志,生成独立的 HTML
文件——不需要服务器、没有依赖,在任何浏览器中打开即可。
It gives you three things: trend charts showing memory growth over
time, automatic detection of continuously growing allocations flagged as
likely leaks, and a detailed table where you can drill into each leak’s
stack trace. From “something is leaking” to “here’s the exact line” in
one click.
它给你三样东西:展示内存增长的趋势图、自动检测持续增长的分配并标记为可能的泄漏、以及详细表格可以深入查看每个泄漏的调用栈。从”有东西在漏”到”就是这行代码”,一次点击。
Let me show you the actual charts. Next slide, please.
让我展示实际的图表。下一页。
Let’s look at real data.
来看看真实数据。
The top chart shows total memory from the top ten allocation sources
over time. See that sharp rise in the first ten seconds? Then it
flatlines at 8.3 megabytes. That pattern tells a story — these
allocations happen during initialization.
上方图表展示了 Top 10 分配源的总内存随时间的变化。看到前 10
秒的急剧上升吗?然后在 8.3 MB
平稳了。这个模式说明了一个故事——这些分配发生在初始化阶段。
But is it a leak? That’s the key question. If the memory is retained
but serves no further function — like loading a config into a buffer you
never read again — it’s effectively a leak. If it’s actively used
throughout the program’s lifetime, it’s just the working set. Context
matters.
但它是泄漏吗?这是关键问题。如果内存被持有但不再有实际作用——比如把配置加载到一个再也不读的缓冲区——那就可以视为泄漏。如果在程序生命周期中一直被使用,那只是工作集。上下文很重要。
The bottom chart zooms into the peak moment. One source dominates
everything: do_anonymous_page at 6.4 megabytes — that’s 77 percent of
the total. When one source is this dominant, you know exactly where to
start investigating.
下方图表放大了峰值时刻。一个来源统治了一切:do_anonymous_page 占 6.4
MB——总量的 77%。当一个来源这么突出时,你就知道该从哪里开始查了。
Finally, let’s look at the alerts and detail table. Next slide,
please.
最后来看告警和详细表格。下一页。
And the cherry on top — automated alerts.
最后的点睛之笔——自动告警。
This report analyzed nine snapshots over 40 seconds. The yellow
warning box immediately calls out three suspected continuous leaks —
allocations that grew in every single snapshot. The worst offender:
kernfs_fop_open, which nearly tripled from 26K to 72K. That kind of
relentless growth is a smoking gun.
这份报告分析了 40 秒内的 9
个快照。黄色警告框立即指出三个疑似持续泄漏——在每个快照中都在增长的分配。最大的”罪犯”:kernfs_fop_open,从
26K 几乎三倍增长到 72K。这种不停的增长就是”冒烟的枪”。
Below that, the detailed table lets you expand any row to see the
full stack trace, search by function name, and sort by any column. It
turns what would be hours of log-grep into seconds of
point-and-click.
下方的详细表格可以展开任意行查看完整调用栈、按函数名搜索、按任意列排序。把原本需要数小时
grep 日志的工作变成了几秒钟的点击操作。
To summarize: eBPF gives us a production-safe, zero-modification way
to detect memory leaks in both user-space and kernel-space. Combined
with our HTML reporting tool, we can quickly identify, visualize, and
drill down into the exact source of leaks. Thank you.
总结一下:eBPF
为我们提供了一种生产安全、零修改的方式来检测用户态和内核态的内存泄漏。结合我们的
HTML
报告工具,可以快速识别、可视化并深入定位泄漏的确切来源。谢谢大家。