Initial support for BEAM (Erlang/Elixir) #289
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Typically JIT is on mmaped anonymous memory. You will need to add hooks to call your unwinder for this memory mappings. For a generic catch it all example, see the If you can extract the exact memory area where JIT code exists directly from the VM, you can refer to
The native winder will not have heuristic for it. You need to implement the code to hook your unwinder for the memory areas where JIT code is at (see above). After that you'll need to have eBPF code that actually unwinds the JIT code. It might be simple if the JIT frame layout is frame pointer based, see e.g. v8 unwinder https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/main/support/ebpf/v8_tracer.ebpf.c, or highly complicated if there is a custom frame layout, see e.g. hotspot unwinder https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/main/support/ebpf/hotspot_tracer.ebpf.c. The unwinder will need to collect the extra needed by the symolization in the next step.
Once the unwinding is done, the core will code interpreter plugins symolization code which will need to extract the symbol data from the target process. Again, see some examples how its done for the
Correct, you will need to implement both the unwinding and symbolization yourself. Depending on the VM internals, this can be highly complicated and extensive work that is needed to cover all the corner cases within the ebpf constraints. |
Aha! Thanks, this was the connection I was missing. I was thinking that since I can't statically know about all the JIT code that might be generated in the future, I can't possibly add it all to the maps, but I believe the BEAM does have ways to pretty easily locate the memory of all the JITted code, so I'll dig into that and the v8 example. The BEAM does use frame pointers, so I believe it should be relatively straightforward to figure out. Thanks for the tips! ❤️ 🚀 |
The range for the interpreter looks suspiciously small - only 128 bytes. This can be valid, if its just a small stub but guaranteed to be on stack for interpreter frames. Alternatively, this could be a function doing something else that is not necessarily on stack when executing interpreted code. You might want to double check which functions are on stack when executing interpreted code. If it can be a set of multiple functions (e.g. several functions with same signature tailcalling each other -- compiler can convert call to jump), you need to extract the range that covers all of these. It would become a problem if these functions are not contiguously in the executable area.
No problem. But in short, you'll need to manually extract those areas and then call You can also provide little bit of context data for each memory area. This could be useful if there's some auxiliary data connected to each memory area the unwinder needs.
Nice! Then
You're welcome. Looking forward to the BEAM support! Thank you for working on this! |
|
FYI: I have opened a open-telemetry/semantic-conventions#1735 with OTel semconv to add a type for beam. |
GregMefford
force-pushed
the
beam_support
branch
from
93d0726 to
caa65c3
Compare
| return 0, nil, fmt.Errorf("failed to read erts version: %v", err) | ||
| } | ||
|
|
||
| return uint32(otpMajor), ertsVersion, nil |
Check failure
Code scanning / CodeQL
Incorrect conversion between integer types
|
I had some more time today to make progress, by copying the v8 implementation of With the "catch-all" mappings, I get logs like this out of the agent, so I assume that means it's working and there's just one large region of memory it's treating as BEAM JIT code: With the fancier JITDump code, I get logs like this, so I am pretty sure it's parsing the file correctly. Those function names look sensible to me and they're function names I recognize as things I did include in my test app I'm running. I was hoping to see logs from the eBPF code using the following (as root), but nothing is showing up: This is the kind of thing I'm seeing in DevFiler, so it's looking promising that it's doing something, but I'm not clear on where to look next for signs of life / why the eBFP unwinder does not seem to be getting called like I was expecting. 
|
To get log lines using the |
Ah, thanks! I wasn't sure how to make that work, which is why I did it this way instead, which did work when I put it in one of the other eBPF programs, but I don't see anything coming from my program still. |
|
With #145 things changed a bit and I missed that part. Sorry that I have missed this one in the first place. Here are steps I used to generate output to I missed, that |
|
I spent some more time today to make progress on this, and was able to get past where I was stuck before, and now I am able to see that my eBPF unwinder is running (not sure yet if it's doing the correct thing, but it's doing something) and my In the output form the Is this because of the PR you mentioned opening to add that as a supported type in the OTEL spec, and it hasn't yet been pulled into DevFiler? Or is there just somewhere in the Thanks again for all your help! |
Yes - devfiler uses a filter on frame types. This might change in the future. For the mean time and to enable you to continue your work, I have created a version of devfiler that handles beam frame types: Working on live processes can be tricky and reproducing edge cases can be hard. For that reason, I recommend looking into coredump. With the tool coredump one can import a core dump of a process and run all the Go and eBPF code in user space just like a regular Go test. Please feel free to ping me, if you need help. |
|
Sorry it took so long to make some time to actually give it a try, but when I tried running the custom I'm planning to try to figure out how to get the core dump testing thing working, though, so not blocking at the moment. I just wanted to see whether it got me further along with the path I was on before, having a version of |
|
sorry - this was not supposed to happen 🙏 |
|
With the updated version of 
|
GregMefford
force-pushed
the
beam_support
branch
from
91d71e1 to
f8f3cb5
Compare
GregMefford
force-pushed
the
beam_support
branch
from
f8f3cb5 to
ea28f8a
Compare
| // Minimal JITDUMP file reader for BEAM | ||
|
|
||
| // This has the minimal code we need to read the JITDUMP files that the BEAM | ||
| // writes to `/tmp/jit-<pid>.dump`. It isn't BEAM-specific, so it could probably | ||
| // be used more generally. The spec for this file format is at: | ||
| // https://raw.githubusercontent.com/torvalds/linux/refs/heads/master/tools/perf/Documentation/jitdump-specification.txt |
There was a problem hiding this comment.
The general idea has been to natively support VMs without jitdump. The main three reasons are:
- on many VMs enabling jitdump can give significant negative performance impact on the VM
- usually the jitdump output is inferior and gives only symbol names. most of our plugins are superior by extracting source code line level information and decode inlined function information
ebpf-profilerwas designed to be a zero changes to system required profiler, and often enabling jitdump requires changes to system
In other words, native support is preferred if possible. Have you looked at all if extract the symbolization information is from the process directly is doable? I do understand this is more work, but as explained above it also gives much better results.
OTOH, we have had discussion on supporting jitdump earlier. And if supporting beam directly is not feasible or possible, I think no one will object on adding jitdump plugin either. But then I think this should be renamed to jitdump plugin and made generic. Though, understandably this may require changes in core code. E.g. to automatically enable jitdump plugin if the corresponding jitdump file is found (instead of using elf file specific regexes).
There was a problem hiding this comment.
Also another potential issue with jitdump format is that, its a linear dump of what the VM does with all of JIT output. Basically the file can grow boundlessly, and reading/parsing it may require non-trivial amount of memory. In other words, depending on the VM work load, it may result the profiler to require a huge amount of memory to track all JITted functions.
Typically the interpreter plugins only track the functions it sees in the traces helping a lot to keep the memory usage within reasonable limits. Though there are caveats here also.
Just a few comments in advance to note the observations we had earlier on the approach of using jitdump. While the initial implementation might be simple(r), the complexities come from trying to enable jitdump in a long living / large processes on a production system.
There was a problem hiding this comment.
Hello! Erlang VM developer jumping into the discussion...
In other words, native support is preferred if possible. Have you looked at all if extract the symbolization information is from the process directly is doable? I do understand this is more work, but as explained above it also gives much better results.
I don't know anything about what is available to you when running ebpf, but if you can get symbol locations and then read any memory from the process executing (which it sound like you can do), then it is definitely possible to get the symbols without dumping.
This gdb macro function etp-cp-func-info-1 shows how you can get to that information.
We also have a jit-reader plugin for gdb that does the same thing: https://github.com/erlang/otp-gdb-tools/blob/master/jit-reader.c.
There was a problem hiding this comment.
Thanks for the input! I had started to look into using jitdump originally because I thought it was necessary for identifying which memory addresses to map to the BEAM process for unwinding, but have later learned that isn't necessary. I left it in there for now in case I ended up needing or wanting it for some other purpose.
Ultimately I agree that it shouldn't be necessary to use jitdump, and I don't mind figuring out how to make it work without that. It seems like it's possible. I've been staring at the etp macros and I think I mostly understand what they're doing - I just need to figure out how to properly implement them in eBPF.
If there's interest in having a jitdump plugin for other non-BEAM processes, we could talk about splitting it out into a separate PR once we get close to finishing this one up.
Using frame-pointers will make your life easier, but it should not technically be needed. We added frame-pointers in order for perf to work as adding a separate crawling scheme to it was not trivial. However crawling the Erlang stack without frame-pointers is quite easy as you just have to rewind the stack looking for all the values that have the two laest significant bits set to 0, stopping at a certain end-marker. Maybe having a look at the gdb unwind code that we have can be inspirational? https://github.com/erlang/otp-gdb-tools/blob/master/jit-reader.c#L222-L312 or gdb scripts if that more to you taste: etp-stacktrace-1. One problem that you will notice though is that we write to |
I think this is the ideal approach for the profiler too. Basically rewrite the gdb jit reader plugin as an ebpf unwinder and a host agent plugin. |
|
Thanks for the tips, @fabled and @garazdawi! I've spent a lot of time staring at how the gdb scripts work, and how it might be different when running on ARM / Apple Silicon vs. x86, but I hadn't dug into the It looks like there's a pretty clear path forward, I just need to understand how to traverse the symbols and memory offsets in eBPF to get to where I need to read the function info, and then how to pass that from eBPF to the Golang agent code. |
Note
The first commit in the PR is so that this branch stacks on top of #288 while I work on this one.
I have begun to work on support for BEAM languages like Erlang and Elixir, and wanted to open this PR early as a draft, so that I can get any feedback you may have to help the process go more smoothly. I don't have much experience with Go or eBPF, so any feedback you have is very welcome. What I have so far is mostly based on digging through the existing support for other languages as well as the BEAM / OTP source code, and trying to understand how all the parts fit together.
I have also been digging through the BEAM / OTP source code, and also the
gdbscripts that it includes for working directly with the memory image of a running system or core dump.So far, I am able to see the logs from my Go code coming through, and confirming that it's working correctly as far as loading and attaching the interpreter support, like this:
However, I can't seem to get any tracing logs out of the eBPF program, so I suspect that it's never being run. If I modify the
nativeeBPF script to write the same kind of log there, I can confirm that I'm seeing it in/sys/kernel/tracing/trace_pipeafter doing the following to narrow down the logs I want to see, but the same doesn't work for mybeamprogram:I was thinking that this was because OTP 27 includes a JIT, so the interpreter might never be used, but I am also not seeing any frames for the native JIT code executing, so I'd love any advice you may have there in terms of how I might go about troubleshooting that. Maybe the native unwinder is just missing some heuristic that's needed for the way the ASMJIT / BEAMJIT works? I'm not clear on how the profiler resolves symbols for JIT code or how those should show up in
devfiler, so maybe it is working and I just don't know how to use to the tool... 😅 But from what I can tell, I don't think the frames are showing up there for anything but the C code for Erlang itself (and built-in C functions). I was expecting to be able to see which Erlang code was running, for example.I also tried building OTP 27 with the JIT disabled to confirm my theory that it just wasn't working, but it behaved the same (though with a different memory address showing for the
interpRanges, which confirms that it really did build a different set of code).