In the rust world, crates are usually statically linked into the produced binary. There is some dynamic linking support in rust called dylib but there’s no stable Rust ABI like the C ABI.

The primary use case for dylib is speeding up incremental compilation, for example, by making a large crate a dylib and prefer dynamic linking: https://github.com/bevyengine/bevy/issues/791. But many crates only support the lib (which is rlib underhood) crate type. It will take lots of effort to just add dylib to the crate-type field of those crates. It is sad that dylib is neither widely supported nor widely used, making producing a fully dynamically linked rust binary pretty hard.

But there is actually a viable workaround, we can create a wrapper dylib crate and add the rlib crate as a dependency to it. cargo-add-dynamic provides a cargo command to do it. (And don’t miss Robert’s blog post: Speeding up incremental Rust compilation with dylibs)

So this made me wonder if I could use some tricks to produce a fully dynamically linked rust binary and inspired me to do this experiment and wrote this blog post.

To do that, the first step is to hack on cargo to make crates support dylib by default.

Hack on Cargo to make dylib first-class citizen

The following patch makes cargo think that every crate can be built as dylib (except proc-macro, which is already some sort of dylib).

Apply the patch to cargo repository and build cargo.

Then let’s do a build of tracexec using our patched cargo:

Wow, what do we get? tons of build errors complaining about panic handler and memory allocator:

So what’s happening here? The failing crates have one thing in common. They are #[no_std].

It turns out that rust requires dylib to ship memory allocator and panic handler with them. However, no_std crates don’t ship them. And if a rust binary crates depends on any dylib, it will just use the allocator from the dylib.

For the purpose of simplifying this experiment, let’s make an incompatible hack in rustc to shift the responsibility of shipping the allocator from dylib to bin crates:

Hack on Rustc to fix compilation

First clone rustc repo and setup it:

Then let’s first bypass the check for weak lang items with the following patch:

This also nicely aligns with the current compiler behavior of rlib.

And then we can apply a simple hack to avoid shipping allocator in dylib:

Then lets build rustc and the libaries:

And try to build tracexec again:

We get different error messages compared with our last attempt! Some progress.

So rust complains about core shows up multiple times. What’s happening here?

This crates are only available in rlib format:

And when building multiple dylibs, each dylib would statically link core crate, which is what the error message is saying:

The solution, from the perspective of this experiment, is dynamically linking core, alloc and other built-in crates.

One complication, though, compiler-builtins crate is not in the rustc repo. Let’s add it as a git submodule:

And mark all those built-in crates as available in dylib format:

Then we should build a new rust compiler with the previously build rust compiler that already contains our hacks.

We could see that the dylib flavor of those crates are now available:

Now let’s try to build something again.

A linker error shows up. Another progress.

What’s happening this time?

By looking at libbpf-rs’s dependencies, it is clear that this comes from vsprintf crate. This crate compiles a C wrapper and links it into the crate. Explicitly specify the linkeage between the C wrapper and the crate solves it: https://github.com/dylanmckay/vsprintf/pull/3

Finally it builds 🎉 🎉 🎉, although we are seeing a lot of warnings about file path collision:

The dylib gets built multiple times and later build overwrite previous build: https://github.com/rust-lang/cargo/issues/6313

It’s time to run it. Because the dylibs are not in runtime linker search path, we need to set LD_LIBRARY_PATH:

Otherwise, it works!

But sometimes the compilation still fails:

For some reason rustc thinks the rlib is a candidate for a dylib. But wait, why are there still rlibs? Actually we don’t need those rlibs.

It turns out that cargo passes --crate-type lib --crate-type dylib to rustc, which makes rustc to output both rlib and dylib. Let’s apply a dirty patch to cargo to produce dylib only:

Clean and build again. This time, all rlibs are gone:

Performance Evaluation

First, let’s compare the full build time:

The dynamically-linked build is slightly slower than the statically-linked build. (approx. $0.15$ seconds)

Then let’s compare incremental build without actual code change:

The dynamically-linked build is slightly faster than the statically-linked build. (approx. $0.2$ seconds)

So dynamic-linking saves us about $11\%$ of build time in incremental case.

How about we actually change some code?

This time dynamic-linking saves us 0.25 seconds, or about merely $5\%$.

Conclusion

So the time saved from doing full dynamic linking is very little for me. But it might still have advantages for other people’s use cases.