Obfuscating Rust Binaries using LLVM Obfuscator (OLLVM)

UPDATE - 30 November 2023

I have made available a Docker image containing all the required steps to build a Rust toolchain using OLLVM. Currently it is targeting both x86_64-unknown-linux-gnu and x86_64-pc-windows-gnu but haven’t tested yet. At least it works for “hello world” programs.

Source: https://github.com/joaovarelas/Obfuscator-LLVM-16.0

NOTE: You are going to need at least 30GB of disk space and patience to compile LLVM 16.

git clone https://github.com/joaovarelas/Obfuscator-LLVM-16.0 && cd Obfuscator-LLVM-16.0
docker build -t rustc-ollvm .
docker run -v /path/to/my/cargo/projects:/projects/ -it rustc-ollvm:latest /bin/bash

Then inside the container:

cd /projects/myproject/
RUSTCFLAGS="-Cllvm-args=-enable-allobf" cargo +ollvm-rust-1.70.0 build --release

The executables will be placed at target/.

Credits to original author https://bbs.kanxue.com/thread-274453.htm.

Introduction

Reverse engineering is the process of analyzing and examinating a product or system to understand their inner workings, extract sensitive information, or modify their behavior. However, Rust compiled binaries present unique challenges due to its design and complexity.

Rust utilizes LLVM (Low-Level Virtual Machine) as its backend compiler infrastructure, which provides powerful optimization and code generation capabilities, enabling efficient execution of Rust code across different platforms and architectures. By leveraging LLVM’s custom passes (such as the Obfuscator LLVM) can further enhance the obfuscation of Rust binaries, making them even more difficult to analyze and reverse engineer.

Complex malware can be a nightmare for analysts and incident responders as it requires a lot of time to examine, which can be critical when responding to an incident. Red teamers on the other side might gain an advantage against blue teams by protecting their tools (e.g. C2 beacon) both to evade antimalware solutions and difficult analysts job while performing adversary emulation or intrusion testing.

(De)compiling “Hello World”

Rust cargo is the package manager and build tool for the Rust programming language. It simplifies the process of managing dependencies, building projects, and running tests. With cargo, developers can easily create, share, and publish their Rust libraries and applications.

  
$ rustc --version --verbose
rustc 1.71.0-nightly (4a59ba4d5 2023-05-12)

$ cargo --version --verbose
cargo 1.71.0-nightly (13413c64f 2023-05-10)

Default Configuration

As an example, the following hello.rs program will be compiled using a nightly version 1.71.0 [1] at the moment of writing. A nightly toolchain may be installed using the rustup installer.

  
pub fn main(){
	println!("hello world");
}

  
$ rustc hello.rs -o hello

$ ls hello
-rwxr-xr-x 1 vrls vrls 4.1M Jun 13 20:17 hello

$ file hello
hello: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked,
interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=7d399b3[...]817e24d9a, 
for GNU/Linux 3.2.0, with debug_info, not stripped

A compiled hello world ELF program using the default configuration is approximately 4.1MB, not stripped and contains debug info. The disassembly and pseudo code looks like the following, which is not that bad since we can easily follow the code and understand the logic (considering the simplicity of the code):

Stripped Release

Now adjusting the compiler settings will make the produced executable a little harder to understand since there will be no debug info. This setting is ideal to use on production environment and released binaries.

The rustc codegen options [2] -Cdebuginfo=0 -Cstrip=symbols -Cpanic=abort -Coptlevel=3 optimize the compilation process by excluding debugging information and symbol tables, resulting in a smaller binary. The panic=abort flag ensures immediate program termination upon encountering a panic, without unwinding the stack.

  
$ rustc -Cdebuginfo=0 -Cstrip=symbols -Cpanic=abort -Copt-level=3 hello.rs -o hello-stripped

$ ls hello*
-rwxr-xr-x 1 vrls vrls 4.1M Jun 13 20:17 hello
-rwxr-xr-x 1 vrls vrls 331K Jun 13 20:18 hello-stripped


$ file hello*
hello: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, 
interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=7d399b3[...]817e24d9a, for GNU/Linux 3.2.0, 
with debug_info, not stripped

hello-stripped: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, 
interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=f786e67[....]6706ec0b2d, 
for GNU/Linux 3.2.0, stripped

The resulting binary is slightly different when compared with the previous. There are a lot of other configurations that can be leveraged to optimize the resulting binary.

Obfuscator LLVM

LLVM Obfuscator [3] is a modified version of LLVM that applies transformations to LLVM intermediate representation (IR) code in order to make it more difficult to understand and reverse engineer. By employing techniques like code obfuscation, control flow flattening, and variable substitution, LLVM Obfuscator aims to increase the complexity and confusion in the generated executable, enhancing the protection of the project and preventing unauthorized code analysis.

Given that Rust uses LLVM to generate MIR and LLVM-IR [4], its possible to write custom optimization passes such as OLLVM that will transform the intermediate representation code generated by Rust.

Compiling OLLVM

Note that original OLLVM project [3] is quite old and the latest version was built on top of LLVM 4.0 released more than 6 years ago. Currently the Rust-fork of LLVM is on version 16.0.2.

OLLVM forks may be found on open-source code repositories such as [5, 6, 7 ,8]. Interestingly enough, this project seems very common among chinese developers including native mobile apps developers and reverse engineers. Most of the custom passes found online have documentation in chinese language.

Additionaly OLLVM was referenced in game hacking communities used by players to protect their cheats and evade anti-cheat systems thus avoiding being banned from the game. For example, in Unknown-Cheats forum there is a guide [9] on how to integrate OLLVM with Microsoft Visual Studio to aid cheat development and protection.

Prerequisites

Rust source-code: https://github.com/rust-lang/rust (1.70.0)
Rust LLVM fork source-code: https://github.com/rust-lang/llvm-project (16.0.0-2023-03-06)
Fork of ObfuscatorLLVM (version should match Rust LLVM):
- Hikari-LLVM [8] has branch for LLVM 16.0 and its compatible with current Rust versions. Feel free to implement your own passes.
Build tools such as cmake, Ninja, git, clang, python3 and so on
Enough storage as the source will easily grow from 15GB to 20GB on disk

NOTE: This demo will focus on building a custom Rust OLLVM toolchain to obfuscate the binaries on Linux (ELF). The original article by ny0c [10] is based on LLVM 14.0 and uses MSYS2 based on MinGW for Windows, have a look if you want to build Windows executables.

Start by cloning the required sources:

  
$ git clone --single-branch --branch 1.70.0 --depth 1 https://github.com/rust-lang/rust rust-1.70.0
$ git clone --single-branch --branch rustc/16.0-2023-03-06 --depth 1 https://github.com/rust-lang/llvm-project llvm-16.0-2023-03-06
$ git clone --single-branch --branch llvm-16.0.0rel --recursive --depth 1 https://github.com/61bcdefg/Hikari-LLVM15 ollvm-16.0

We want to add obfuscation passes into the Rust LLVM fork to make it compatible with compiler. By doing a diff it’s trivial to compare which files are different and then apply a patch without damaging the original source.

$ git diff llvm-16.0-2023-03-06/llvm ollvm-16.0/llvm/ > llvm.patch

Output from diff will include a lot of tests and unit-test files that are not really interesting so we might manually remove them. Once we get a cleared version of the diff we can use it to patch the Rust LLVM fork with the Obfuscation module from OLLVM. The ideia is to apply the OLLVM patch with minimal modification of Rust LLVM to avoid incompatibility, dependency problems or bugs.

In resume, the important modifications to apply from the OLLVM on Rust LLVM fork are the following:

  
$ cat llvm.patch | grep -iE "^diff " | grep -ivE "/llvm/test/|/llvm/utils/lit/tests/|/llvm/unittests/" | cut -d '/' -f 3-  | cut -d ' ' -f 1

llvm/include/llvm/InitializePasses.h
llvm/include/llvm/LinkAllPasses.h
llvm/include/llvm/MC/MCSubtargetInfo.h
llvm/include/llvm/Transforms/Obfuscation/[...]
llvm/lib/IR/AutoUpgrade.cpp (*)
llvm/lib/Passes/PassBuilderPipelines.cpp
llvm/lib/Support/Unix/Path.inc (*)
llvm/lib/Target/X86/X86ISelLowering.cpp
llvm/lib/Target/X86/X86InstrSSE.td (*)
llvm/lib/Transforms/CMakeLists.txt
llvm/lib/Transforms/IPO/CMakeLists.txt
llvm/lib/Transforms/IPO/PassManagerBuilder.cpp
llvm/lib/Transforms/InstCombine/InstCombineCompares.cpp (**)
llvm/lib/Transforms/Obfuscation/[...]
llvm/tools/xcode-toolchain/CMakeLists.txt

(*) should not be patched and remain the same as Rust LLVM
(**) should be partially patched by adding lines, but not removing existing ones

Finally apply the patch and fix any error that may occur by looking for *.rej files, they are automatically created when conflicts are detected.

  
$ cd llvm-16.0-2023-03-06/
$ git apply --reject --ignore-whitespace ../llvm.patch
$ find . -name "*.rej" # no output, means we are good

Building

  
$ cd llvm-16.0-2023-03-06/
$ mkdir build && cd build/
$ cmake -G "Ninja" ../llvm -DCMAKE_INSTALL_PREFIX="./llvm_x64" -DCMAKE_CXX_STANDARD=17 \
  -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;lld;" -DLLVM_TARGETS_TO_BUILD="X86" \
  -DBUILD_SHARED_LIBS=ON -DLLVM_INSTALL_UTILS=ON -DLLVM_INCLUDE_TESTS=OFF -DLLVM_BUILD_TESTS=OFF \
  -DLLVM_INCLUDE_BENCHMARKS=OFF -DLLVM_BUILD_BENCHMARKS=OFF -DLLVM_INCLUDE_EXAMPLES=OFF \
  -DLLVM_ENABLE_BACKTRACES=OFF -DLLVM_BUILD_DOCS=OFF  -DBUILD_SHARED_LIBS=OFF \
  -DCMAKE_CXX_COMPILER=clang++-16 -DCMAKE_C_COMPILER=clang-16

  
$ cmake --build . -j8 # will take some time depending on hardware... go grab a redbull
$ cmake --install .

Once its built and installed we can check if the binaries are working as expected. In this case, it was installed inside the build/ directory inside the llvm_x64 as specified previously by -DCMAKE_INSTALL_PREFIX.

Note that install location will be used in the future by rustc to use our custom LLVM infrastructure.

  
$ ./llvm_x64/bin/llvm-config  --version
16.0.0

$ ./llvm_x64/bin/clang --version
clang version 16.0.0 (https://github.com/rust-lang/llvm-project 2b9c52f66815bb8d6ea74a4b26df3410602be9b0)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /mnt/VMs/ollvm/llvm-16.0-2023-03-06/build/./llvm_x64/bin

Verify that build/llvm_x64/lib/ directory contains *.a static lib files, in particular libLLVMObfuscation.a. Also, save the absolute path for build/llvm_x64/bin/llvm-config as it will be needed next.

Bootstrapping Rust Compiler

Having the OLLVM ready, it’s time to move into Rust compiler. Rust compiler bootstrapping involves building the compiler using another language (e.g. C) and then using the compiler to compile a self-hosted version, resulting in a final Rust compiler.

This process consists of 3 stages:

Stage 0: an initial version of the Rust compiler is built using Rust - initially OCaml was used [11].
Stage 1: the compiler from Stage 0 is used to compile a self-hosted version of the compiler.
Stage 2: the self-hosted compiler is used to compile the final version of the Rust compiler, which can be used for further development.

Before starting the build of Rust compiler, we should adjust config.toml file to specify the modified LLVM we just built. This can be defined by setting llvm-config = /path/to/llvm-config.

Copy config.example.tomlto config.toml
Edit config.toml
In [rust] section, set debug = false
In [rust] section, set channel = "nightly"
In [target.x86_64-unknown-linux-gnu] section, set llvm-config = "/path/to/ollvm/bin/llvm-config"
- If you are targeting a different system, make sure to rename section [target.x86_64-unknown-linux-gnu] to [target.x86_64-pc-windows-gnu] for example.

Once the configuration is done, it’s time to build the Rust compiler. Currently Rust provides a build script named x.py that simplifies the process of bootstrapping. To build just execute:

  
$ cd rust-1.70.0/
$ python3 x.py build # will take some time... 

Observe that during assembly of stage1 compiler will already make use of OLLVM (however with obfuscation flags disabled, as they are not needed at this point yet). Once finished, check if rustc was successfully compiled:

  
$ ./build/x86_64-unknown-linux-gnu/stage1/bin/rustc --version --verbose

rustc 1.70.0-nightly (90c541806 2023-05-31)
binary: rustc
commit-hash: 90c541806f23a127002de5b4038be731ba1458ca
commit-date: 2023-05-31
host: x86_64-unknown-linux-gnu
release: 1.70.0-nightly
LLVM version: 16.0.0

Finally, the cargo must be built too. This can be accomplished by using the x.py tool once again:

  
$ python3 x.py build tools/cargo
$ ./build/x86_64-unknown-linux-gnu/stage1-tools-bin/cargo --version
cargo 1.70.0-nightly (ec8a8a0ca 2023-04-25)

Add Custom Toolchain

The rustc and cargo can be added as a custom toolchain to the Rust setup. The toolchain allows to easily switch between different versions of Rust when developing projects.

  
$ rustup toolchain list
stable-x86_64-pc-windows-gnu
stable-x86_64-unknown-linux-gnu
nightly-x86_64-unknown-linux-gnu (default)

$ rustup toolchain link ollvm-rust-1.70.0 /mnt/VMs/ollvm/rust-1.70.0/build/x86_64-unknown-linux-gnu/stage1/

$ rustup toolchain list
stable-x86_64-pc-windows-gnu
stable-x86_64-unknown-linux-gnu
nightly-x86_64-unknown-linux-gnu (default)
ollvm-rust-1.70.0 <---------------------------- the new toolchain

Results

The fresh toolchain can now be selected to compile Rust programs through custom LLVM including obfuscation apsses. Unless it is set as default (using rustup default ollvm-rust-1.70.0) it must be specified when executing rustc or cargo by adding the format +toolchain.

Going back to the initial hello.rs example, lets recompile it using the new OLLVM toolchain:

$ rustc +ollvm-rust-1.70.0 hello.rs -o hello-ollvm

The command will work but none of the obfuscation techniques were enabled. Looking at the OLLVM passes configuration or Hikari documentation Wiki, we can find the available flags:

Flags should be passed to LLVM. The Rust compiler allows to pass flags to LLVM via codegen llvm-args [12]. For simplicy let’s use -enable-allobf to enable all the features available:

  
$ rustc +ollvm-rust-1.70.0 -Cllvm-args=-enable-allobf hello.rs -o hello-ollvm

$ ls hello-ollvm 
-rwxr-xr-x 1 vrls vrls 482K Jun 15 21:23 hello-ollvm

$ file hello-ollvm 
hello-ollvm: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, 
interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=4cbc70187af8[..]]2fe42ca921c4, 
for GNU/Linux 3.2.0, not stripped

$ ./hello-ollvm 
hello world

Symbols are still present on this executable since it was not compile on release mode and they may be removed by specifying additional compiler flags (stripped):

  
$ rustc +ollvm-rust-1.70.0 -Cllvm-args=-enable-allobf -Cdebuginfo=0 \
    -Cstrip=symbols -Cpanic=abort -Copt-level=3 hello.rs -o hello-ollvm-strip

$ ls hello-ollvm-strip 
-rwxr-xr-x 1 vrls vrls 367K Jun 15 21:25 hello-ollvm-strip

$ file hello-ollvm-strip 
hello-ollvm-strip: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, 
interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=ad38d6ff[...]]849e395, 
for GNU/Linux 3.2.0, stripped

$ ./hello-ollvm-strip
hello world

Resulting disassembly is now much harder to follow and requires additional attention when reversing this program. The simple “hello world” is now obfuscated and difficult to understand.

Example of main function of program compiled with default settings (debug info) VS. OLLVM stripped:

Conclusion

This approach describes how to integrate the known LLVM Obfuscator project into an updated Rust toolchain and allow developers to obfuscate and protect their software by applying known techniques such as control flow flattening, bogus control flow and string encryption and so on.

Additionally, it is a good baseline for those seeking to add their own custom passes (i.e. optimization methods) and introduce newer and better obfuscation techniques. Other languages besides Rust such as C/C++ (Clang), Golang and Swift also make use of LLVM infrastructure, giving the opportunity to use OLLVM between different programming languages and potentially cross-platform. Some other interesting projects also focus on code protection such as Rust goldberg [13] and obfstr [14].

Modern obfuscation mechanisms include code virtualization used to protect software by transforming its code into a virtualized form and or bytecode that is executed by a virtual machine or interpreter. A proof-of-concept can be found at xVMP repository [15]. The respective paper is available at [16] with the title xVMP: An LLVM-based Code Virtualization Obfuscator which improves current obfuscation methods based on LLVM.

From the other perspective, there has been a focus on finding better approaches to reverse Rust binaries. Essentially compiled programs on C/C++ and Rust have some similarities and differences considering the resulting binary executables. There is a really good article by Checkpoint Research [17] that gives a solid overview on reversing Rust binaries.

Source Code

Patch for Rust-LLVM is available on my repository:

https://github.com/joaovarelas/Obfuscator-LLVM-16.0/

It should be manually applied when compiling LLVM and Rust thereafter. Feel free to reach out on Discord: vrls or open an issue on GitHub repo if you find any difficulty reproducing the steps.