Within the abyss of assembly lies an RL environment

TLDR: I recently took a take-home that asked me to design an RL environment. I misread the prompt and built something rooted in my reverse-engineering background instead of the intended task. Rewrote it before I submitted, but I liked the misread version enough to give it its own write-up. I’m also writing this freehand because I’m tired of AI writing, so if it’s cooked, it is what it is. I’m writing from the chest.

Introduction

I have a weird background. When I utter the words “Cybersecurity, Privacy, and Intelligence”, people’s minds immediately go to some 007 classified lifestyle when in reality, I just like the math behind intelligent programs, and reverse engineering systems/people is fun. So, when I got this take-home assignment to build an RL Environment, I was initially confused because while yes, I’ve built a Materials Science RL environment, the constraints on this one were different. Anyways, like the best moments of my life, I full send it without reading the full instructions.

The backstory to this is that in Spring 2026, I took all cybersecurity coursework to fulfill my degree requirement. Why did I do that? Again, my background is weird, so I had only challenging senior-level classes left (I started GT back in 2015, left in 2019 for reasons (?), floated in the wind until I returned in 2025 and finished my degree). My schedule included a Research Writing class, Cybersecurity of Drones, Infosec, and Malware Reverse Engineering. Yes, this schedule tested my sanity, but one class in particular tested everything under the sun.

Throughout the semester, my partner and I had to reverse engineer Windows malware samples of increasing complexity one week at a time. The samples ranged from malware such as Michelangelo.1 to DOS7 to SQLSlammer to Lucious to Harulf. I spent 30 hours/week commenting malware binaries line by line, praying to IDA Pro and occasionally OllyDbg gods to grant me insight into the abyss of assembly so I could sleep and never look at a binary again (I occassionaly still open IDA Pro up and poke to this day).

Environment Building

Throughout this journey, I stumbled upon something called Polymorphic Decryptor Engine. In plain terms, malware authors are devious artists. They not only understand a system deep enough to exploit it, but also design programs in such a way to evade reverse engineers. One evasion technique is to use encryption to change sections of the binary to hide its true intent, and this is usually done with a polymorphic decryptor engine. It is a subroutine in malware that produces unique decryptors for each malware sample. Finding the decryptor within a malware payload can be a genuinely hard task, resistant to automation, but it’s easily checkable with ground truth.

To build environments, we need a couple of things:

• Prompt
• Data/Tools
• Judge

Prompt

Your task is to train a classifier that is able to locate the region(s) of a binary that contains the polymorphic decryptor stub. The polymorphic decryptor stub is a mutable piece of code responsible for decrypting the main payload. For data, you have access to the Malware Binary Dataset in the folders /train_malware_samples to train the classifier and /test_malware_samples to test the Classifier. For tools, you have access to a disassembler, Python scripting, and decompiling through a headless version of Ghidra, a tool to analyze the entropy of different segments of the binary, and a hex bytes checker. When you are done, you should create a file /model.py which contains a function load_model() that takes no arguments, and returns a Malware Binary Classifier of type torch.nn.Module.

For simplicity, this scope will be static analysis, which means not executing the binary and just looking over the bytes. Dynamic analysis, although realistic, is a different can of worms where you’ll have to manage runtime behavior, explosion in complexity, etc.

Tools

To simulate human analyst conditions, the LLM would have access to:

• Headless version of Ghidra (includes a Disassembler, decompiler, and Python scripting functionality)
• Entropy Measurement window
• Hex and Byte comparison tools

Data

To create the dataset, we will build a synthetic data generator to create binary samples with a curriculum defined by the subroutines’ locations in memory and the size of the binary. The output will be a dataset of binary samples with labels of the correct regions within the sample.

The judge will have access to a folder called /eval_malware_samples to serve as the held-out set.

Judge

The judge performs the following steps: (1) Verifies the file /model.py exists. If not, fail. (2) Check to see if load_model() function exists and has right arguments. If not, fail. (3) Evaluate the Intersection Over Union (IOU) of the model on the dataset within /eval_malware_samples. IOU will highlight the model’s precision in selecting the correct regions. Overselection means the agent’s spray pattern is wide, while Underselection means the agent guessed the wrong region with no overlap. Return a numerical score of the average IOU across /eval_malware_samples.

One possible reward hacking behavior is the spray cheat, which is where an oversized region is produced to ensure IOU overlap without covering the exact upper and lower bounds.

Conclusion

I designed this environment because the task was near and dear to my heart, so I thought it would be an interesting task to give to an agent. The task is hard, non-automatable, has a vast search space, and potentially may lead to some novel heuristics to help with malware forensics down the line.

Hopefully you enjoyed this write-up. It was an off-the-cuff inspo I regret not submitting as my first submission, but everything happens for a reason. I’ll be writing more using this style, so till next time. Peace