skip to content
Jordan Madison
Table of Contents

TLDR: A take-home assignment asked me to design an RL environment. I misread the prompt and built something rooted in reverse engineering instead of the intended task. I rewrote it before submitting, but the misread version was interesting enough to document here.

Introduction

My interests sit at the intersection of cybersecurity, privacy, intelligent programs, and reverse engineering. When I received a take-home assignment to build an RL environment, I initially misread the constraints: I had built a materials-science RL environment rather than the intended task.

In Spring 2026, I took several advanced cybersecurity courses, including Malware Reverse Engineering.

Throughout the project, my partner and I reverse engineered Windows malware samples of increasing complexity. The samples ranged from Michelangelo.1 and DOS7 to SQLSlammer, Lucious, and Harulf. Working through these binaries made the gap between static analysis and automated reasoning especially clear.

Environment Building

Throughout this journey, I stumbled upon something called Polymorphic Decryptor Engine. In plain terms, malware authors are devious artists. They not only understand a system deep enough to exploit it, but also design programs in such a way to evade reverse engineers. One evasion technique is to use encryption to change sections of the binary to hide its true intent, and this is usually done with a polymorphic decryptor engine. It is a subroutine in malware that produces unique decryptors for each malware sample. Finding the decryptor within a malware payload can be a genuinely hard task, resistant to automation, but it’s easily checkable with ground truth.

To build environments, we need a couple of things:

  • Prompt
  • Data/Tools
  • Judge

Prompt

Your task is to train a classifier that is able to locate the region(s) of a binary that contains the polymorphic decryptor stub. The polymorphic decryptor stub is a mutable piece of code responsible for decrypting the main payload. For data, you have access to the Malware Binary Dataset in the folders /train_malware_samples to train the classifier and /test_malware_samples to test the Classifier. For tools, you have access to a disassembler, Python scripting, and decompiling through a headless version of Ghidra, a tool to analyze the entropy of different segments of the binary, and a hex bytes checker. When you are done, you should create a file /model.py which contains a function load_model() that takes no arguments, and returns a Malware Binary Classifier of type torch.nn.Module.

For simplicity, this scope will be static analysis, which means not executing the binary and just looking over the bytes. Dynamic analysis, although realistic, is a different can of worms where you’ll have to manage runtime behavior, explosion in complexity, etc.

Tools

To simulate human analyst conditions, the LLM would have access to:

  • Headless version of Ghidra (includes a Disassembler, decompiler, and Python scripting functionality)
  • Entropy Measurement window
  • Hex and Byte comparison tools

Data

To create the dataset, we will build a synthetic data generator to create binary samples with a curriculum defined by the subroutines’ locations in memory and the size of the binary. The output will be a dataset of binary samples with labels of the correct regions within the sample.

The judge will have access to a folder called /eval_malware_samples to serve as the held-out set.

Judge

The judge performs the following steps: (1) Verifies the file /model.py exists. If not, fail. (2) Check to see if load_model() function exists and has right arguments. If not, fail. (3) Evaluate the Intersection Over Union (IOU) of the model on the dataset within /eval_malware_samples. IOU will highlight the model’s precision in selecting the correct regions. Overselection means the agent’s spray pattern is wide, while Underselection means the agent guessed the wrong region with no overlap. Return a numerical score of the average IOU across /eval_malware_samples.

One possible reward hacking behavior is the spray cheat, which is where an oversized region is produced to ensure IOU overlap without covering the exact upper and lower bounds.

Conclusion

I designed this environment because its combination of difficulty, a large search space, and measurable ground truth makes it an interesting task for an agent. It may also support novel heuristics for malware forensics.

This misread version was worth documenting because it shows how a technical background can shape the design of an RL environment. The same approach—define a difficult task, expose useful tools, and make success measurable—generalizes beyond malware analysis.