Enhanced Interactive Generative Model Agent for CTF Challenges
Talor Abramovich1, Meet Udeshi2, Minghao Shao2, Kilian Lieret3, Haoran Xi2, Kimberly Milner2, Sofija Jancheska2, John Yang4, Carlos E. Jimenez3, Farshad Khorrami2, Prashanth Krishnamurthy2, Brendan Dolan-Gavitt2, Muhammad Shafique5, Karthik Narasimhan3, Ramesh Karri2, and Ofir Press3
Although language model (LM) agents are demonstrating growing potential in many domains, their success in cybersecurity has been limited due to simplistic design and the lack of fundamental features for this domain. We present EnIGMA, an LM agent for autonomously solving Capture The Flag (CTF) challenges. EnIGMA introduces new Agent-Computer Interfaces (ACIs) to improve the success rate on CTF challenges. We establish the novel Interactive Agent Tools concept, which enables LM agents to run interactive command-line utilities essential for these challenges. Empirical analysis of EnIGMA on over 350 CTF challenges from three different benchmarks indicates that providing a robust set of new tools with demonstration of their usage helps the LM solve complex problems and achieves state-of-the-art results on the NYU CTF and Intercode-CTF benchmarks, managing to solve more than three times more challenges of NYU CTF benchmark compared to previous best agent (the NYU CTF agent).
Want to try it yourself and explore our new agent? We are completely open-source! You can try it out in the SWE-agent repository, read our documentation and explore more about the research work in our paper.
Results
Benchmark | Model | % Solved |
---|---|---|
NYU CTF | EnIGMA w/ Claude 3.5 Sonnet | 13.5 |
EnIGMA w/ GPT-4 Turbo (1106) | 7.0 | |
EnIGMA w/ GPT-4o | 9.0 | |
NYU CTF agent w/ GPT-4 Turbo | 4.0 | |
InterCode-CTF | EnIGMA w/ Claude 3.5 Sonnet | 67.0 |
EnIGMA w/ GPT-4 Turbo (1106) | 72.0 | |
EnIGMA w/ GPT-4o | 69.0 | |
InterCode-CTF Agent | 40.0 | |
Google DeepMind Agent w/ Gemini Ultra | 24.0 | |
HackTheBox | EnIGMA w/ Claude 3.5 Sonnet | 26.0 |
EnIGMA w/ GPT-4 Turbo (1106) | 18.0 | |
EnIGMA w/ GPT-4o | 16.0 | |
NYU CTF agent w/ GPT-4 Turbo | 20.0 |
How it Works
Interactive Agent Tools In Action
BibTeX
Will be uploaded soon