I have a decent working local LLM setup working and its been great for home assistant, coding questions, and debugging. That said I want to take it a step farther. I know of Agentic AI and have set up some basic things with it, but I want to do more complex tasks. I’ve heard Wendell talk about turnstone and I’ve taken a look at it but feel a bit out of my depth and could use some help just wrapping my head around what it does and how I can leverage it.
Let me start by stating my overall goals. I’m a pentester by trade, and like most pentesters do CTFs for fun and to keep my skills sharp. I’m not looking to replace what I do with AI, more just accelerate myself and let the AI do the trivial and boring things in order to jump right into the meat and potatoes of a challenge. I was perfectly fine doing all that myself when its just for fun, but I was at Defcon with my work team and like every year we participated in the red team village CTF. We normally do pretty good, have traditionally placed in the top 10 and a couple of years broke top 5. The issue is this year the top 10 was already decided 4 hours into the event because the top teams all just used AI agents to solve the problems for them. I personally hate this. Turns a test of skill and ability into a pay to win game and that feels completely antithetical to the hacker spirit.
That said, the organizers basically said, “welp that’s the world we live in now, so you either get good at using tokens and ai tools or you don’t place.” I don’t really care if we place high or not, but the fear is if we don’t bring back impressive placements in CTFs then the company may stop sending us to Defcon, which of course we want to avoid at all costs.
So instead of buying tokens in a public cloud I figured it would be more fun to figure out how to set up my own environment for doing these sorts of challenges so we can stay competitive. The three main categories I want to get an AI setup good at solving are web exploitation, reverse engineering, and binary exploitation. This covers the vast majority of challenges the Red Team Village set up every year.
So with all that background, and knowing I’m technical but haven’t delved too deep into AI stuff yet, where should I start when it comes to setting up the AI environment and AI tooling to solve these challenges?
I would argue the first step is defining the repetitive tasks involved with the CTF challenges. I am not a pentester by any means, but the framework followed for other applications is the same.
OSINT, for example, starts with bulk data collection and ingestion. Kicking off a spiderfoot scan is boilerplate, but I would have to define what a scan should look like and build tools and skills in a harness like Turnstone to nudge the LLM in the correct direction. The agent could then surface good pivot candidates to be approved or denied by me. It also is beneficial to have an LLM help create tooling specific to the needs of the investigation.
A heuristic I follow that has served me well is: given a set of rules and data, an LLM can play a given game well, but they cannot create a new game.
Also, where a script is suitable for your workflow, develop a script. I got myself stuck plugging LLMs in for LLMs sake. Happy to brainstorm further
sure, I’ve built many a script (and more complicated programs) for pentesting. Currently working on tetanus which is a pentest automation, project management, and note generation tool that also has C2 and collaboration functionality. It’s still very much in the early days of development, I’ve had the idea kicking around in my head for about 6 years, and have started, deleted and restarted 4 times because I hated the structure it ended up becoming, but now I think I’ve got a good base, the attack client is usable, mainly for the project management side, and there is a bare bones server implementation built right into the primary binary, as well as very basic implant binary generation, though there’s no real obfuscation happening on the implant yet so while it gets past some EDRs, its pretty easy to reverse and get things out of it like the cert used to TLS encrypt network coms, and the password for the relay server (I’m still working on figuring out the best way to obfuscate it.) Despite that it does get past some EDRs already… mainly because some EDRs are still REALLY bad at doing behavioral analytics on rust binaries, but it is showing some promise. here’s the code if you’re curious about it ( pyro/tetanus: Rust based pentesting tool. Will work as an implant generator, c2 server, c2 client, distrobox manager, engagement data management, and markdown note generator (and maybe editor?). - tetanus - Gitea: Git with a cup of tea )
I do have ollama set up with odysseus on my home server, and have it connected to hermes desktop on my hacking laptop, and it’s great for helping me debug my code or learn new concepts that I dig into. I have tried to get qwen3 to do some recon for me just using the search tool, but struggle with getting it to actually do the recon due to guard rails. Is there a way to turn the guardrails off?