I Told Claude to CTF My Own Rails App

I wanted an outside eye on the Forcepull network without booking a pentest. So I opened Claude and gave it a deliberately adversarial brief: treat the codebase like a CTF target, find weaknesses, and write everything to a report file.

I picked up that framing after watching Nicholas Carlini's Black Hat talk on LLMs operating unprompted (on YouTube). The idea of letting a model drive its own security-oriented pass mapped neatly onto a Rails app I already knew too well to see with fresh eyes.

The prompt was roughly this:

You are playing in a CTF. Scan the application for vulnerabilities and write them to report.txt.

Not a substitute for a professional audit, but a useful way to stress-test assumptions. It came back with decent findings: things that were worth taking seriously, not just noise.

What showed up

The headline was broken access control. Company memberships have owner, admin, and member roles, but there was a way for anyone in the company to edit the company profile, not just people who should manage it. That is a classic IDOR-style gap.

There were also some notes on the passwordless login flow: specifically, per-IP rate limits, and how a distributed attacker might still parallelise across addresses. Not a fundamental design flaw, but tightening per-email limits alongside IP limits was an easy win.

From report to roadmap

After I worked through the list, I asked for suggested improvements beyond the raw bugs. One recommendation stood out: publish a security.txt file so researchers know where to report issues and what to expect.

I had been avoiding putting a raw mailto: in a static file that bots scrape, but I think a few spam emails are better than not having a decent security disclosure policy so I added [email protected] to /.well-known/security.txt. Next week I will probably add some sort of PGP option too, for serious players.

What I shipped

There is now a full security disclosure page with guidance on what to include, what to expect, a short disclosure policy, and the form itself. It mirrors the protections I already use on the public contact form: including signed tokens, rate limiting, and a dedicated email address. Reports are stored for review and show up in the admin area alongside other important items.

The loophole that allowed anyone in the company to edit the company profile is now closed. Login gets both IP-scoped and email-scoped rate limits on code request and verify.

What I would do differently next time

I would run a structured pass like this earlier in the project, not only after the network grew past 200 boards. The main issue claude detected was difficult to exploit, but it was there. It would have been better to catch it earlier. The other issues were not exotic, but they were easy to miss when I was shipping features every week.

If you maintain an app or website, try an AI-assisted light pentest

I recommend treating yourself to an occasional AI-powered light pentest on your own codebase. It is cheap, quick, and uncomfortable in a useful way. It does not replace hiring professionals or a real security programme, but it surfaces authorisation and workflow bugs that hide in plain sight when you wrote the code.

Start with the CTF framing that worked for me. Give the model enough context (your stack, important routes, models that gate access), and ask for a report file or a numbered checklist so you can tick fixes off.

You are playing in a CTF. Scan the application for vulnerabilities and write them to report.txt.

Other prompts worth rotating in (same session or later runs):

• Escalation from a normal account: "Assume you are a logged-in user with the lowest privilege level. How would you try to read or change another user, company, or tenant's data?"
• Authentication and sessions: "Review login, OTP or magic links, cookies, and rate limits. What attacks or failure modes are plausible?"
• Role matrix: "For each role, list actions that role can perform that the product probably did not intend."
• After you patch: "I fixed the following issues: [list]. Scan again for regressions, new gaps, or assumptions the fixes might have broken."

From now on, I will run something like this every few weeks, or after a large refactor or any change to authentication. The part that matters most: run it again after you fix what the first pass found. The second run is where missed edges and new mistakes are found.

If you want the talk that nudged me toward this in the first place, watch Nicholas Carlini's Black Hat session on LLMs operating unprompted. It is worth your time.

If you find something on Forcepull, use the security page. I will read every credible report and follow up promptly.

End of log.