What a 'Pen-Tester Agent' Taught Me About My SaaS Security — And What Portsmouth SMEs Should Ask Their Software Providers

Last Sunday morning, while most of Portsmouth was still in bed, a small piece of software running on my Mac mini fired off twenty test attacks against my own production website. It tried to read other customers' Stripe billing details. It tried to cancel their subscriptions. It tried to drain their AI credits. It tried every trick a junior attacker would think of.

It found eight problems. Twelve hours later, all eight were fixed.

The robot has a name — "Pax" — and I built it because I'd realised something uncomfortable: nobody else was looking. My SaaS — PICMS, an ISO compliance platform — has paying customers across the UK. Some of them store sensitive incident reports, hazard registers, and employee training records. If a security gap shipped on a Friday and nobody noticed until Monday, those customers were exposed all weekend.

The fix was simple: build a small agent that hammers the platform from outside, every week, automatically. Two days of engineering, ongoing cost £3 a month.

The interesting bit, for me, isn't that I built it. It's what it implies for every Portsmouth SME paying for software they don't control.

You're Trusting a Lot of SaaS — When Did You Last Ask?

Run a mental audit of your business right now. Count the SaaS subscriptions. Accounting software. CRM. Email marketing. Booking system. Payroll. Cloud storage. The ISO management system, if you've got one. Most Solent SMEs I work with are paying twelve to twenty different vendors monthly.

Each one of those holds a piece of your business. Customer names, addresses, payment details, employee records, supplier lists, financial data. If any of them gets breached, you're the one who has to explain it to the ICO — not the SaaS vendor. UK GDPR makes you the data controller; they're the data processor.

Yet how many of those vendors have you actually asked: "What does your security testing look like?"

The Three Questions Worth Asking

From building Pax, here's what I learned matters. Send these three to your most-critical SaaS provider this week and see how they answer:

1. "How often do you actively test for security regressions, and who does the testing?"

Good answers include: a third-party penetration test once a year, plus continuous internal testing (weekly or daily). Or: a structured vulnerability disclosure programme. Or — and this is rare but excellent — a documented automated security testing process with results visible to customers on request.

Bad answers: "We use HTTPS" (that's transport, not testing). "We have a security team" (great, what do they do?). Silence (run).

2. "Show me your last three security findings and how long they took to close out."

This question separates the talkers from the doers. A vendor that's actually doing the work has a triage system, a ticket trail, and timestamps. A vendor that says they do the work but actually doesn't will dodge.

The right answer might be uncomfortable for them — most vendors will admit to recent findings if pressed. That's a good sign. The vendor that claims zero findings ever is either lying or not looking.

3. "What happens to my data if I cancel?"

This isn't strictly a security question, but it sits right next to one. A vendor that can't easily extract your data is a vendor whose breach risk you inherit indefinitely. Look for vendors offering structured exports — JSON, CSV, or industry-standard formats. "We'll send you a PDF" doesn't count.

What "Good" Looks Like — A Working Example

I've been transparent with PICMS customers about the Pax build. The findings table is visible in the Master Admin dashboard. Critical issues automatically generate support tickets that our fix-attempt agent picks up within sixty seconds. The whole loop — detect, triage, fix, deploy, verify — runs in under twenty-four hours.

That's not the only model. A solo developer can do excellent security work without robots. A larger SaaS might have humans in a security operations centre. The form doesn't matter; what matters is that something is happening, on a schedule, and the results are visible.

If you ask your vendor and the answer reveals nothing is happening — that's not necessarily a deal-breaker. But it should change how you size the risk. A small SaaS provider with no security testing might still be fine for low-stakes data. The same provider managing your payroll? Different conversation.

What This Has to Do With AI Search

You might be wondering why a Solent Signal blog post is talking about pen-testing instead of AI search visibility. Two reasons.

First: when ChatGPT, Gemini, and Google AI Overviews recommend a business, they're increasingly factoring in trust signals — and structured security disclosures (a security.txt file, a published vulnerability disclosure policy, a clean record on Have I Been Pwned) are some of the cheapest, most overlooked signals available. Most Portsmouth SMEs aren't doing them. Five minutes of work moves the needle.

Second: AI assistants increasingly broker decisions for users. When a Portsmouth resident asks "is this booking platform safe?", the AI's answer comes from publicly available signals. Vendors who've thought about security and made it visible win. Vendors who haven't, lose.

The lesson from Pax isn't "build a robot." The lesson is: visibility into security matters now in ways it didn't five years ago. Asking your vendors hard questions, and being prepared to answer them yourself, is fast becoming a competitive edge in Portsmouth's small business market.

Where to Start This Week

• List your top three SaaS subscriptions by sensitivity of data held
• Email each one with the three questions above. Subject line: "Quick security review request — [your business name]"
• If you don't get a credible answer within a week, that's a data point worth noting
• For your own business: at minimum, check whether haveibeenpwned.com has any of your business email addresses listed in breaches. Free to check.

If you'd like to talk through your specific stack and what to ask whom, that's exactly the kind of conversation Solent Signal exists to have. Drop us a line — no obligation, no fee for an initial chat.