There is a particular sort of confidence that only a successful technology demonstration can produce. The questions are sensible, the answers are excellent, the network behaves, and nobody accidentally shares the wrong screen.
For twenty minutes, the future has arrived.
Then the meeting ends and the future meets a customer.
I like a good demonstration. It helps people understand what a system might do for them. The trouble starts when we mistake that understanding for evidence that the system can withstand people doing things its designers did not expect.
A demo is usually organized around success. Security testing needs room for failure, confusion, and deliberate interference. That can make the second meeting rather less comfortable than the first but considerably more useful.
With AI, it is tempting to reduce this to collecting amusing jailbreaks. Ask the assistant to ignore its instructions, persuade it to say something embarrassing, and take a screenshot. There is value in discovering those weaknesses. There is also a danger that we measure how entertaining the failure looks instead of how much it matters.
For a business deploying an AI service, the important question is what an attacker could achieve.
Consider a hypothetical assistant that searches a company knowledge base. Its job is to help employees find approved information. A meaningful test might ask whether an ordinary employee can obtain another customer’s confidential material. Another might ask whether a retrieved document can redirect the assistant towards an unauthorized task.
OWASP distinguishes instructions supplied directly by a user from those arriving through external material such as files or websites. It also warns that retrieval-augmented generation does not fully remove prompt injection risk. Connecting a model to your documents does not make every document a trustworthy instruction. Source: OWASP, Prompt Injection.
That is why the test should follow the route the real system takes. Include its retrieval, identities, tools, and approval steps. If it accepts images or audio, evaluate those routes too. A text-only assessment cannot tell you much about a path it never exercised.
I would write the business failure first, in plain English. “An employee must not retrieve records belonging to another customer” is something people can discuss. “We achieved a score of ninety-three” invites a much more awkward question about what happened to the other seven.
Next, define what counts as success for the attacker. An attempted tool call may be important even when the backend rejects it. A final answer saying “I cannot do that” needs checking against what the workflow actually did beforehand. Keep those outcomes separate so the report tells a useful story.
Repeated testing matters too. One unsuccessful attempt tells us that one attempt failed under particular conditions. We need variations in wording, context, and conversation length, including attempts that develop over several exchanges. We should record the configuration so somebody else can investigate the result.
Automation helps explore more of those variations. Human judgment is still needed to decide whether the scenarios represent the business, whether a reported failure is genuine, and which consequences deserve attention first.
There is another test that is easily forgotten: can legitimate users still get their work done after we add the controls? A service that refuses every request may appear admirably cautious. It will also encourage people to find a different service, possibly using a personal account and considerably less caution.
The output of testing should therefore support a decision. Explain the exposed business process, the demonstrated failure, the relevant limitations, and the proposed fix. Give the work an owner and agree on how to verify the improvement. Screenshots can come along, provided they have something useful to say.
A convincing demo earns the next conversation. A credible evaluation helps decide whether the system earns access to the business.
I would quite like both meetings in the diary.