Contents · 3 sections
Part of What Is Jev? Six Hands-On Experiments With TypeSafe's System One Model. Every number here comes from a saved run you can download at the end of the post.
This one is about our own product. We had a hunch that some people were writing into the prompter as if it were a chat, expecting an answer back, and we knew that a small number of requests were the kind an image provider refuses. What we didn't have was numbers. So we took every human-typed prompt from the web app since May (148 of them, from 23 people, agents and API traffic excluded) and asked ten questions about each.
A note on privacy: the prompts stayed on our machine, were anonymized before analysis, and this article only reports aggregates and a few harmless examples. Nothing anyone typed is reproduced beyond that.
Setup
state: { tool: "an AI image generation and photo editing studio", where_typed, user_text }
questions:
form: Choice image_description · order_to_assistant · question · chat · feedback_on_result · other
expects_text_reply: Noul "Written as if expecting a written reply from an assistant?"
refers_to_previous: Noul "Refers to a previous result as if the system had memory?"
needs_text_answer: Noul "Would a satisfying response need text rather than an image?"
brief_quality: Score ["nothing to generate", "vague", "usable", "detailed"]
frustration: Score ["neutral", "mildly annoyed", "clearly frustrated"]
language: Choice en · es · pt · other
policy: Choice none · sexual_explicit · sexual_suggestive · undress_real_person · minors · violence_gore · hate_harassment · other_prohibited
violates_policy: Noul "Would this likely trigger a provider safety block?"
evasive: Noul "Is the wording trying to sneak something past a filter?"Results
Are people talking to a chatbot?
Less than we feared, and more concentrated than we expected. Eleven percent of prompts read as if the person expected a written reply, and they came from three of the twenty-three people. One of them conversed with the prompter in nine of seventeen prompts ("the 'live' text isn't very legible, can you fix it?"). Another pasted a whole project description four times, waiting for a conversation that never came.
The bigger finding was about memory rather than chat: 22% of prompts referred to a previous result as if it were still in context ("same position, don't change any detail of the character, just make him…"), and eighteen of those were typed in a mode where no image is attached. Those requests fail quietly today. Nobody sounded frustrated (zero prompts scored above 1.5 on the frustration scale); they simply didn't get what they expected.
Requests a provider would refuse
Eighteen percent of prompts were flagged by either signal, from nine people. The explicit ones scored 0.93 to 0.96 on block risk with their category at 97 to 100%. Requests to undress or sexualize a person in an uploaded photo were correctly placed in their own category (79 to 83%). Nobody disguised anything: the evasion question never went above 0.5.
Two things worth knowing. First, the model was soft on two prompts our provider actually blocked, a suggestive prompt written in Chinese; it scored 0.34 on risk but still put the category at "suggestive 52%", so reading both signals catches it. Second, one product-photo edit phrased as "remove the red cap from the image and reveal what's under it" tripped the undress category at 53% while the risk score stayed at 0.38. The category and the risk score disagree on edge cases, and both should be read.
The pre-flight check
From those numbers we wrote four rules, and they now run as a check before any prompt reaches a provider. One call, ten questions, about 190 milliseconds and five thousandths of a cent per prompt.
- Block and count a strike: block risk above 0.75, or an explicit, undress or minors category above 70%. On our data this caught 15 of the 27 flagged prompts, including everything explicit.
- Allow but log a soft strike: risk in the grey zone, or any flagged category above 50%. This is how the Chinese prompt gets caught.
- Retrying a blocked prompt word for word counts as an extra strike. One person had sent the same explicit prompt five times.
- In quick mode, if the prompt reads as a message to an assistant or refers to a previous result, don't spend credits: show "this returns images only; to edit the last result use Restyle". About twenty prompts would have gotten that message instead of a silent miss.
The thresholds come from 148 prompts and nine flagged people, so we'll log the scores for a month and tune them. What we like about the design is that the model never decides anything on its own; it hands back probabilities, and four lines of plain code decide.
