Contents · 3 sections
Part of What Is Jev? Six Hands-On Experiments With TypeSafe's System One Model. Every number here comes from a saved run you can download at the end of the post.
Reviews are messy: typos, mixed languages, a title that says one thing and a body that says another. They're also plentiful, which makes them a good test of a model that's meant to be fast and cheap. We took the hundred latest US reviews of Spotify and the fifty latest of Instagram from Apple's public feed and asked five questions about each.
Setup
state: { app, title, review, stars, version }
questions:
type: Choice bug · request · praise · complaint · question · other
severity: Score ["no problem", "minor annoyance", "serious: loses features, time or money", "can't use the app"]
blames_update: Noul "Does it blame a recent update?"
area: Choice performance · ui · billing · account · content · ads · support · other
language: Choice en · es · pt · otherReviews come with something most text doesn't: a star rating the writer chose themselves. That gave us an outside check. If the model's severity is meaningful, five-star reviews should come out "no problem" and one-star reviews should come out serious. And to make sure it wasn't just copying the number, we ran every review a second time with the stars removed from the state.
Results
Severity followed the stars: rank correlation −0.55 for Spotify and −0.74 for Instagram. And it wasn't copying them. With the stars hidden, severity scores stayed almost identical (correlation 0.97 and 0.99 between the two runs, average shift under 0.1) and still tracked the stars at −0.51 and −0.72. The type came out the same with and without stars on 94 of 100 and 48 of 50 reviews.
The mismatches were informative. Spotify's four-star reviews were mostly complaints at "minor annoyance" (1.08): people who like the app and grumble about ads. A star rating alone can't show that; the two numbers together can.
Reading the meaning, not the word
We asked whether each review blamed a recent update. Three Spotify reviews came back above 0.5, and none of them contained the word "update": "LEAVE IT ALONE, what is y'all's problem with changing everything that needs no change" (0.75), and "as it 'improves' it's just overcooking what previously worked" (0.81). A keyword search would have found none of them. Language detection did the same: every disagreement with the stop-word heuristic we used to grade it was the heuristic's mistake, on short Spanish texts like "Excelente, me encanta".
Stability and ambiguity
Thirty reviews, three runs each: severity moved by 0.01 on average and never more than 0.07; the type was identical all three times for all thirty. Where the model was unsure, the review was genuinely two things at once. "No ads — Cuz there ads" came out 53% complaint, 32% praise, which is about right for a title that contradicts its body.
| Spotify (100) | Instagram (50) | |
|---|---|---|
| Praise | 60 | 17 |
| Complaint | 29 | 21 |
| Bug | 3 | 8 |
| Request | 8 | 3 |
| One-star share | 14% | 54% |
| Top area | content 38, ads 22 | account 13, content 13 |
The two apps read like two different weeks. Instagram's latest fifty were 54% one-star with "account" as the top area (people losing access), eight bugs among them. Spotify's hundred were 60% praise, with complaints concentrated on ads and shuffle. Both pictures came out of raw text in about three seconds each.
One thing we'd fix
Nineteen percent of Spotify reviews landed in "other" for area, and a few praise reviews got stretched into "performance" ("Love it, works great!" → performance 53%). Our rubric had no "not about a specific area" option, so the model chose the nearest thing. It always picks one of the answers you give it; if none should apply, you have to say so. We'd add a `none` option and expect the picture to sharpen.
