The "somewhat strangely offers to refer you to counseling" bit is the real tell. That's a human-review fallback UX. The classifier fired, they don't have enough signal to action it, so they route you to a soft warning that also doubles as a dark pattern if you're actually guilty. For normal users it reads as Kafka.
Building detection systems I've seen this pattern a lot. False-positive rates on any NSFW or abuse classifier at Meta's scale are measured in millions of daily hits. "megastorage" tokenizing into something adjacent to a flagged term is the kind of thing that slips past ablation testing.
The part worth worrying about isn't the accusation. It's that the appeal loop probably doesn't exist. If a classifier flags you, that signal persists in your profile's feature vector indefinitely. No one tells you, and nothing you do in the product removes it.
The "somewhat strangely offers to refer you to counseling" bit is the real tell. That's a human-review fallback UX. The classifier fired, they don't have enough signal to action it, so they route you to a soft warning that also doubles as a dark pattern if you're actually guilty. For normal users it reads as Kafka.
Building detection systems I've seen this pattern a lot. False-positive rates on any NSFW or abuse classifier at Meta's scale are measured in millions of daily hits. "megastorage" tokenizing into something adjacent to a flagged term is the kind of thing that slips past ablation testing.
The part worth worrying about isn't the accusation. It's that the appeal loop probably doesn't exist. If a classifier flags you, that signal persists in your profile's feature vector indefinitely. No one tells you, and nothing you do in the product removes it.
Yeah somewhere deep in Facebook they’ve put a black mark against my profile “filthy 300 CD player buyer, keep an eye on him”.