Standard Text Embeddings Fail to Capture Human Preferences
Published on arXiv, the paper argues that off-the-shelf text embeddings are the wrong tool for collective decision-making systems. Standard embeddings measure semantic similarity, not preferential agreement. When style and wording correlate with stance, the geometry looks correct. When that correlation breaks, it fails.
The researchers formalize this as an invariance problem. Embedding models encode both preference-relevant signals (stance and values) and semantic nuisance (style and wording). A geometry that leans on nuisance can appear preference-correct even when it is not. That is a structural flaw, not a tuning problem.
The fix is targeted synthetic training data designed to break the semantic-preference correlation. The approach provably shifts the optimal scorer away from nuisance-dominated cosine similarity and improves preference prediction across 11 online deliberation datasets. Any operator building AI-mediated polling, civic deliberation, or recommendation systems that rely on standard embeddings should treat this as a calibration warning, not a theoretical footnote.