SinoGlyphBench Shows Language Models Remain Brittle to Chinese Glyph Obfuscation
Researchers introduced SinoGlyphBench, a diagnostic benchmark for Chinese glyph-level obfuscation in language-model moderation, on 5 September 2026. Across 176,916 paired evaluations of 12 LLMs and MLLMs, obfuscation increased harmful false-negative and false-positive rates by 6.1 and 4.7 percentage points and reduced four-way accuracy by 5.0 points. Models retained 75.7 percent of originally correct decisions but remained brittle to non-canonical glyphs, especially under full-scope and cross-script perturbations.
Researchers have introduced SinoGlyphBench, a diagnostic benchmark for Chinese glyph-level obfuscation in language-model moderation, in an arXiv paper titled SinoGlyphBench: A Diagnostic Benchmark for Chinese Glyph-Level Obfuscation in Language-Model Moderation. Version v1 was submitted on Saturday, 5 September 2026 at 03:15:49 UTC and is listed at 414 KB. The submission is from Yifan Wang, with co-authors Zimu Wang, Suliu Qin, Changyu Zeng, Tong Chen, Siqi Chen, Yijie Lin, Lingyu Jiang, Jionglong Su, Yushan Pan, Haiyang Zhang, Wei Wang, and Qiaoyu Tan. The authors write that glyph-level obfuscation can leave harmful Chinese content readable to humans while degrading automated moderation. The arXiv record provides a PDF together with an experimental HTML version. The diagnostic benchmark identifies label-critical semantic anchors and creates matched original and glyph-obfuscated inputs in text and image modalities. By perturbing anchors, background context, or both, this design distinguishes corruption of moderation-relevant evidence from general surface variation. The conditions include anchor-only perturbations, background-only perturbations, and full-scope perturbations of both. The work reports 176,916 paired evaluations of 12 LLMs and MLLMs on those matched original and glyph-obfuscated inputs. According to the authors, obfuscation increases harmful false-negative and false-positive rates by 6.1 and 4.7 percentage points, respectively, and reduces four-way accuracy by 5.0 points. The models retain 75.7 percent of the decisions that were correct on the matched original inputs. Full-scope perturbations cause the largest degradation, and anchor-only perturbations are more damaging than background-only perturbations. The authors further report that cross-script substitution is particularly difficult in the text modality. Analysis of structured outputs identifies observable mismatches in visible-form reading, intended-message recovery, and final safety judgment. The authors conclude that the evaluated models therefore remain brittle to Chinese content written with non-canonical glyphs. Resources are available at the URL given with the paper.