Selection-as-Nonlinearity: Bridging Attention and Activation via a Joint Game–Decision Lens for Interpretable, Discriminative Visual Representations

Self-attention with separate pre- and post-projections can be a universal approximator (on compact domains) under mild conditions.Yet we observe a striking gap: an attention-only Transformer (w/o FFN layers) exhibits a marked accuracy drop relative to its standard interleaved attention--FFN baseline.We term this the **weak-independence** challenge of attention.We study this through a new conceptual lens, **Selection-as-Nonlinearity (SaN)**, which interprets effective nonlinearity as directed, cost-constrained selection, offering a coherent account of attention as context-gated activation.In this joint game–decision view, attention performs a resource-constrained cooperative allocation over values: each query distributes a unit-mass weight budget over shared values to optimize representational utility, under a normalizer (e.g., $\mathrm{softmax}$), and guided by context-derived scores (e.g., q-k similarities).SaN interprets *weak-independence* as a structural tension: the value weights almost cannot simultaneously attain the decoupled per-query (row-wise) and the per-value (column-wise) optimums under shared budgets, thereby limiting attention's stand-alone capacity.Guided by SaN, we introduce **CSaN**, an interpretable, efficient attention compensation paradigm with two key insights: **1) hierarchical budget calibration,** *re-allocate* row budgets via inter-query correction signals; and **2) public-private cooperation,** enhancing the *public* attention pathway with a per-token *private* value pathway to decouple conflicting demands.CSaN is evaluated on various vision benchmarks and demonstrates *level-jump gains* across popular Transformer families (Swin, ViT, Hiera), enabling models to rival much heavier same-family counterparts $\sim2\times$ as large.