Beyond Euclidean Gossip: KL-Barycentric Consensus on Heterogeneous and Imbalanced Images

Fully decentralized deep learning removes global servers and ensures local data privacy. However, Euclidean consensus, averaging weights, gradients or momentum, may degrade under non-i.i.d. data and client size imbalance. We propose a geometry-aware approach based on natural gradient variational inference. Clients communicate in the expectation parameter space of an exponential family, where simple linear mixing yields a forward KL barycenter consensus. The aggregate is the model closest to all client distributions, aligning updates across heterogeneous sites and mitigating distribution shift. We further provide a lightweight decentralized Adam implementation, in which each client maintains a diagonal-Gaussian posterior and both updates and gossips in the expectation space. We prove convergence for convex losses on connected graphs. On CIFAR-100 and a medical image segmentation benchmark, our method\footnote{All code is included in the supplementary materials and will be publicly released.} substantially outperforms Euclidean-space consensus baselines under severe non-i.i.d. and client-imbalance cases, achieving around $20\%$ accuracy gain on CIFAR-100, while matching the communication budget and improving training stability.