Skip to yearly menu bar Skip to main content


Oral Session

Oral Session 6D: Large-Scale Neural Modeling

Sun 7 Jun 1 p.m. PDT — 2:15 p.m. PDT
Abstract:
Chat is not available.

Sun 7 June 13:00 - 13:15 PDT

Efficient Unrolled Networks for Large-Scale 3D Inverse Problems

Romain Vo ⋅ Julián Tachella

Deep learning-based methods have revolutionized the field of imaging inverse problems, yielding state-of-the-art performance across various imaging domains. The best performing networks incorporate the imaging operator within the network architecture, typically in the form of deep unrolling. However, in large-scale problems, such as 3D imaging, most existing methods fail to incorporate the operator in the architecture due to the prohibitive amount of memory required by global forward operators, which hinder typical patching strategies. In this work, we present a domain partitioning strategy and normal operator approximations that enable the training of end-to-end reconstruction models incorporating forward operators of arbitrarily large problems into their architecture. The proposed method achieves state-of-the-art performance on 3D X-ray cone-beam tomography and 3D multi-coil accelerated MRI, while requiring only a single GPU for both training and inference.

Sun 7 June 13:15 - 13:30 PDT

FedAdamom: Adaptive Momentum for Improved Generalization in Federated Optimization

Wenjie Hou ⋅ Tianxiang Chen ⋅ Feng Wang ⋅ Tiantong Wu ⋅ Zhiming Zheng ⋅ Shaoting Tang ⋅ Wei Yang Bryan Lim

Federated learning (FL) has emerged as a widely adopted training paradigm for privacy-preserving machine learning. Despite the past success of SGD-based methods, they still suffer from severe data heterogeneity and the lack of adaptivity in practical applications. While several adaptive federated optimization methods (such as FedAdam) have been proposed and demonstrated to achieve faster convergence, they fail to show significant improvements in generalization performance under highly heterogeneous data distributions, and their optimization and generalization mechanisms remain insufficiently understood. To fill this gap, we introduce diffusion theory into the adaptive federated optimization framework and analyze the distinct effects of adaptive learning rate and global momentum from the perspectives of saddle-point escaping and flat-minima selection. Theoretical results show that although FedAdam outperforms FedAvg/FedAvgM in escaping saddle points, the latter escapes sharp minima more efficiently. The root cause lies in that adaptive learning rates, while enhancing saddle-point escape, weaken the preference for flat minima. Motivated by these insights, we propose FedAdamom, a new adaptive federated optimization algorithm that adapts the momentum hyperparameter rather than the learning rate. FedAdamom maintains strong saddle-point escaping capability while enhancing flat-minima selection. We further establish its convergence guarantees under non-convex objectives. Extensive experiments demonstrate that FedAdamom significantly outperforms existing adaptive federated optimization methods in terms of convergence speed, generalization performance, and preference for flat minima.

Sun 7 June 13:30 - 13:45 PDT

SimScale: Learning to Drive via Real-World Simulation at Scale

Haochen Tian ⋅ Tianyu Li ⋅ Haochen Liu ⋅ Jiazhi Yang ⋅ Yihang Qiu ⋅ Guang Li ⋅ junli wang ⋅ Yinfeng Gao ⋅ Zhang Zhang ⋅ Liang Wang ⋅ Hangjun Ye ⋅ Long Chen ⋅ Hongyang Li

Achieving fully autonomous driving systems requires learning rational decisions in a wide span of scenarios, including safety-critical and out-of-distribution ones. However, such cases are underrepresented in real-world corpus collected by human experts. To complement for the lack of data diversity, we introduce a novel and scalable simulation framework capable of synthesizing these crucial massive unseen states upon existing driving logs. Our pipeline utilizes advanced neural rendering with a reactive environment to generate high-fidelity multi-view observations controlled by ego trajectory perturbations. Furthermore, we develop a pseudo-expert trajectory generation mechanism to provide feasible action supervision for these newly simulated states to provide action supervision.Upon the synthesized data, we find that a simple co-training strategy on both real-world and simulated samples can lead to significant improvements in both robustness and generalization for various planning methods on challenging real-world benchmarks, up to +6.8 EPDMS on navhard and +2.9 on navtest. More importantly, such policy improvement scales smoothly by increasing simulation data only, even without extra real-world data streaming in. We further reveal crucial findings of such a sim-real paradigm, includingthe design of pseudo-experts and the scaling properties for different policy architectures. Simulation data and code would be released.

Sun 7 June 13:45 - 14:00 PDT

Texvent: Asynchronous Event Data Simulation via Text Prompt

Ruofei Wang ⋅ Peiqi Duan ⋅ Ka Chun Cheung ⋅ Simon See ⋅ Boxin Shi ⋅ Renjie Wan

Current event simulation methods focus on employing videos to synthesize new event data, suffering from costly video capture and limited scalability across viewpoints, motions, and lighting. To this end, we propose a Text-to-event simulation framework (Texvent) that can directly generate asynchronous event data from simple text prompts. Texvent first renders prompt-driven videos via multimodal large language models and subsequently applies a new physical simulator to generate event streams. Specifically, an adaptive brightness-aware frame interpolation approach is proposed to enhance the temporal resolution of the rendered videos. A balanced logarithmic intensity comparison strategy and a cache–based voltage refreshment mechanism are introduced into the simulator to generate event data.To narrow the sim-to-real gap, we also introduce background activity noise injection and dense time stamp reconstruction operations. Extensive experiments demonstrate Texvent’s superior computational efficiency and its ability to generate more realistic event data than existing simulators.

Sun 7 June 14:00 - 14:15 PDT

WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World

Ao Liang ⋅ Lingdong Kong ⋅ Tianyi Yan ⋅ Hongsi Liu ⋅ Yu Yang ⋅ Ziqi Huang ⋅ Wei Yin ⋅ Jialong Zuo ⋅ Yixuan Hu ⋅ Dekai Zhu ⋅ Dongyue Lu ⋅ Youquan Liu ⋅ Guangfeng Jiang ⋅ Linfeng Li ⋅ Xiangtai Li ⋅ Long Zhuo ⋅ Lai Xing Ng ⋅ Benoit R. Cottereau ⋅ Changxin Gao ⋅ Liang Pan ⋅ Wei Tsang Ooi ⋅ Ziwei Liu

Generative world models are reshaping embodied AI, enabling agents to synthesize realistic 4D driving environments that look convincing but often fail physically or behaviorally. Despite rapid progress, the field still lacks a unified way to assess whether generated worlds preserve geometry, obey physics, or support reliable control. We introduce WorldLens, a full-spectrum benchmark evaluating how well a model builds, understands, and behaves within its generated world. It spans five aspects - Generation, Reconstruction, Action-Following, Downstream Task, and Human Preference - jointly covering visual realism, geometric consistency, physical plausibility, and functional reliability. Across these dimensions, no existing world model excels universally: those with strong textures often violate physics, while geometry-stable ones lack behavioral fidelity. To align objective metrics with human judgment, we further construct WorldLens-26K, a large-scale dataset of human-annotated videos with numerical scores and textual rationales, and develop WorldLens-Agent, an evaluation model distilled from these annotations to enable scalable, explainable scoring. Together, the benchmark, dataset, and agent form a unified ecosystem for measuring world fidelity - standardizing how future models are judged not only by how real they look, but by how real they behave.