When Do Models Actually Decide? Mapping the Layer-Wise Decision Timeline in Pretrained Neural Networks

Neural networks are often treated as monolithic black boxes that process all inputs uniformly through all layers. However, researchers intuitively wonder: do simple images require all 50 layers of ResNet-50, or is the prediction effectively decided much earlier? We investigate when pretrained models make up their minds during a forward pass by training linear probes at each layer of ResNet variants on ImageNet, without modifying the base model. Our findings reveal substantial computational heterogeneity across architectures: ResNet-50 and ResNet-101 exhibit mean decision depths of 5.5--5.6 layers (k=2 stability), while ResNet-18 requires deeper relative processing at 7.4 layers. We discover pronounced bimodal patterns with distinct populations of early and late deciders, where 39--43\% of samples in deeper ResNets achieve stability within the first third of the network, while 39--54\% require processing beyond 70\% depth. The decision layer is highly sensitive to stability criteria, with mean depths increasing from 2.6--4.1 (k=1) to 9.0--10.0 (k=4). Linear probe accuracy exhibits sharp jumps in final residual stages, reaching 73--75\% for ResNet-50/101 and 65\% for ResNet-18, indicating that semantic consolidation occurs late. These findings expose computational heterogeneity in standard inference and provide actionable guidance for early exit strategies.