ShiftwiseConv: Small Convolutional Kernel with Large Kernel Effect

$$Large kernels play a crucial role in enhancing the performance of standard convolutional neural networks (CNNs), enabling CNNs to outperform transformer architectures in computer vision. Scaling up kernel size has significantly contributed to the advancement of CNN models like RepLKNet, SLaK and UniRepLKNet. However, the relationship between kernel size and model performance varies across these work. It implies that large kernel convolution may involve hidden factors that affect model performance. Instead of merely increasing the kernel size, we reassess the role of large convolutions and decompose them into two separate components: extracting features at a certain granularity and fusing features by multiple pathways. In this paper, we contribute from two aspects. 1) We demonstrate that $3 \times 3$ convolutions can replace large convolutions in existing large kernel CNNs to achieve comparable effects. 2) We develop a multi-path long-distance sparse dependency relationship to enhance feature utilization. Specifically, we introduce the Shiftwise (SW) convolution operator, a pure CNN architecture. In a wide range of vision tasks such as classification, segmentation and detection, SW surpasses state-of-the-art transformers and CNN architectures, including SLaK and UniRepLKNet. Code and all the models at \url{https://anonymous.4open.science/r/shift-wiseConv-8978}.$$