Weakly-supervised Temporal Action Localization (WTAL) has achieved notable success but still suffers from a lack of snippet-level annotations, leading to a performance and framework gap with fully-supervised methods. While recent approaches employ pseudo labels for training, three key challenges: generating high-quality pseudo labels, making full use of different priors, and optimizing training methods with noisy labels remain unresolved. Due to these perspectives, we propose PseudoFormer, a novel two-branch framework that bridges the gap between weakly and Fully-supervised Temporal Action Localization (TAL). We first introduce RickerFusion, which maps all predicted action proposals to a global shared space to generate pseudo labels with better quality. Then, we leverage both snippet-level and proposal-level labels with different priors from the weak branch to train the regression-based model in the full branch. Finally, the uncertainty mask and iterative refinement mechanism are applied for training with noisy pseudo labels. PseudoFormer achieves state-of-the-art results on the two commonly used benchmarks, THUMOS14 and ActivityNet1.3, demonstrating its extensiveness in Point-supervised TAL (PTAL).