Texture recognition has more recently relied on Neural Networks that are Convolution, Transformer and Graph based. However, many of these methods fail to effectively incorporate frequency characteristics exhibited by visual and latent texture attributes. In addition, effective orderless representation of textures before mapping from latent to visual texture attributes has not been fully explored. Finally, there is no loss function that has been designed specifically for texture and material recognition tasks. In this study, we introduce the Chebyshev Attention Depth Permutation Texture Network (CAPTN), which by using texture frequency attention mechanisms and convolution operations to generate latent texture attributes. These attributes are then enhanced by permuting the feature space. CAPTN then incorporates a non-linear learnable Chebyshev function to improve mapping of orderless enhanced latent texture attributes to visual texture attributes. Finally, we propose Latent Texture Attribute Loss to understanding spatial texture characteristics and enforce distributional consistency of orderless latent texture attribute representations. CAPTN allows end-to-end training without the need to fine-tune pre-trained CNN backbones. Experiments show that CAPTN achieves state-of-the-art results on multiple benchmark texture and material datasets.