Monocular Depth Estimation (MDE) has emerged as a pivotal task in computer vision, supporting numerous real-world applications.However, deploying high-performing depth estimation models on resource-constrained edge devices, especially Application-Specific Integrated Circuits (ASICs), remains a formidable challenge due to the substantial computational and memory demands of state-of-the-art models. Recent advancements in foundational depth estimation deliver impressive results but further amplify the difficulty of deployment on ASICs. To address this, we propose QuartDepth which adopts post-training quantization to optimize and accelerate MDE models specifically for ASICs. Our approach involves quantizing both weights and activations to 4-bit precision, significantly reducing the model size and computation cost. To mitigate the performance degradation typically associated with aggressive quantization, we introduce an activation polishing and compensation algorithm applied before and after activation quantization, as well as a weight reconstruction method for minimizing errors in weight quantization.Furthermore, we design a novel flexible and programmable hardware accelerator by supporting kernel fusion and customized instruction programmability, enhancing throughput and efficiency.Experimental results demonstrate that our proposed framework achieves competitive accuracy while enabling fast inference and higher energy efficiency on ASICs, bridging the gap between high-performance depth estimation and practical edge-device applicability.