Despite being trained on massive data, today's vision foundation models still fall short in detecting open world objects. Apart from recognizing known objects appeared in training, a successful Open World Object Detection (OWOD) system must also be able to detect unknown objects that were never seen before, without confusing them with the backgrounds. Unlike the prevailing former works that learn "objectness" using probability models, we focus on learning fine-grained class-agnostic attributes that can be used to detect both known and unknown object classes in an explainable manner. In this paper, we propose Partial Attribute Assignment (PASS), aiming to automatically select and optimize a small, relevant subset of attributes from a larger attribute pool. Specifically, we model attribute selection as a Partial Optimal Transport (POT) problem between known visual objects and the attribute pool, in which more relevant attributes signify more transported mass. PASS follows a curriculum schedule that progressively selects and optimizes a targeted subset of attributes during training, promoting stability and accuracy. Our method enjoys end-to-end optimization by minimizing the POT distance and the classification loss on known visual objects, demonstrating high training efficiency and superior OWOD performance among extensive experimental evaluations. Our code will be made public.