Actually... is it confirmed in canon that it really is just Yōmu who maintains the entire Hakugyokurō? To my understanding, the place is about just as expansive as Kōmakan. Maybe a significantly bigger garden, but still.

Like... what if Yōmu really isn't a Sakuya equivalent, but a Meiling equivalent? She's just the gardener and sword instructor. The chief servant of the estate is somebody else, her superior. Maybe a full phantom. And her fellow servants are likely also phantoms. Truly, a phantom hive...