While it is tempting to imagine that Gensōkyō is stuck in the past where Japanese people were genuinely really short 🤔 the fact that outsiders do move in does make it unlikely that the gene pool of the Human Village is that isolated.
Like, shit, what if Kosuzu is actually one quarter Scottish?