Maybe because of Memories of Phantasm, I'd imagine the Rumia fight happening over a forest. But then considering the dialogue, I suppose it makes more sense that it's literally the backyard of the shrine?
Anyway, Reimu does say this just before fighting Rumia in EoSD.