The males are also walled in the monster compounds that keep them; for example, you could have goblins raiding the red oni for their men.
We need to consider if we want to make it game cannon that species which use human men for breeding stock produce only females of their own kind or human males as well; if there are multiple sources of human males (not just human women) the ecology changes dramatically.