This was my first thought, as well! Population size is the only vaguely linear factor that wouldn't be too obviously linked to specific technology. It also gels with witches in cottages, elves in the woods and wizards in towers, if you're planning to incorporate legends as blurred fact.