At the root of the idea of wilderness elitism is a fundamental problem of how many people can be supported by an area, or a world. "Supported" can mean many things in many contexts, though.
First, consider Aldo Leopold, growing up around the beginning of the twentieth century. He lived in a lot of places, from the northern midwest through the western frontiers of Arizona and New Mexico and down into Mexico proper; and in many of these places, he worked and lived in areas where very few people went. As a cowboy he lived on the range, watching herds of beef cattle being driven through lands that could easily support their numbers... until more and more cattle were driven, and sought out the easiest grazing, and started to strip areas and cause soil loss. As a hunter and wilderness denizen he watched first the solitude of the remote places start to fade as people got closer to them, then saw the elimination of habitats and the erosion of predation change the distribution and numbers of animals in ways that caused plagues and deforestation. As a forester, he watched places whose sublime beauty engendered a deep appreciation for the natural world get lost to clear-cutting, soil erosion, strip-farming, recreational wilderness adventures becoming more superficial and more impactful on the land, and everywhere the steady progress of humanity's impact on the land.
When Leopold looks at a small parcel of pristine land, he sees many things. A preservation of unique species and ecosystems, a part of the cycle of life, an object lesson in the interrelatedness of things. Perhaps most important, but least obvious, is that being in that place, living there, seeing it over a long enough period of time to become acclimated to it, and thus becoming able to (as he put it) "think like a mountain", changes a person. Makes him able to appreciate the world and his place in it, and make more appropriate, sustainable choices.
Today, we hear often about questions of man's impact on the world around him. Topics that used to be all the rage, like strip-mining, deforestation, recycling, and pollution, are now absorbed into more current and broader topics, like global warming, biodiversity, and extinction. And other questions that might not seem immediately related, like social justice and poverty, hang over us. Though on closer examination these things turn out to be related: in much of the world, one of the biggest environmental problems is how to keep people from taking the quick way to wealth when their alternative is grinding poverty, for instance.
All of these issues bring up a central common theme: there's just too many of us. To be sure, the world is capable of feeding and housing us all, and will be for at least a few more generations, though unchecked geometric expansion guarantees that not only will we exceed any theoretical maximum eventually, it'll come sooner and faster than people generally expect. But the world isn't feeding and housing all of us, even today. Why not?
Every day we're learning more and more how much we depend on things in the natural world we never even realized we needed before. Two hundred years ago we didn't know we needed bees for our food crops. Today, we're becoming aware that we need "harmful" bacteria for our bodies to work. We're seeing unexpected impacts in things like the ozone layer, the life cycles of frogs, and the health of sea creatures we barely even know about. Tomorrow will surely reveal more unexpected ways in which previously innocuous things turn out to impact us.
Right now, we could probably feed and house everyone in the world, and do it in a way that preserved the health of the world well enough to preserve our own health, if we did everything worldwide in the wisest way we know how. Fact is, though, we can't even get a single family in a well-to-do nation to do things in those wise ways. Let alone the developing nations where the alternative is starvation. Let alone everyone.
So what would it take to get to where everyone understood how important it is to live in harmony with the world? Leopold would probably say that they all need to spend a few years living in the unbroken wilderness. Okay, so Leopold is a bit touchy-feely and unrealistically demanding, but he makes a good point. Long before the world can't support our numbers in absolute, mathematical precision, it can't support us in a way that lets that connection to the natural world become as personal as it was for Leopold's generation. By the time we're able to see the problem, we're already moving past the point where we can make six billion people each, individually, see the importance of the solution.
There are surely other ways to make people realize these things. I do not share Leopold's ultimately bleak elitist attitude (though I can't help wondering if I had a way to share his experiences, if I might come to the same conclusion he seems to come to, that people like me just can't really appreciate the world in the necessary way, even if we think we do, without those experiences. How can you be sure?).
But time and again, any time you look at the issue of the environment, the problem always boils down to how many people there are today, and how many people there will be tomorrow. Our numbers at once make it harder for us to unite in deciding to behave in the wisest way, and make it more and more important that we do unite that way. It's a catch-22. And it also suggests that one of (not the only, but one of) the most important things we can be doing to help the human race stop digging itself into a hole is to contain population growth, worldwide. Not merely to avoid hitting the absolute carrying capacity, not merely to reverse the geometric expansion into something more linear. But also to reduce our impacts on all the parts of the world that are valuable in themselves; and to reduce our impacts on all the parts of the world that we depend on, maybe without knowing it; and to preserve our ability to realize we need to do all of that.