I'm going to inevitably be speaking from the viewpoint of my experiences in Lusternia here, and using it for examples, but the subject matter is more generally applicable than that, and informed by my experiences in a lot of other places too.
Most people are nowhere near as good as they think they are at separating in character (IC) from out of character (OOC). The division is also asymmetrical. People who are fairly good at avoiding having their characters act on information they only know from OOC sources are still often bad at having their character's IC experiences color their OOC feelings and decisions, even when those decisions in turn affect their IC decisions. And in virtually all cases, a person's natural proclivities -- including, nay especially, those they don't get to act on in real life -- come out in their character's actions, in their choices of a kind of character to play, and in their character's attitudes.
With this in mind, games in which the "bad guys" are a player organization (as opposed to NPCs) run a grave risk.
First, people are more likely to play a character in an evil organization if they have something inside themselves that draws them to it. It's hard to talk about this without sounding like you're making hugely unfair generalizations and accusations, but while none of this can be said with confidence about any specific player, it is nevertheless the case that for a larger number of people playing "evil" characters, at least one of these things will be true. They might have a frustration inside with their own powerlessness in life that expresses itself as a desire to make other people unhappy as a means of exercising control. They might feel anger which they're tired of having to suppress. They might dislike having to rein in their willingness to let the ends justify the means and be constrained by sometimes-abstract ethical principles. They might be drawn to a chance to act on darker impulses which their real life doesn't allow them to even admit having, maybe even to themselves.
Second, being involved in an organization which preaches cruelty, betrayal, hatred, backstabbing, and ruthlessness tends to make those concepts feel more comfortable. No matter how often the players protest that they're "not really evil" these ideas do tend to have an effect on making them more willing to engage in actions that hurt other people, even without much benefit to themselves, because the IC/OOC barrier is just not as good as people think. When one's character spends a lot of time rationalizing his actions and justifying them with arguments about how the other guy asked for it by becoming involved, the player starts to be persuaded too.
These tendencies can be kept at bay by a number of things, including the players reining themselves in and policing one another, the administration nudging them from time to time, and feedback in OOC areas like forum discussions. One important factor that has helped at times in Lusternia, though, is avoiding letting the evil organizations be entirely clichéd.
Don't get me wrong. Clichés are not all bad. The difference between a cliché and an archetype is that you like one and dislike the other. Embracing and working with clichés is a powerful way to make an experience compelling, and the most successful roleplaying games always do this. Lusternia is no exception: all its factions are built on a foundation of clichés.
But if you build an "evil" organization out of nothing but the usual clichés, you can easily run into a couple of problems. First, there's not really much survival value in being always eager to betray every trust you've built, nor in causing harm for its own sake without necessarily seeking benefit for yourself, nor to the pursuit of destruction of even that on which you depend. Second, it's not very compelling as a reason for doing things, as something you can defend in an argument and stand by, that you can build something up from, to just be evil. There needs to be something behind it, some method to the madness.
But more germane to the topic of this post is the fact that too clichéd a design for an evil organization tends to exacerbate the already-existing tendency for its players to spend too much time causing other players -- not just their characters but also the players -- to be miserable. That's a tendency that is always there, like a rock's drive to roll downhill, and the fact that one can oppose it doesn't change that it's always there. It's best to diminish the tendency as much as you can, too.
Lusternia's design of the evil organization lends it a very potent additional layer. They're not simply in support of evil for its own sake, in doing harm because they reject an ethos that says they shouldn't, in conquest because they want to have more stuff. They are also embracing a concept that a particular force -- the excreta of a Soulless God -- can be used to strengthen and transform.
It's tapping into an age-old dilemma: is a by-product of something bad always something bad? For instance, if someone does ethically wrong things that make decent folk recoil in horror, but as a result makes a discovery that could be put to beneficial use, should the discovery be thrown away? Will its use inevitably corrupt whatever good use it's put to? This is a theme that occurs a lot in fantasy, and Lusternia builds on it and provides a lot of fodder to allow the evil guys to be about something, something they can really defend and justify.
At some times in Lusternia's history, the people of most influence in Magnagora, the "evil" city, have taken this and run with it. But at other times, there's a backslide into a more cartoonish and simplistic evil, one which dwells on hatred and cruelty for its own sake. The problem with this isn't that it's clichéd; it's that it also tends towards making the players in Magnagora more geared towards causing other people to be miserable. And that in turn is self-perpetuating: those who feel that way rise in the ranks and become more active, while those who would prefer to have Magnagora be about something are discouraged and become less active. Unfortunately, this shift also affects everyone else, and tends to drag the game down.
This tendency isn't universal, nor is it unstoppable. During Lusternia's four years it's ebbed and flowed, because even though it's self-perpetuating, there are some things that'll change it, such as the sudden disappearance of an influential player (because he goes to another game, for instance, or stops playing MUDs). But when it happens, as it has been happening for the last few months (and perhaps as bad as, or worse than, it's ever been before), the game as a whole suffers.
Still, even if it got twice as bad as it's been lately, it wouldn't compare to what it was like in Aetolia, where cartoonishly shallow evil pretty much ran the game. (At least when I played. I don't know if it's still like that. It seems unlikely that poison rooted that deeply can be pulled out, but who knows? I'm certainly not interested in checking.) There's still enough room in Lusternia to work around the idiots even when they're mostly banded together, and even when their actions are starting to cause their opposites to adopt their inconsiderate disregard, there's room to avoid being colored by it.
Still, I hope the administration is planning some kind of event that will break the momentum of escalating retributive vindictiveness, or preparing to exert some influence away from the direction fo spite and malice. It's been a while and we're about due.