Most stories have a villain of some sort to generate conflict (also known as an antagonist). Beginning writers are advised to give their hero or heroine (protagonist) a goal, and to have an antagonist who works against the protagonist, preventing him or her from reaching their goal. The tension rises as the protagonist struggles to achieve the goal and is knocked back more and more decisively; eventually a point of despair is reached, then a solution is envisaged and there is a final confrontation, during which the antagonist is defeated.
This is a classic structure for many genres, even though the nature of the conflict may vary. In a recent epic fantasy novel I read, the antagonist is a sorcerer of immense power bent on global domination, immortal, and the ruler of hordes of enslaved minions. In ‘Pride and Prejudice’ there are several antagonists: Mr Wickham, Caroline Bingley and Lady Catherine de Bourgh all act against the interests of the Bennet family, although only Mr Wickham is truly wicked; the others are merely selfish and arrogant.
But what happens if there is no obvious antagonist? In my own novels, only one of the three has a clear antagonist. In ‘The Incursors’, there are (in a sense) two ‘sides’ to the conflict: the people of Callina, bound to strict adherence to their rules, handed down five thousand years earlier by the three founding fathers; and the incursors, explorers who happen to wander across the border and are captured by the warriors who defend their country. There is a ton of conflict in this culture clash, but there is no good or evil, no obvious bad guys, just two sides with different philosophies on life bumping into each other.
In ‘The Plains of Kallanash’, there is an antagonist of sorts, but it isn’t an individual, and again it isn’t anyone pursuing an evil agenda. Rather, evil is being done because one group is making decisions which are perceived to be for the greater good, but in fact have many unintended consequences. It’s a classic example of good intentions gone awry, where innumerable small decisions, each one seemingly for the best, end up perpetuating a great evil.
In ‘The Fire Mages’, finally there is an antagonist of the classic type, a single individual bent on thwarting the aspirations of the protagonist and pursuing his own ambitions regardless of the consequences. Even there, it isn’t obvious that he is genuinely evil, at least until the final confrontation, where (to be honest) he goes off the rails a little.
So is it necessary to have an antagonist? No, clearly not, but it is necessary for a story to have conflict. The protagonist has to have something (or someone) to fight against in order to have a story worth telling. It doesn’t have to be an evil dark lord wanting to cover the land in eternal night; sometimes it’s much more interesting to place a protagonist in an interesting situation, drop a pebble into the pool and watch the ripples spread outwards.