Don’t get too confused or twist yourself into a knot over this one, as Voldemort WAS a villain. But the fact is that he wasn’t the worst villain since he was a charismatic and powerful individual that gathered followers through force of will and threats of violence. But he was at least direct in what he wanted to do, while the real villains of the story belong to the Ministry of Magic, which would have much more to answer for if anyone could compel them to do so. Much like many governing bodies, the Ministry was the final word, more or less, in what went on in the wizarding world and what was allowed and what was punished with severe consequence. One might think that only a few of those within the Ministry was the real problem, but upon taking a closer look at the movie it’s pretty evident that from the second movie on the Ministry is a bit of a problem. Not only did they make Harry’s life miserable in every possible way, but they were easily swayed by Voldemort and his followers as well. One might think that the Ministry of Magic would be beyond reproach and composed of wizards and witches that were either neutral or could at least see the value in both sides of an argument or policy and determine a fitting compromise that would work for everyone. That appeared to be the case for a while, but after the Goblet of Fire movie, it was evident that the Ministry was so afraid of Voldemort’s return that they staunchly denied it and sought to discredit anyone that said anything of the sort.
This would obviously wreak havoc with Harry’s life and would even label Dumbledore as a bit of a kook as he was run out of his own school at one point. Not only that, but once Dumbledore was gone they brought in Dolores Umbridge, a woman who actually tortured the students if they stepped out of line and employed draconian measures within the school while denying the students any chance to learn spells that might help them. Once she was gone one might have thought that things would ease up and the Ministry would finally wise up and take precautions to defend against Voldemort and his Death Eaters, but obviously this wasn’t the case. When the Minister stepped down in shame after denying Voldemort’s return for a solid year, the Ministry remained a divided place as Umbridge was actually brought on as a judge at one point and placed in charge of a Horcrux, making it kind of obvious that she was a part of Voldemort’s plan in one way or another. The Ministry, supposedly some of the most powerful wizards and witches around with access to untold magicks that likely defy easy explanation, were never employed, and their ranks were infiltrated by Death Eaters as Voldemort continued his campaign against his enemies.
In effect, the Ministry was useless unless it had to do with keeping people in the dark and hampering them as a way to allow Voldemort to make his plans and carry them out. They did next to nothing other than to stand in the way of anyone that might have warned the wizarding world that was coming, and those that could have done something about it. To be realistic, Voldemort and his Death Eaters wouldn’t have been able to stroll into the Ministry as if they owned it without the actual Ministry simply standing aside instead of standing up for the citizens of the wizarding world. So in a big sense, the Ministry of Magic turned out to be the biggest bunch of villains in the story since not only did they choose not to believe that the greatest threat in their world had returned, but they went ahead and punished the individual that had made the announcement, and then didn’t bother to apologize for all that had happened once the war was over. At that point, it feels as though the survivors were just happy to be alive, but the Ministry likely needed a serious overhaul, after all, that and was no doubt in need of someone that could weed out the bad seeds and bring the Ministry back to prominence as the ruling body within the wizarding world that went about their business with hopefully little to no bias in any given matter.
But keep in mind, the Ministry was divided and thereby a villain by dint of not being able to do anything, or choosing not to do anything about Voldemort. The dark lord on the other hand had a decided purpose and meant to follow it through to the end, which makes him worse in the eyes of many. But the Ministry’s inability to come together as a ruling body was what made them the villains of the story.