If he failed to kill a baby and an adolescent boy, then Voldemort must be a pushover, right? In hindsight, both of those things definitely aren't easy to do, unless you're a serial killer. That's why stopping and defeating Lord Voldemort is a monumental task that requires all the efforts in the Wizarding World... or at least a whole school's effort at least.
Even so, there are actually some individuals in the Wizarding World who could have forced Voldemort to use a Horcrux or two. After all, he's not the only talented wizard around. Certain characters in the Harry Potter lore can and have proven that they can defeat Voldemort in a fair fight. Let's take a look at those individuals, shall we?
10 ALBUS DUMBLEDORE
This one's already a given. Albus Dumbledore's the most obvious person who can defeat Voldemort without even sweating (not that he has much sweat left, anyway). Rowling herself has explicitly stated in the books that Dumbledore is the only person Voldemort fears since he's actually no match against him squarely.
A quick look at their fight in Order of the Phoenix should be enough to see how Dumbledore fares against Voldemort. In fact, Voldemort would have been defeated then and there if not for Harry's presence. That means if Dumbledore ever had another fair chance to fight Voldemort, the Dark Lord's defeat would be inevitable.
9 GELLERT GRINDELWALD
Objectively, Gellert Grindelwald actually had better "Dark Lord" achievements than Voldemort. Voldemort just happened to be more evil and more driven, hence more dangerous, but he is by no means more powerful than Grindelwald. It's worth noting that Grindelwald pretty much conquered all of Europe in just a few years.
Voldemort, by comparison, struggled to take over a small region of the Wizarding World (including a school) in more than a decade. Dumbledore himself admitted that Grindelwald was a "touch more skilled" when both were at the peak of their powers and skill.
8 ANTONIN DOLOHOV
Death Eaters don't exactly have a long life expectancy due to the occupational hazards but the ones that do survive their own temperamental boss and the dangers of the job are among the most powerful and cunning. Antonin Dolohov, for that matter, is the most decorated Death Eater in Voldemort's ranks.
He has the highest win/loss ratio among all Death Eaters and his peers are no match for his dueling skill. He was able to defeat Remus Lupin and Mad-Eye Moody while also having the rare ability to cast strong spells without words. If he plays it smart and careful like Harry, he just might get a chance against his own boss
7 NEVILLE LONGBOTTOM
It might sound like a stretch, but Neville Longbottom could have been as good as Harry. Because in an alternate universe, he was the official Chosen One and just as Harry stood against Voldemort and defeated him in a straight duel, Neville most likely could have done the same.
It just so happens that Harry was the luckier of the two, somehow. Meanwhile, Neville, being the "other" Chosen One, could have easily fulfilled the prophecy just as Harry did. After all, Harry mostly just won because the Elder Wand refused to kill him.
6 MINERVA MCGONAGALL
McGonagall was among the oldest and most experienced people in Hogwarts and she was a worthy successor to Dumbledore as the headmistress. In fact, she has proven her mettle against Voldemort and survived a duel against him. To be fair, it was McGonagall, Horace Slughorn, and Kingsley Shacklebolt who dueled Voldemort at the same time.
Sadly, he incapacitated all three of them after being enraged at the news of Bellatrix Lestrange's death. However, the fact that McGonagall was able to withstand Voldemort is proof enough that she can take on him under the right circumstances and with luck. She was qualified enough to be a headmaster, after all.
5 FILIUS FLITWICK
Big things come in small packages and that's what Professor Filius Flitwick is all about. He might only be half as tall or big as Voldemort but he surely has the dueling capability to withstand the Dark Lord. As proof of Flitwick's mastery of dueling, he's the one who defeated Antonin Dolohov, Voldemort's most decorated Death Eater.
It's safe to assume that Flitwick has a fighting chance against Voldemort himself in the dueling department at the very least. Voldemort would also most likely underestimate Flitwick due to his size, which can prove to be his undoing. Flitwick did defeat his best Death Eater.
4 KINGSLEY SHACKLEBOLT
Kingsley Shacklebolt is one of the three Hogwarts professors who held Voldemort commendably and he's also one of the veteran members of the Order of the Phoenix. Shacklebolt's skill and talents did net him the top position in the Ministry of Magic and that says a lot about his capabilities.
When it comes to duels and wizard battles, Kingsley has shown that he can easily fight and survive multiple Death Eaters all at once. He's one of the few who came to Harry's rescue in Order of the Phoenix. In any case, anyone worthy enough to become a Minister of Magic is competent enough for a duel with a Dark Lord.
3 CREDENCE BAREBONE
Gellert Grindelwald was smart and thought ahead. The biggest threat to his radical plans was Albus Dumbledore. Needless to say, he wasn't confident that his talent would be enough to kill Dumbledore. Hence, Grindelwald sought someone (or something) that could have easily dispatched Dumbledore: Credence Barebone.
Credence possesses powers which Grindelwald was sure would make quick work of Dumbledore and sought it for himself. Heck, even Voldemort didn't have direct means to deal with Dumbledore on his own. So it stands within reason that a creature as powerful as Credence would easily win against Voldemort.
2 SEVERUS SNAPE
Severus Snape was similar to Voldemort in that both of them were astonishingly brilliant back in their youth. Whereas Voldemort's achievement back in his school days were his Horcruxes, Snape's was his own potions knowledge and his very own curse-- someone no else but Harry knew about.
Moreover, Snape can easily resist Voldemort's Legilimens attempts like he has done so many times to hide his allegiance with Dumbledore. Voldemort and Nagini did kill him, but it wasn't really much of a fight. Snape's no slouch and could have easily use his hidden knowledge and spells against Voldemort in order to win especially if he has a cause to fight for. He might even win, who knows.
Dumbledore might have been the best student ever in Hogwarts and the most powerful wizard of his time but the most powerful wizard of all time in the Wizarding World remains Merlin. His ideals and goals were the exact opposite of Voldemort's even though both of them got sorted into Slytherin.
To that end, Merlin was even more powerful than Dumbledore at their respective peaks. Voldemort would have easily backed off immediately if a person like Merlin was present during his reign of terror. In fact, Merlin does not even need a wand to cast any form of magic at full efficiency, even Dumbledore wasn't that powerful.