Batman has been through many trials and difficult times in his crime-fighting career, but Bane managed to do something that even the Joker couldn't - break the Bat. When Joker killed Jason Todd it was one of the most brutal things to ever happen to Batman, but his loss to Bane was still somehow more ruinous.
Bane grew up in the prison of Santa Prisca, where he was imprisoned from birth to finish serving the sentence of his absent father. Throughout his childhood, he had nightmares of a giant bat terrorizing him and eventually, after he escaped Santa Prisca, he set his sights on Gotham City, insistent that if he defeated the big bat that roamed the city, his nightmares would cease. Batman has faced many dangerous villains in his time and while Gotham's villains are some of the most terrifying in comics, very few are able to physically match him... that is, until Bane appeared.
After using Gotham's villains to run Batman through an impossible gauntlet, Bane then broke the Dark Knight's back over his knee. While the physical fight was devastating, Batman was also emotionally devastated by the encounter, distraught in a way he wasn't even when Joker murdered the second Robin, Jason Todd. So what made his defeat to Bane so specifically spirit-breaking for him? In Batman #498 by Doug Moench and Jim Aparo, Alfred explains that Batman couldn't have prevented Jason Todd's death, but Bane took him on one-to-one and Bruce lost fairly. While the Batman has suffered horrific losses before, Alfred argues this is the first time he has personally "faced another man squarely and lost."
Batman did eventually manage to recover from Bane's attack, both spiritually and physically, but it took a long time. Batman is someone who is always used to being in control and always having a plan for any eventuality, and for once in his career, he met someone who outplayed him tactically and physically. Looking over their history together, Bane has been one of Batman's greatest enemies. Bane managed to take over Gotham City, killed Alfred, and broke the Bat. While Joker killed Jason Todd and has preyed on Batman's allies for years, Bruce's greatest failures - the losses he could actually have prevented - have come at the hands of Bane.
Bane has been an incredibly powerful and consistently dangerous villain since he first appeared in comics - a threat not just to all of Gotham, but to everyone around the world who ever stood against him. It makes sense why Bane would be the first real failure that Batman ever encountered considering the unique tactical and physical supremacy he brought to the table. While Joker hurt Batman forever by killing Jason Todd, Bane actually showed himself to be a true match for the Dark Knight, forcing Batman to confront true failure and, thankfully, emerge from the other side.