Why did Peter Capaldi's Twelfth Doctor regenerate in Doctor Who? There's a sense in which regeneration is the key to Doctor Who's success. It means the series can continually reinvent itself; if one Doctor's viewing figures are beginning to slide, or if the actor wants to move on, the BBC can just cast a replacement. Even better, while there are common traits between the different Doctors - the same intelligence, the same whimsical nature - each actor is allowed to portray the Doctor in a way that they feel works. David Tennant is a demonstrably different version of the Doctor to Matt Smith.
Peter Capaldi's Doctor had an interesting ride. He came out of a difficult regeneration, one that was only possible because of a boost of regeneration energy from the Time Lords, and as a result the Doctor initially struggled to work out just what kind of man he really was. In the end, though, he bowed out with some choice words of advice for his successor. "Never be cruel or cowardly," he advised. "Remember, hate is always foolish - and love is always wise... Laugh hard, run fast, be kind." It was a perfect finale, reflecting his journey to define himself.
In-universe, Peter Capaldi's Twelfth Doctor regenerated because of the accumulated damage he had sustained in battle with the Cybermen. Unusually, though, the Twelfth Doctor resisted the regeneration for several weeks. He was scared, afraid of the change, which he likened to a death in itself. But resisting the transformation carried its own risks. “We have a choice, either we change and go on, or we die as we are,” he realized. It took an unexpected encounter with his first incarnation, who likewise was resisting a regeneration, to convince the Doctor he had to let it happen. "I suppose one more lifetime won’t kill anyone," he reflected, in a bittersweet line of dialogue. "Well, except me."
Out-of-universe, Capaldi feared that his performance would suffer if he didn't quit at this point. He found Doctor Who to be incredibly demanding, with precious little preparation time, which meant Capaldi was really beginning to feel the pressure. "Doctor Who is a great show and experience but to be at the centre of that brand is a lot of work," he explained. "There’s a lot more than just acting. It was hard to maintain that level of commitment with that schedule any longer." After four years, he took the opportunity to bow out of the public arena for a while, enjoying taking the time to relax and play his guitar.
It's possible that Peter Capaldi's exhaustion is one of the reasons new showrunner Chris Chibnall has changed Doctor Who's format a little, with shorter and less frequent seasons. Certainly that will mean Jodie Whittaker won't face quite the same pressure, and hopefully she'll be able to stick around for a longer time as a result.