Darth Vader's betrayal was the only way Palpatine could possibly lose in Return of the Jedi. The Emperor was a master manipulator, a Machiavellian schemer whose plans brought about the downfall of the Old Republic and the rise of the Empire. He fulfilled the dream of countless generations of Sith, conceiving the Clone Wars as the ultimate Jedi Trap, ruling over the galaxy for decades.
The full scale of Palpatine's genius was only revealed in Star Wars: The Rise of Skywalker. The Emperor had learned the secret of conquering death, an ancient Sith power called "Essence Transfer," which would allow his spirit to inhabit another body at the moment of death. But there was a catch; it seems a Sith Lord's spirit can only possess someone if they are under the influence of the dark side at the moment the Sith Lord dies. This was why Palpatine wanted Luke Skywalker to strike him down in anger, and why he wanted Rey to do the same years later, in Star Wars: The Rise of Skywalker.
The Rebel Alliance was right to recognize Endor as a trap, but they weren't the Emperor's primary target. Luke Skywalker was his real goal; the son of the Chosen One, heir to all the power of Anakin Skywalker, a potential vessel for Palpatine's spirit. Treachery is the way of the Sith, and the Emperor knew Darth Vader's loyalty was questionable now he had learned his son was alive. He most likely guessed that, in The Empire Strikes Back, Darth Vader had attempted to seduce Luke to the dark side by offering him the chance to rule the galaxy at his side. And so he didn't expect Darth Vader to intervene and save his life.
But there was one thing the Emperor hadn't counted on; that there was still goodness in Darth Vader. Palpatine believed his apprentice to be thrall to the dark side of the Force, and thus the only betrayal he could conceive was one where Vader was party to killing his master in rage and fury. Such an action would free his spirit from his decaying body, allowing him to take his pick of the two powerful hosts around him. Instead, though, Luke reached something good at the heart of Darth Vader, and the Emperor's apprentice betrayed his master because of love rather than hate. At the moment of his death, when his spirit soared out of the Death Star Reactor to look for a host, Palpatine found only the light side of the Force awaiting him.
This was the one way he could lose. Had Luke struck him down, he would have triumphed and taken him as a host. Had Vader and his son worked together against the Emperor, Palpatine would have won. But the Emperor had overlooked the last flickering trace of goodness in Darth Vader's soul, a candle that Luke Skywalker fanned into flame. The Emperor's trap in Return Of The Jedi failed, and he only survived it thanks to the efforts of his disciples at Exegol, who conducted an ancient ritual to bring him back at a place where the veil between life and death was at its thinnest.