The Ewok celebration music swells. Now a full-fledged Jedi Knight, Luke Skywalker winks at the happy Force Ghosts of Obi-Wan Kenobi, Yoda, and his father Anakin Skywalker, and then runs off to embrace his sister and their Rebel friends; Han Solo, Chewbacca, Lando Calrissian, and the Droids. And they lived happily ever after.
That's what audiences wanted to believe when the credits rolled on Return of the Jedi in 1983. Over 30 years later, when audiences finally caught up with Luke Skywalker in Star Wars: The Last Jedi, he wasn't quite as expected. The Luke Rey found on Ahch-To was not the legend she'd heard of, nor was he the Expanded Universe Jedi Master fans had long imagined.
This was now Luke Skywalker in the twilight years of his life, living in self-imposed exile, wallowing in his failures, and waiting to die. Decades after their initial ending, he, Han, and Leia were estranged and unhappy, having survived yet more trials and pain, and not the better for it. The galaxy had been embroiled in the resumption of the war between a great fascistic evil opposed by a good, rebellious few that Luke and his family had fought in their youth. Ben Solo, a.k.a. Kylo Ren, was the key to it all. Luke blamed himself for his nephew and apprentice's fall to the Dark Side - which he helped instigate during a tragic moment of fear and weakness. The optimistic adventurer fans had thrilled to back in the 70s and 80s was gone, replaced by an embittered and depressed old hermit. To make matters worse, the release was dogged by quotes from Mark Hamill saying he disagreed with Rian Johnson's vision for the character, as if this was an affront to the very idea of Star Wars.
Of course, it is understandable how, despite the three decades separating the younger Luke with his older self, how some fans had trouble accepting this as a progression of the same character they once knew. But it is nevertheless a logical progression. Let's look at how young Luke can be reconciled with "Old Luke" to understand the throughline from Return of the Jedi to The Last Jedi.
Luke Didn't Change From The Force Awakens To The Last Jedi (This Page)
Rian Johnson's Luke Skywalker Wasn't That Different From J.J. Abrams'
Now, it is true Luke in The Last Jedi didn't match expectations, which was the point. Luke was a shell of himself, dwelling on his own failures and what he believed to be the failures of the entire Jedi legacy. Fans who felt Luke would reignite the Jedi and the hope that came with it during the decades in between the trilogies were dismayed to find his attempt ended in failure. Luke Skywalker, a character who was literally "A New Hope", had given up. By The Last Jedi, he believed "it's time for the Jedi to end." This was understandable in the context - given his failure with Ben, Luke would naturally blame both himself and the Jedi-vs.-Sith cycle that had brought so much death and destruction to the galaxy - but was this in character for Luke in the first place? Actually, yes.
What's interesting is that this drastic change to Luke's perspective can really be chalked up more to J.J. Abrams than to Rian Johnson. So much of The Last Jedi's reaction comes from expectations laid down in The Force Awakens, yet Johnson merely built upon the foundation that Abrams laid: Ben Solo's betrayal; the destruction of the Jedi Academy; and Luke's conviction that he had failed. Why else would he seclude himself on Ahch-To if not because he felt it was best for the galaxy that the Jedi are taken off the playing field? Further, despite Hamill's comments about Johnson's changes, in terms of actually altering to the plan the only evidence we have is that exiled Luke was intended to still be in contact with the Force.
Some of that may be logical insinuation, but it was all established in The Force Awakens, not The Last Jedi. And what Episode VIII does is take that evolution and bring it right back to the original film.