From a narrative perspective, Luke Skywalker had to remain hidden in The Force Awakens. The Director of Star Wars VII, J J Abrams remarked upon Luke’s role himself. Mr Abrams noted that if Luke is there for more than just the closing scene, he overshadows everything else that’s going on. New characters like Finn, Rey and Poe Dameron need time to time live and breathe. The new characters needed the opportunity to form their relationships among themselves and with the audience. Luke has to remain hidden not just from the other characters, but from the audience as well. If Luke had been there, he would have stolen the audience’s attention away from new and old characters, alike. But Luke’s presence would have especially short-changed Han Solo. And Han, he needed his moment in the sun, even if that moment was under a sun whose power was being drained to fuel a massive weapon. Han needed a moment in the sun, even if it was his death scene.
Luke had to remain hidden in Episode VII because a mentor has to die in the opening film of a Star Wars trilogy. Given the array of characters, it had to be either Han or Luke. Because Harrison Ford had been asking George Lucas and Kathleen Kennedy to kill of Han Solo since the early 1980s, it was Han’s time. Near the end of Episode VI was Han’s time to go. But if a mentor is going to die, it should be after he builds a meaningful relationship with the characters and the audience, like Alec Guiness did in the first Star Wars. So, in order for Han to build that relationship with the audience, Luke has to remain hidden, lest he steal all the sunlight and rob Han’s death of it’s meaning. Or Luke has to be the mentor who dies. I’m not sure anyone wanted to see that story.
The relationships among the characters and the relationships they form with the audience that give a film resonance, across years and decades. Relationships allow a film to hold up despite ageing special effects. Ridley Scott’s Blade Runner holds up for the conflicted relationships and search for meaning among the characters. George Clooney’s Ocean’s 11 and Ocean’s 12 stands up after decades because of the camaraderie among the characters and of the characters with the audience. Even the original Star Wars and Empire Strikes Back hold up better than the special editions, because of the characters and the relationships they form with each other and with the audience. All these movies hold up after years and decades, because of the relationships among the characters and they way the characters form relationships with the audience. Having Luke hidden throughout the first episode of the new trilogy gives characters, new and old, time to live and breath and flourish. Even though for some characters, it’s only briefly.
So, I can accept from a narrative perspective, why Luke must remain hidden throughout all but the closing minutes of Episode VII. HOWEVER, J J Abrams didn’t have to start his story with the Resistance already on the run. He didn’t have to give the First Order the chance to destroy the homeworlds of the New Republic. He could have picked up somewhere before that. The new characters could still have formed relationships. Han could still have died at the hands of Kylo Ren/ his son Ben in a meaningful way. Luke would have seen the First Order placing its boots on the necks of the Galaxy. He would have seen the Resistance and returned to the Galaxy in its hour of need.
Why Luke stayed in exile even as the First Order gained strength and placed the Galaxy in jeopardy is the real question. The flashback sequences in The Force Awakens showed Ben Solo / Kylo Ren starting a revolt at Jedi training. I can see why Luke would go into exile after that. He would feel like he had failed Han and Leia by allowing the Dark Side to seduce their son. I’m sure it would take some time to get over that. But when the First Order arises, sends the New Republic running and his sister pulls the Resistance together, Luke must know this. Luke saw Han and Leia suffering on Cloud City in The Empire Strikes Back. So, he must have seen the rise of the Resistance. Why wouldn’t he return to help the galaxy in his generation’s greatest hour of need?
At D23 Expo 2017, a few weeks ago, Mark Hamill, who plays Luke Skywalker, said that Episode VIII Writer/Director Rian Johnson’s script had really taken Star Wars to a place it had never been before. Mr Hamill said that Rian Johnson had really cracked open the story. If part of that story explains why Luke Skywalker remained in exile for so long from a story and plot perspective, then it will be long remembered in the annals of science fiction storytelling.