There's a difference between sci-fi as a setting, and science fiction as a genre.
Science fiction, sometimes differentiated as speculative fiction, is all about exploring the implications of fictional science and technology. Frankenstein is often considered the first work of science fiction, because it truly is a "what if" story centered on a piece of fictional technology. Hard science fiction, such as the Martian, takes this principle to its limit by trying to rely only on known or highly plausible science, with the technology as close to currently feasible as possible. These kinds of stories are deeply grounded, and compel because they consider the consequences of everything they add. The story follows from the science. Black Mirror is a perfect example of this writing philosophy, looking at how people and technology change one another.
The "sci-fi" setting isn't really science fiction at all. Star Wars is set in space with lasers and robots, but nothing about it is driven by science. Star Wars is more interested in special effects set pieces, the politics of fascism, and pseudo-buddhist philosophy. Star Wars is space fantasy, and its philosophical themes almost make it good, but not quite. Star Wars would be amazing if it was good, but alas.
I think that all great fantasy is also about exploring the implication of fictional science. It's just, this science is less sciency-feeling to laypeople. Lord of the Rings is linguistic fiction. The entire world exists to serve a story about the evolution of language. It's awesome! Mistborn is also a really great "fantasy" series where the rules of magic are scientific, and clever readers can tell when unreliable narration is lying to them using the scientific method. These are the stories I love.
Harry Potter sucks because it isn't science fiction. After the second one came out, lots of fans were eagerly expecting the series to delve into the ethics of slavery using House Elves as a narrative device. "What if slavery is enforced by magic rules?" is an interesting sci-fi question. Instead of writing good political fiction, Robert Galbraith chose to say "Dobby's just weird for wanting to be free." That's boring. There's nothing for us to sink our curiosity into in that plot point. It's intellectually lazy. It's not science fiction.
I posit that all good works of fantasy fiction treat their magic the way good speculative fiction treats its science, at least a little bit. And if you're not going to include any of that, then what's even the point? Your universe is wasted on your story, and I'd rather read what AO3 has to say about your setting. I've read lots of HP fanfiction that was better than HP because it cared about science. Even Harry Potter and the Natural 20 is better than Galbraith's garbage.
And I really need to get around to reading Trans Wizard Harriet Porber and the Bad Boy Parasaurolophus...
I like the Hobbit more than I like Lord of the Rings, but I don't think it's good fantasy. Fun is different than good. The Hobbit manages to be engaging the whole way through, and it's great for kids, but it doesn't have anything interesting to say. It has even fewer narrative themes than Harry Potter, which at least had "love is the most powerful magic". Which sucks ass and has negative social implications for the asexuality spectrum. But the Hobbit has even less. For adults, the Hobbit is junk food literature. It will not nourish your mind. It will nourish the minds of children, who are natural little scientists always learning about the world, and who need lots of food for their growing imaginations. So, the Hobbit is closer to being science fiction for kids because kids don't know much, and thus everything is science to them. Harry Potter is worse for kids because it actively teaches bad life lessons.
Also, holy shit and thank you so much. Ideas from the Black Magician have been floating around in My head for over a decade, but I forgot the name of the book! Now I can finally give that series a reread! And though due to My long-rusted memory I can state naught with confidence, I believe Black Magician is science fiction in the way I have described above. I will likely be able to elaborate later.
Finally, I'd like to say Star Wars is definitely not consistent. Many of its individual entries are consistent, but the whole is endlessly contradictory. The Last Jedi is My favourite Star Wars movie because it is the most science fiction. It clearly tries to build a unified theory of the Force which George Lucas vehemently disagrees with.
But more interesting for our conversation is the bombers at the start of the movie. See, many fans criticised that scene because "well why did the bombs fall? It's space, there's no gravity." And I love the bombers because those fans are wrong. Space has gravity. It's orbit that doesn't have gravity. Usually in space movies, ships are in orbit. The Enterprise is always flying into orbit. But in Star Wars, antigravity technology is cheap! You don't need orbit when you can make your ships hang in the air in much the same way bricks don't! That battle wasn't in orbit, and that's why they couldn't just rotate the bombers to shoot the bombs at the enemy ship! That's science fiction! Consider the consequences of cheap antigravity tech, and write based on that worldbuilding.
The Hobbit is about courage and subverting social expectations and perceived limitations.
hey don't forget potter's prominent themes of "progress is bad and adherence to the status quo is the ultimate goal" and "slavery is good and anyone seeking to abolish it should be ridiculed".
i think "fun is different than good" is a pretty bad stance. "guilty pleasures" are just a way to deal with feeling shame for liking something. you have to like it ironically because genuinely liking it is seen as shameful due to it not reaching some standard. screw that, i say.
i'm reading it as you saying "good" is somehow a more objective quality? correct me if i'm wrong. i'd instead reach for something like quality. the hobbit is unquestionally high quality writing, and it is set in a fantasy world, which i would say makes it fantasy without being science fiction.
"junk food" can be another word for "comfort food". sometimes it's not about nourishment, but about feeling good.
my take on the last jedi is that it definitely tried to set up something bigger, unlike the rest of the disney creations. but sci-fi? nah. it's at best a political thriller with some vfx. there is no detail given to the inner workings of the world, things are just there.
as i've understood the lens you've presented, i'd say the phantom menace is much more sci-fi than the last jedi, to its detriment. it tries to set up utterly massive systems, spends a long time on examining their details, and just spins its wheels for about an hour of its runtime.