Maybe I started with the cart before the horse in focusing on what are the sharp edges that people knock against in Spelljammer. Perhaps first we ought to start with 'why bother at all'? In other words what is a Spelljamming campaign supposed to be like?
(photo of own copies of Spelljammer core books, Polyhedron #151 magazine)
From the original 1989 boxed set the point was to add space to AD&D. It said it on the box "AD&D Adventures in Space" - throw a toolkit to players and DMs and leave it up to them what they did with it. I find only a single page in the Lorebook of the Void that suggests what a party might do once in space - trading, pirating, privateering or enlisting, exploration, intrigue or reconnaisance and adventuring or "cruising around and seeing what attacks us." This wide open, everything-in approach left it very much up to tables what they did with all this.
Some of the elements introduced as part of the setting - the hippo-folk Giff, Krynn-ish comedy artificer Gnomes and Giant Space Hamsters - get highlighted as the silly aspects of things that put people off. I was neither here nor there on the Giff, I liked Giant Space Hamsters, because why not? and Gnomes got shoved into the background for me by the far more interesting illithids and neogi. Interestingly, beholders also got a major treatment as a space-based faction but they did not spark with me.
"Shadow of the Spider Moon" - the d20 mini-game from 2002 has "Swashbuckling Space Fantasy" as its sub-title, urging players to "strap on your cutlass and trusty sidearm, and hoist the mainsail! It's time to take fantasy to the stars." Summarising the pitch in the contents page we get that corsairs and planetary explorers ride stellar winds to plunder ships and ruins, explore strange worlds and fend off the predations of space based aggressors. The editorial for the issue of Polyhedron which contained Spider Moon takes a swipe at the sillier aspects of the original and boasts it has dispensed with those.
So what did people do with all this at the table?
Stories seem to split between the wacky intersection of Spelljammer and traditional terrestrial play (such as a tale of how a level one character sold their starting ship and used the cast to build a castle and retire) and full-bore spacefaring ship shenanigans.
I noted a strong trend as I plumbed the net researching this - views got distinctly more generous as time passed.
Just after 3e dropped, there was deep hostility to the setting on Dragonsfoot (2002) mostly focused on the gonzo aspects of the setting getting away from sword & sorcery or high fantasy roots of the game.
I swept up positive comments on various forums with the topic of 'what is so good about Spelljammer' or 'what was done right' or 'what do you want to see in a new Spelljammer' including a Pathfinder conversion effort to give a rough inventory of elements people liked.
A critique from Yawning Portal (2017) hammers on Spelljammer as missed potential - that it focused on being a way to stitch together all the various existing TSR properties without taking the time to be its own thing. Planescape is offered as the counter example of what should have happened. Surprisingly, an item that cropped up again and again across multiple forums was praise for the way that Spelljammer could be used to easily stitch together the many and varied worlds of D&D without disturbing them. There was also an expressed sentiment that Spelljammers provided a more reasonable way for relatively ordinary folk to move between worlds (compared to using portals to cross the planes). I was surprised to see this multiple times myself.
The next most popular factors are more intuitive - the cool ships, the vast and weird environment to explore, strange aliens and the 'space-buckling' atmosphere - identified mostly as "Star Wars / Firefly / Farscape with magic".
The next group down starts to become the wishlist of factors to emphasise - space-faring powers and factions, the unique Ptolemaic celestial mechanics and a desire to have both good mechanics for space travel and exploration and a solid toolkit for generating new crystal spheres. This is coherent with laments for lack of support (2017) that highlight a missed opportunity to make exploration a great experience.
Pieces celebrating the setting (2010) flag the lack of steampunk aesthetic which is an unexpected take. Planescape did fold in an aesthetic that sat comfortably with steampunk even if there was little to no steam involved. Noting that Spelljammer has no steampunk at all is an interesting point given that steampunk elements are now so deeply embedded in the aesthetic of D&D at this point.
Finally from 2010 the Goblins Return and Heart of the Enemy are flagged as the original adventures that did the best job of show casing what Spelljammer was all about. The Scro - organised space-orcs - are flagged as great drivers for action - particularly as inciting a campaign when they arrive on a parties homeworld and start to organise the locals. (Checking my shelves, I have both modules more or less still in their raggedy covers that I found them in a Bristol games store. Priced at £3.99 and £4.99 respectively which gives you an idea of the age. These shall be reviewed in due course.)
If we are to take guidance from all of the above for a new Spelljammer revamp or reboot it would seem to be that the most popular aspects were the ships, aliens and the swashbuckling in space atmosphere combined with the sense of high weirdness to be found in the outer reaches. All this suggests that the way to go is to turn up the dial on everything that is unique to Spelljammer since none of these things can be found anywhere else. The only factor to watch is to retain its 'snap on' aspect where a given groundling campaign can head for space without requiring a massive ret-con of the world.