What I really don't have a solution for are islands only accessible by scientists.
I think that even all (??) of those can be visited with time and money. For instance - there are occasional (expensive) tourist voyages to Henderson, Wrangel, Heard/Mcdonald and NZ Sub-Antarctic islands. Not that I plan to go on any of them!
Among other such islands I "counted"
a. Surtsey as our ship passed inside the maritime boundary of the core zone defined around the current (albeit sinking) island.
b. Gough Island as our zodiac was given permission to sail in "close" and we touched one of the stacs!! Later it didn't matter as UK added Inaccessible Island where we had actually landed together with a large maritime area which we also passed through. And we did actually see the endemic bird species of Gough on the shore
As you say - what counts as a "visit" is a personal matter - clearly it is better to actually enter the core zone and see/touch what was regarded as giving the site OUV but even seeing from outside the core zone can be enough. We circumnavigated both Surtsey and Gough so could actually have "seen" much more than someone who made a single brief landing in one place on a foggy day! In the case of Surtsey for instance would actually walking on the volcanic ash/rocks, and touching (if allowed!) the newly growing plant life (which we could see perfectly well through binoculars) have added anything to our experience/knowledge (we had done this on Heimaey just a few kms away with "new" rocks of a similar age anyway). This of course raises a whole series of philosophical questions about "why" visit anywhere in the first place!! Does it make one a better/more knowledgeable person than someone who hasn't visited a particular place but perhaps has become a great (armchair/academic) expert on it.
A couple of years ago we visited the Indian Sundarbans and were surprised, when carrying out a very detailed analysis of the UNESCO documentation, to discover that it is not possible as a tourist to visit inside the core zone (this isn't true of the Bangladeshi Sundarbans by the way) - so all those tours advertised to the Indian Sundarbans WHS are not strictly correct. OK - by a very strict definition we haven't visited it even though we have
a. seen inside the core from a distance
b. travelled and walked through land which is essentially the same and is situated inside a protected area albeit not an inscribed one
It would seem to be "obsessive" not to "count" it!