Several sobering revisions later.....
I decided to go through my samples, resampling everything down to mono where necessary, resampling from 48kHz to 44.1kHz where obvious, & redesigning as basic as possible, my sample player!
Now I have previously tried such refinements as all mono samples (didn't seem to make any difference), but missed different sample rates as a possible cause...!
However!
A paranoid thought occured that if I were checking 'quietist', for stealing mode in the polyphonizer, the quietist note it could steal was a note-on with velocity 0. ie a note-off!
Now I'm sure, Seb, you have not missed this one (note-stealing generates a note-off I imagine...), but polyphonizing with note-stealing = oldest, eg, I don't get the problem! Well, so far at least.
So, changing the polyphonizer's note-stealing algorithm seems to fix it! Which agrees with your (Seb's) analysis of where the root lies. I just hope this observation might help.
I've been half-following Jersmi's polyphonic envelope thread, which might help with the clicks on note-offs, but otherwise, I'm reckoning on trying it out next week on a gig.
Very basic sample-player here.....
I'm sure you're all way ahead of me...

dave