Ah, I hadn't read the book, but I have a copy of Barlow's Aliens that covers it - intriguing idea, but haven't sprung it on my players yet. If I did, I'd probably hide it on a waterworld though, to make it a little harder to figure out what the heck's going on.
The biological equivalent of 'grey goo'? My players kinda freak at the very idea of rogue nanites in Traveller - nuke the site from orbit, interdict, and even then, you can't -really- be sure. MGT's Trojan Reach has a planet that's basically a constant struggle against these things.
You could use nanites to pull a more Solaris type effect - the planet could be essentially a more serious version of Magrathea - a built planet that builds ... well, artificial lifeforms, spaceships, moons, etc. as needed by its original architects (hopefully long absent) with matter replenished by drones (ship/creature nanite constructs) that farm the asteroid belts, comet belts, etc. Now looking for purpose, said planet might, once it understands the player's language, wants needs, try to give them their heart's desire - and either misunderstand, or draw the attention of every starfaring power in the area to claim it.
IIRC, there's also a world in the Solomani Rim that has something freaky, uber Gaia-ish going on (from Interstellar Wars era at least - it might have survived to other eras, idrc.), but it shifts lifeforms not automagically in front of your eyes, but over months, etc. Subtle for a certified traveller visiting once in awhile, but crazy-ridiculous to any scientist paying attention over a few years. The anecdote was that a tree long known to be poisonous altered itself so that it was delicious and nutritious when a stranger stopped by and sampled it, scaring the heck out of the farmer who's land it was on.
