The facts, few as they are
Joss, in an online interview which is no longer on-line, stated that the Firefly verse started as a campaign of a major Sci-Fi RPG. He's also stated that he quit playing RPG's after college. He refused to identify the system.
Since he graduated in 1986... we're looking at 1982-1986.
On to the logical speculation
The criteria "major" restricts us to games that were in commercial circulation; since his first couple years undergrad were in the UK, that restricts it further
Major Sci-Fi games in print included: GDW's Traveller & 2300, Palladium's Mechanoids trillogy & Robotech, FASA's Star Trek, FGU's Starships and Spacemen and Space Opera, TSR's Star Frontiers, and ICE's SpaceMaster, and SPI's Universe.
If we eliminate the games without slugthrowers, we're left with Traveller, 2300, Universe, Mechanoids, and Robotech.
Universe was not in very wide circulation; SPI went bankrupt in 1982, and the Universe game was one of SPI's last RPG products. My experience is that SPI games didn't have much of an impact in the UK.
2300 was later in the era - summer 1986 - he wouldn't have been playing it during his college career.
Mechanoids lacks ship rules, and is focused on active duty games. Plus, no big bad government. No Mercantile focus, either.
Robotech is very light on the slugthrowers, and lacks a mercantile focus.
2300, Space Opera, and Traveller have world generation rules, but only Traveller and 2300 have system generation rules.
Space Opera has everything but the right weapons.
Also, Traveller and Star Frontiers were available worldwide by 1983... and Traveller was even produced in local editions for the UK.
Which leaves, pretty much...
Traveller or...
something that isn't major, or...
serious houserules from hell.
Notes in support of Traveller
I'll note as well: The Firefly class is very much, feature for feature, comparable to the specifications for the Type R Subsidized Merchant in Traveller, but with a second launch added in place of 20 tons of cargo. Deckplans differ, but still, functionally, very close.
Further, the Verse uses a high speed deep space drive, and an atmosphere capable maneuvering drive. The rest of the tech lines up really well, tho'. You need only redefine jump drive to a sub-C secondary n-space constant speed drive, and you have the ability to play the verse with stock traveller rules. Including the small vs big ship dichotomies.
Further, I've accidentally generated 5 star systems before using Classic Traveller's Book 6: Scouts; it's the only one of the lot that I recall being able to generate stellar clusters of that size; I can't get to Space Master to check.
Traveller is often derided for the "Sixguns and Starships" approach... an approach which Firefly shares.
Classic Traveller adventures often have PC's skirting local laws to do what basically amount to good deeds in the end, and/or to make enough bankroll to keep flying. Plus, the central government is often treated as the enemy in those adventures.
Evidence from the Footage
Several references to "Travelers" in the footage are made. The most memorable being Wash, shouting, "Hang on, Travellers!" during some sharp maneuvers while trying to evade the Dortmunder in the pilot.
I recall that there are at least 2 such comments on film.
Note that the leaked shooting script uses the double L spelling.
Zoe Travellers pick 'em up cheap at government auction. A few modifications and they serve well enough for a one-way push to the outer planets.
Bushwhacked, Act 1
And also in the pilot, another use in the script - but not in the footage:
INT. DINING ROOM - LATER
We see a sparse but none-the-less inviting spread - Book and Kaylee have made a salad of tomatoes, and grilled up some root vegetables along with the pasta and protein/starch mush that is the usual diet of space travellers. To us, not much. To this crowd, a banquet
.
This could be a Britishism, due to Joss having gone to college in the UK.
Worlds in both.
Regina is a major world in both settings. In CT, it's a subsector capital, and important trade center
Bellerophon is a water world in both settings; See CT Adventure 9: Nomads of the World Ocean.
Ariel (Ley Sector, Ikhnaton subsector, hex 0103) is a significant world in the Judges' Guild produced Ley Sector - on a major route, but not actually core.
Persephone is a low-population world in the Spinward Marches. In Traveller, it's an Imperial Way Station, with a thin but breathable atmosphere, close to earth size.
Conclusion
We can't be certain, but the most likely candidate is Traveller.
A Couple Notes on the "Official" 'Verse
The official verse has at least 5 natural stars, plus 7 human-ignited protostars. At least according to the QMX produced additional materials.
It's worth noting, however, that the additional elements and detailed astrography seem to be later growth after filming.
