What would happen to a penal/prison system of a world (or solar system) once it joined the Empire?
Depends on how you choose to handle worlds joining. It's usually portrayed as being a fairly long process, with the equivalent of a formal treaty required to be ratified to join. Think of it as how the Republic of Texas joined the USA, for example.
If the prison world is the claimed territory of some other polity, there's going to be some diplomatic stink raised, as recognition of the sovereignty of that other polity is going to prohibit grabbing chunks of their turf. However, if the original polity joins the Imperium, then there would presumably be some language in the treaty that covered joining that addressed the status of the subordinate world if the diplomats involved have enough brain cells to come in out of the rain. Member worlds have the right to pass their own laws, as long as they recognize some basic "sophont's rights" issues (defined by the GM, but should probably start at "no slaves", and go as far as the GM would like), though Imperial law does override in cases of conflict.
Would those convicted have a second chance?
If they can convince someone in a position of authority that their conviction or sentencing violated Imperial law, then probably; otherwise, it would seem unlikely. Obviously, this is going to depend on how you define "Imperial law" -- if you as the GM decide that it will include some sort of "freedom of religion" clause, then someone imprisoned for the crime of belonging to an unpopular faith could very well have grounds for an appeal. Will anyone dig deeply enough to find out what actually happened to the prisoners? Will the actual records be trumped up to make it look like they were all convicted of other crimes instead? Will the nature of the prison be made clear to Imperial officials, or might it be presented as a voluntary sequestration? ("Those guys? Oh, they're monks, who have chosen a life of solitary contemplation and celibacy to demonstrate their dedication to the Flying Spaghetti Monster. No, they renounce all non-essential technology, so you can't call them to ask about it.")
Would a penal world were all the guilty are dumped (like Australia was) be considered a separate world and given status?
Someone's going to have established sovereignty over the world -- you'll note that (for example) France never tried to seize Australia once it had become clear that Britain laid claim to it. While it's possible that various parties might press for a penal world to declare its independence and then apply for Imperial membership, the Imperium has to establish some position in regard to the sovereignty of all neighbor states, and that position will define what can happen next. There's no "astra incognita" where unknown polities with no previously-defined relationship to the Third Imperium could exist, at least in games not set in something like Milieu Zero. Biting off a world like that would require failing to recognize some other polity's claim to the world; it's a very aggressive diplomatic move.
What would happen if the IISS landed on a world that was actually a penal world and did not know it was some other systems dumping ground?
Why wouldn't they know? There's got to be some sort of orbital presence to keep unfriendlies from arriving and either freeing prisoners or providing them with things that the imprisoning government probably doesn't want them to have. When the Scouts do their initial survey, they'll certainly spot that orbital presence, and try to figure out what it is, and who put it there.
If the prisoners some how were able to send a distress signal to a ship that had no knowledge that the world was a penal system?
Things like "political allegiance" will be some of the most basic information available in any sort of navigational database. You may not have detailed physical information, and your communications codes may be years out of date, but "this system belongs to Polity X" is just about at the same level as "stellar class" or "habitable world present".
The details of "this is a prison" may not be included, but the local pickets will certainly be a clue of some sort. The absence of proper port facilities and navigational aids will be an issue as well. There's going to have to be some explanation for how the inhabitants got where they are, how they knew to signal passersby, what they're doing there in the first place, and why their living situation is the way it is. It's going to be hard to pass themselves off as survivors of a wrecked ship when there's no wreckage to see, and they may or may not have any particular equipment that you'd expect even shipwrecked spacers to have. There's also the question of what sort of things
were provided for the prisoners, and do they match up with the story; do they have any clothes that aren't orange jumpsuits labeled "Republic of Whatsis Penal Colony", for example?
