Traveller Store CotI Features New Posts Mark Forums Read Register


Go Back TravellerRPG.com > Citizens of the Imperium > General Traveller Discussions > Software Solutions

Software Solutions Discussions on Traveller related software.

Reply
 
Thread Tools Display Modes
  #11  
Old March 26th, 2009, 12:18 AM
inexorabletash's Avatar
inexorabletash inexorabletash is offline
Citizen: SOC-14
 
Join Date: Sep 2005
Location: San Francisco
Posts: 1,038
Gallery : 1
inexorabletash has disabled reputation
Default

Ooh boy, one of my favorite topics.

First off, I just put this page up a few days ago:

http://www.travellermap.com/formats.htm

This is the portion of MWM's article from Challenge #26 that describes the Standard UPP Format for sector data interchange. I figure this represents the "1.0" standard. It's quite retro - check it out! Note that the two halves of the article don't (data format, data interpretation) exactly match.

I would retroactively call the format used for the GEnie uploads, which forthekill describes as the "standard .sec format", as the "2.0" standard. The differences are add Name, replace G with PBG, drop Tradeworld (app-specific), drop Explored? (app-specific), reorganize fields (alleg/zone/gg to zone/pbg/alleg/stellar) Note that the field sizes themselves can be determined by parsing the header, and this allows for extensibility - field lengths can be changed or added - although the format doesn't strictly dictate how to identify the start or end of the header or where the data starts.

But as is noted, it all goes higglety pigglety after that. There are several other variations as well:

* Inconsistent delimiting of world names from hex location (sometimes '.' terminates a world name, sometimes no space at all)
* Sometimes stellar data is collapsed (G2V)
* Funky stellar data like * and []
* Some funky allegiances like Im[V or JP/Jr
* Inclusion of LRX fields
* Era-specific zones (e.g. B in TNE data); sometimes no space delimeter
* Novel comments like O:1234 ("owned by hex 1234")
* Routes with $ prefix


And so on. Jeepers!

I used to try and parse everything, but eventually gave up and cleaned up the data I cared about. I still use a regex, as robject points out you can match on the Hex, UWP, and PBG pretty reliably. The one case you really need to watch out for is:

Code:
NavalAndScoutBase    0101 A123456-7 A                         123 IM
AmberZone            0101 A123456-7                         A 123 IM
That's why the regex coliver988 quotes restricts the codes to starting within a few characters of the UWP.

BTW, the only reason TravellerMap.com doesn't spew out stellar data is that the map doesn't consume it, so I haven't made a robust parser and/or cleaned up the data. I'll do that at some point.

TravellerMap.com doesn't care about field lengths at all; if I was feeling clever I'd have coded it to spit out the fields with dynamic length (just enough to fit the longest name)
Reply With Quote
  #12  
Old March 27th, 2009, 03:27 PM
forthekill forthekill is offline
Citizen: SOC-11
 
Join Date: Oct 2008
Location: Boston
Posts: 88
Gallery : 0
forthekill Citizen
Default

The variable lengths wouldn't be an issue if the data was just somehow delimited with something other than spaces.

inexorabletash, your post, and that page, is pretty informative.

I like the idea of "versioning" the different .sec formats to make it easier to reference them, even if it is an informal label. It might make it easier to get people writing or maintaining Traveller software to decide to support one version or another, or many, or least for others to understand exactly what format the software expects or spits out.

Part of the reason this is of interest to me is that I've been rewriting the old gensec software (in C++ rather than C) so that it is more modular (and adding a couple additional features), and it will allow you to choose the output from among the most common .sec formats.

The other reason is that I don't want to have to reprogram every tool I use to add support for various formats, so I'd like to write a simple converter that:

a. is independent of other tools
b. allows non-programmers to convert between .sec formats for use with whatever tools they need

I would like to be able to label these formats by something as simple as a version number, so maybe we can come up with a good system.
Reply With Quote
  #13  
Old March 27th, 2009, 03:37 PM
forthekill forthekill is offline
Citizen: SOC-11
 
Join Date: Oct 2008
Location: Boston
Posts: 88
Gallery : 0
forthekill Citizen
Default

inexorabletash,

I noticed that page you link to does not say what bytes the stellar data resides in. It lists the tables for the stellar descriptions, but that's it.

Is something missing?
Reply With Quote
  #14  
Old March 27th, 2009, 07:27 PM
Space Hamster's Avatar
Space Hamster Space Hamster is offline
Citizen: SOC-12
 
Join Date: Jan 2002
Location: Mission, TX, USA
Posts: 216
Gallery : 0
Space Hamster Citizen
Default

Fixed width fields are the old way of doing file formats. XML is not overkill. It is the logical successor to the fixed width fields. Most modern programming languages have build in support for reading and writing XML files. XML is by far easier to read than a fixed width field file as you know what values your working with because it has the information wrapping the values.

Fixed width fields are so 80’s. Get with the 2000’s and embrace variable length fields.
__________________
I used to be respected, you know. My word carried weight. One tiny mistake, and suddenly no one trusts me!
My troops were acting strangely-plotting something. Obviously, they were traitors; all the warning signs were there.
The smart move was to kill them all. How could I know they were planning a surprise party for my promotion?
- Captain Jeelg
Reply With Quote
  #15  
Old March 28th, 2009, 12:04 AM
inexorabletash's Avatar
inexorabletash inexorabletash is offline
Citizen: SOC-14
 
Join Date: Sep 2005
Location: San Francisco
Posts: 1,038
Gallery : 1
inexorabletash has disabled reputation
Default

Quote:
Originally Posted by forthekill View Post
I noticed that page you link to does not say what bytes the stellar data resides in. It lists the tables for the stellar descriptions, but that's it.

Is something missing?
Nope - but the page is from MWM's Challenge #26 article, which itself has two halves:
  • Description of the file format (used in Trader, for the Apple II, and intended as an interchange format)
  • Compilation of what the fields mean, from the various sources.
As noted on the page, these don't agree - the tables list nIn and nAg whereas the file format stipulates 2-letter codes (Ni and Na). Stellar data is not found in the file format.

Interestingly, in the first part of the article (not included, I could type it up), MWM laments the paucity of good Traveller software available at the time, and suggests that a lack of a standard file format is one cause.

Regardless of what standard is suggested for the future, there is a lot of legacy data out there. It seems a worthwhile exercise to document that, so that tools that produce and consume some future standard format can also interoperate with old data. (This is analogous to the HTML 3.2 standard, which was basically "what do all of the current browsers do?" rather than attempting to forge any new ground.)
Reply With Quote
  #16  
Old March 28th, 2009, 08:53 AM
coliver988's Avatar
coliver988 coliver988 is offline
Baron
 
Join Date: Dec 2003
Location: Asheville
Posts: 1,534
Gallery : 103
Visit coliver988's Blog
coliver988 Citizen+coliver988 Citizen+coliver988 Citizen+
Default

Quote:
Originally Posted by Space Hamster View Post
Fixed width fields are the old way of doing file formats. XML is not overkill. It is the logical successor to the fixed width fields. Most modern programming languages have build in support for reading and writing XML files. XML is by far easier to read than a fixed width field file as you know what values your working with because it has the information wrapping the values.

Fixed width fields are so 80s. Get with the 2000s and embrace variable length fields.
depends. I work with banks, credit card companies, telcos for data exchange. ALL use fixed-width fields. I don't see that changing anytime soon, and I've been doing this since 1986. Inertia has a lot going for it
Reply With Quote
  #17  
Old March 28th, 2009, 09:12 AM
Gadrin's Avatar
Gadrin Gadrin is offline
Citizen: SOC-14
 
Join Date: Mar 2001
Location: Redlands, CA USA
Posts: 1,574
Gallery : 1
Gadrin Citizen
Default

Quote:
Originally Posted by inexorabletash View Post
TravellerMap.com doesn't care about field lengths at all; if I was feeling clever I'd have coded it to spit out the fields with dynamic length (just enough to fit the longest name)
Yeah, I thought so.

I did 3 imports by hand and not script last summer and I think they were
all different: Spinward Marches, Far Frontiers and Reaver's Deep.

>
__________________
No plan survives contact with the enemy.

Most plans don't even survive contact with reality.

Thanks, Gadrin
Reply With Quote
  #18  
Old March 28th, 2009, 09:16 AM
Gadrin's Avatar
Gadrin Gadrin is offline
Citizen: SOC-14
 
Join Date: Mar 2001
Location: Redlands, CA USA
Posts: 1,574
Gallery : 1
Gadrin Citizen
Default

Quote:
Originally Posted by coliver988 View Post
depends. I work with banks, credit card companies, telcos for data exchange. ALL use fixed-width fields. I don't see that changing anytime soon, and I've been doing this since 1986. Inertia has a lot going for it
Yeah, I did a parser for a credit card strip reader and all that info was
fixed width data.

>
__________________
No plan survives contact with the enemy.

Most plans don't even survive contact with reality.

Thanks, Gadrin
Reply With Quote
  #19  
Old March 28th, 2009, 01:35 PM
inexorabletash's Avatar
inexorabletash inexorabletash is offline
Citizen: SOC-14
 
Join Date: Sep 2005
Location: San Francisco
Posts: 1,038
Gallery : 1
inexorabletash has disabled reputation
Default

Quote:
Originally Posted by Gadrin View Post
I did 3 imports by hand and not script last summer and I think they were
all different: Spinward Marches, Far Frontiers and Reaver's Deep.
Hrm... well, looking at the code (I'm rigging up stellar data), it appears that I'm using a fixed-width format right now for the SEC.aspx output. You may have been clicking on the links at the bottom of the main map page (credits section) which just give you the source data (often from the original publisher). Those are definitely random.

The SEC.aspx output format is:

Code:
"{0,-25:name} {0,4:hex} {0,9:uwp} {0,1:base} {0,-25:codes} {0,1:zone} {0,3:pbg} {0,2:allegiance} {0,-20:stellar}";
(The stellar chunk isn't live yet)

I also plan to add a "v2" compliant header, i.e. names of the fields and their lengths.
Reply With Quote
  #20  
Old March 28th, 2009, 02:34 PM
aramis's Avatar
aramis aramis is offline
Administrator
 
Join Date: May 2001
Location: Anchorage, AK, USofA
Posts: 29,355
Gallery : 53
Visit aramis's Blog
aramis has disabled reputation
Send a message via ICQ to aramis Send a message via AIM to aramis Send a message via Yahoo to aramis
Default

Quote:
Originally Posted by coliver988 View Post
depends. I work with banks, credit card companies, telcos for data exchange. ALL use fixed-width fields. I don't see that changing anytime soon, and I've been doing this since 1986. Inertia has a lot going for it
Inertia and clarity of coding...

then again, I recently (last year) saw an add for a bank looking for a person familiar with Cobol, and C++, to rework one of their programs into C++...
__________________
~ Aramis
aramis.hostman.us /trav
Smith & Wesson: The Original Point and Click interface!

Archduke of Sylea (CORE 2118)
Duke of the Third Imperium (SPIN 0534)
Count Terra (SOLO 1827)
Count Gorod (REFT 1302)
Count of the Third Imperium (SPIN 2232)
Viscount of Adabicci (SPIN 1824)
Marquis of the Solomani Rim (SOLO 0606)
Marquis of the Third Imperium (SPIN 2410)
Baron of the Third Imperium (SPIN 2231)
Knight of the Iridium Throne (CORE 1434)
Sir William Hostman (OLDE 0512)
Sir William Hostman (DAGU 0622)
Knight of Deneb (REFT 2239)
Knight of Deneb (Spin 2532)
SEH w/Diamonds for Extreme Heroism - Battle of Boughene
MCG - Battle of Boughene
TAS: William Hostman (CORR 2506)
TAS: Bearer (DAIB 1326)
IMTU ct+ tm++ tne tg-- tt+ tmo+ t4- t20+ to ru+ ge+ 3i+ c+ jt au ls pi+ ta he+ st+
Wil Hostman 0602 C539857-9 S A724
OTU: 95% 3i an+ au+ br- cpu dt f+ fs++ ge ih- inf j jf+ jm+ jt+ ls- n= nc+ pi+ pp-- tp+ tr+ tv- vi-- xb+-
Unless there is bold red text, presume my posts to be my personal material only.
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Galactic Sector FIle BetterThanLife T20 - Traveller for the D20 System 2 May 22nd, 2006 10:45 PM

This website and its contents are copyright ©2010- Far Future Enterprises. All rights reserved. Traveller is a registered trademark of Far Future Enterprises .
Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright (c) 2010-2013, Far Future Enterprises. All Rights Reserved.