Traveller Store CotI Features New Posts Mark Forums Read Register


Go Back TravellerRPG.com > Citizens of the Imperium > General Traveller Discussions > Software Solutions

Software Solutions Discussions on Traveller related software.

Reply
 
Thread Tools Display Modes
  #1  
Old June 12th, 2014, 11:20 AM
robject's Avatar
robject robject is offline
Marquis
 
Join Date: Mar 2002
Posts: 13,933
Gallery : 9
Visit robject's Blog
robject has disabled reputation
Default JSON Sector Data Service

I want to discuss with Thalassogen, Draconian, and Joshua Bell about a sector data model.

I am updating my website, and I find that my server-side code is inexorably, slowly moving away from HTML generation and instead serving JSON data, which my webpages then render however they like.

So I am thinking about "how would I do this in a way that is convenient for me BUT also useful for others"?

My current format is verbose and detailed. There are many ways to organize this data, this is just one. I'm wondering if y'all would use it if it were in a different (better?) structure:

http://eaglestone.pocketempires.com/...pl?asJSON=true

The reason I am thinking about this is because my strengths are in server-side data crunching and production, but not persistence, and not presentation. Presumably, others are better at presentation (e.g. Travellermap!!!!) and other services I am currently forgetting. So, I would want others to be able to consume my data. Some level of interoperability seems therefore to be at least a good idea to think about.


I know we've been over this before. But talking about it is therapeutic to me.


Code:
{
  'meta': {
    'generated': 'Thu Jun 12 15:10:45 2014',
    'prng': 'burtle',
    'density': 'Standard',
    'sectorUID': 'mw-orion-4991',
    'hash': 'MD5',
    'tlCap': 21,
    'systems': 619,
    'civilization': 'Civilized'
  },

  '2632': {
    'population': 0,
    'importance extension': {
      'importance': -1,
      'expected ship traffic': 0
    },
    '_stellar': 'A6 II 10:A3 V G1 V',
    '_trade codes': 'Wa Co Sa',
    'pop mult': 0,
    'primary': {
      'spectral': 'A',
      'position': 'primary',
      '_uwp': '  A6 II',
      'decimal': 6,
      'spectral flux': 2,
      'size': 'II',
      'size flux': 3
    },
    'near companion': {
      'spectral': 'G',
      'orbit': '',
      'decimal': 1,
      'size': 'V'
    },
    '_uwp': '2632 Angindsan       B89A002-0    Wa Co Sa                003',
    'law level': 2,
    'starport': 'B',
    'hz variance': '+1',
    'size': 8,
    'hz': 9,
    '_extensions': '{-1} (C04-3) [1151]',
    'orbit': 4,
    'atmosphere': 9,
    'cultural extension': {
      'acceptance': 1,
      'symbols': 1,
      'strangeness': 5,
      'homogeneity': 1
    },
    'gas giants': 3,
    'mainworld type': 'Far Satellite',
    'government': 0,
    'belt data': [],
    '_bases': '  ',
    'economic extension': {
      'resources': 12,
      'efficiency': -3,
      'infrastructure': 4,
      'labor': 0,
      'ru': -144
    },
    'tech level': 0,
    'planetoid belts': 0,
    'tas': '',
    'name': 'Angindsan',
    'gas giant data': [
      {
        'number': 1,
        'orbit': '14.21',
        'satellites': 3,
        '_uwp': 'YLGG000-0:3 ',
        'size': 27,
        'code': 'LGG',
        'ringed': ''
      },
      {
        'number': 2,
        'orbit': '11.71',
        'satellites': 5,
        '_uwp': 'YLGG000-0:5 ',
        'size': 28,
        'code': 'LGG',
        'ringed': ''
      },
      {
        'number': 3,
        'orbit': '12.81',
        'satellites': 3,
        '_uwp': 'YLGG000-0:3 ',
        'size': 26,
        'code': 'LGG',
        'ringed': ''
      }
    ],
    'near star': {
      'spectral': 'A',
      'orbit': 10,
      '_uwp': '10:A3 V    hz7',
      'decimal': 3,
      'size': 'V',
      'hz': 7
    },
    'hex': '2632',
    'hydrographics': 10
  },

  ...(and so on)...

}

A sector using my verbose format below takes up a bit over 1 megabyte, and is a bit over 55,000 lines long.

That's a FAR CRY from a 30k SEC file.
__________________
Imperiallines magazine
My Helpful Stuff for Traveller5
IMTU tc+ t5++ 3i(+) au ls+ / OTU 44% an+ dt+ ge- j- jf+ n- pi+ pp+ tr+ tv- uwp+ xb+
Tools Link

Last edited by robject; June 12th, 2014 at 04:24 PM..
Reply With Quote
  #2  
Old June 13th, 2014, 07:00 PM
Draconian Draconian is offline
Citizen: SOC-12
 
Join Date: Dec 2012
Location: Toronto, Canada
Posts: 296
Gallery : 0
Draconian Citizen+Draconian Citizen+
Default

I'm not a computer guy, so that may be over my head.

I've generated a few sectors on Thalassogen's SectorMaker utility and they are datafiles about 1 MB too. T5 generates mainworlds and an average of about 9 more secondary worlds per stellar-occupied hex. So the memory-size may be well-night unavoidable. The generation is subject to any future T5 WorldMaker (?) errata (if you add the correction, they are corrigenda).
Reply With Quote
  #3  
Old June 13th, 2014, 07:56 PM
robject's Avatar
robject robject is offline
Marquis
 
Join Date: Mar 2002
Posts: 13,933
Gallery : 9
Visit robject's Blog
robject has disabled reputation
Default

See, I've thought about Thalassogen's code, too. He doesn't write Perl. BUT that's no reason I couldn't potentially call his service, get a passel of JSON in return, and use it in some way. Vice versa too, of course. Those of us who are better at providing services therefore don't have to write client code, and those who are good at client code can mix 'n match existing services. Seems like a good idea to me.
__________________
Imperiallines magazine
My Helpful Stuff for Traveller5
IMTU tc+ t5++ 3i(+) au ls+ / OTU 44% an+ dt+ ge- j- jf+ n- pi+ pp+ tr+ tv- uwp+ xb+
Tools Link
Reply With Quote
  #4  
Old June 13th, 2014, 11:56 PM
inexorabletash's Avatar
inexorabletash inexorabletash is offline
Citizen: SOC-14
 
Join Date: Sep 2005
Location: San Francisco
Posts: 1,041
Gallery : 1
inexorabletash has disabled reputation
Default

Just a quick note to remember to set CORS headers so pages can "mash up" the data across sites. Basic tutorial on how to do it:

http://enable-cors.org/server.html

Then JS code in web page served from example.com can make XHR requests to a JSON service on your server.
Reply With Quote
  #5  
Old June 13th, 2014, 11:58 PM
inexorabletash's Avatar
inexorabletash inexorabletash is offline
Citizen: SOC-14
 
Join Date: Sep 2005
Location: San Francisco
Posts: 1,041
Gallery : 1
inexorabletash has disabled reputation
Default

Also, you need to quote strings in "double quotes" in JSON data. http://www.json.org/
Reply With Quote
  #6  
Old June 14th, 2014, 07:58 AM
tjoneslo tjoneslo is offline
Citizen: SOC-14
 
Join Date: Feb 2001
Location: Ferrisburgh, VT, USA
Posts: 2,902
Gallery : 2
Visit tjoneslo's Blog
tjoneslo Citizen+tjoneslo Citizen+tjoneslo Citizen+
Default

Quote:
Originally Posted by robject View Post
A sector using my verbose format below takes up a bit over 1 megabyte, and is a bit over 55,000 lines long.

That's a FAR CRY from a 30k SEC file.
This is similar to my experiences from a few years ago doing a similar effort with XML. My two responses to his are:

A megabyte, you're worried about a megabyte in this era where disk space starts at hundreds of gigbytes?

Json, like XML, compresses wonderfully. I was compressing my ~1Mb XML sector data into 15Kb. That is, smaller that the original SEC file, but with more data. The reason this is relevant for you is something like 98% of browsers understand data in a "gzip" format. This is a recommended practice when sending a large group of small files, or when sending a large blob (say a Mb of JSON). I don't remember (nor may have learned in the first place) the details of how this works, but it will be worth your time if you are worried about bandwidth.
__________________
Archduke of the Solomani Rim - Terra (Solomani Rim 1827)
Duke Akumid - Akumid (Vland 1628)
Marquis Yeremyh - Yeremyh (Solomani Rim 1804)
Marquis Hysyl - Hysyl (Deneb 2425)
Baron Regina - Regina (Spinward Marches 1910)
TAS member - Vipan (Empty Quarter 1038)
Be part of the history of Traveller:

https://wiki.travellerrpg.com/Main_Page
Reply With Quote
  #7  
Old June 14th, 2014, 10:24 AM
robject's Avatar
robject robject is offline
Marquis
 
Join Date: Mar 2002
Posts: 13,933
Gallery : 9
Visit robject's Blog
robject has disabled reputation
Default

Quote:
Originally Posted by inexorabletash View Post
Also, you need to quote strings in "double quotes" in JSON data. http://www.json.org/
Yes, we've been working with the JSON spec at work really a lot lately, and even though the official rules don't permit single quotes, it appears as though they're taken for granted willy-nilly. I don't know what the JS community thinks of this; I'm a PON* and YAML guy myself.


* Perl Object Notation, invented whenever Perl decided it needed a way to store references to data structures. It's very close to JSON. Of course the C community would naturally tend to gravitate toward curly braces for maps and square braces for arrays, wouldn't they?
__________________
Imperiallines magazine
My Helpful Stuff for Traveller5
IMTU tc+ t5++ 3i(+) au ls+ / OTU 44% an+ dt+ ge- j- jf+ n- pi+ pp+ tr+ tv- uwp+ xb+
Tools Link
Reply With Quote
  #8  
Old June 14th, 2014, 12:39 PM
inexorabletash's Avatar
inexorabletash inexorabletash is offline
Citizen: SOC-14
 
Join Date: Sep 2005
Location: San Francisco
Posts: 1,041
Gallery : 1
inexorabletash has disabled reputation
Default

Quote:
Originally Posted by robject View Post
Yes, we've been working with the JSON spec at work really a lot lately, and even though the official rules don't permit single quotes, it appears as though they're taken for granted willy-nilly. I don't know what the JS community thinks of this; I'm a PON* and YAML guy myself.
If you're using a conforming parser - like the one that's built into JS for the last umpteen years, instead of just eval()ing your code - it will fail.

Code:
JSON.parse("{'a': 1}") // throws SyntaxError: Unexpected token '
Reply With Quote
  #9  
Old June 14th, 2014, 03:02 PM
aramis's Avatar
aramis aramis is offline
Administrator
 
Join Date: May 2001
Location: Anchorage, AK, USofA
Posts: 29,463
Gallery : 56
Visit aramis's Blog
aramis has disabled reputation
Send a message via ICQ to aramis Send a message via AIM to aramis Send a message via Yahoo to aramis
Default

Quote:
Originally Posted by robject View Post
* Perl Object Notation, invented whenever Perl decided it needed a way to store references to data structures. It's very close to JSON. Of course the C community would naturally tend to gravitate toward curly braces for maps and square braces for arrays, wouldn't they?
Square brackets for arrays, absolutely - that's standard C/C++; the Curly Braces are used in C/C++ for defining code chunks almost exclusively - if something requires a single function as target object, but you want to do multiple things, wrap them in curly braces, and they become an unnamed function object.
__________________
~ Aramis
aramis.hostman.us /trav
Smith & Wesson: The Original Point and Click interface!

Archduke of Sylea (CORE 2118)
Duke of the Third Imperium (SPIN 0534)
Count Terra (SOLO 1827)
Count Gorod (REFT 1302)
Count of the Third Imperium (SPIN 2232)
Viscount of Adabicci (SPIN 1824)
Marquis of the Solomani Rim (SOLO 0606)
Marquis of the Third Imperium (SPIN 2410)
Baron of the Third Imperium (SPIN 2231)
Knight of the Iridium Throne (CORE 1434)
Sir William Hostman (OLDE 0512)
Sir William Hostman (DAGU 0622)
Knight of Deneb (REFT 2239)
Knight of Deneb (Spin 2532)
SEH w/Diamonds for Extreme Heroism - Battle of Boughene
MCG - Battle of Boughene
TAS: William Hostman (CORR 2506)
TAS: Bearer (DAIB 1326)
IMTU ct+ tm++ tne tg-- tt+ tmo+ t4- t20+ to ru+ ge+ 3i+ c+ jt au ls pi+ ta he+ st+
Wil Hostman 0602 C539857-9 S A724
OTU: 95% 3i an+ au+ br- cpu± dt± f+ fs++ ge± ih- inf± j± jf+ jm+ jt+ ls- n= nc+ pi+ pp-- tp+ tr+ tv- vi-- xb+-
Unless there is bold red text, presume my posts to be my personal material only.
Reply With Quote
  #10  
Old June 14th, 2014, 07:05 PM
robject's Avatar
robject robject is offline
Marquis
 
Join Date: Mar 2002
Posts: 13,933
Gallery : 9
Visit robject's Blog
robject has disabled reputation
Default

Quote:
Originally Posted by inexorabletash View Post
If you're using a conforming parser - like the one that's built into JS for the last umpteen years, instead of just eval()ing your code - it will fail.

Code:
JSON.parse("{'a': 1}") // throws SyntaxError: Unexpected token '
Another problem I've seen is that some JSON appears to use "bare string" keys, and even I know that's a no-no.

Hmm. Looks like the JSON I return for my world-building stuff is double-quoted, and yet both it and this sysgen program use the same module for outputting the data.

Very interesting... I wonder what's going on...


AHA! My world-builder code has this directive to the dumper:

$Data:: Dumper::Useqq = 1;


Duh. Apparently I knew about this when I wrote it. Funny how easy it is to forget things.
__________________
Imperiallines magazine
My Helpful Stuff for Traveller5
IMTU tc+ t5++ 3i(+) au ls+ / OTU 44% an+ dt+ ge- j- jf+ n- pi+ pp+ tr+ tv- uwp+ xb+
Tools Link
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

This website and its contents are copyright ©2010- Far Future Enterprises. All rights reserved. Traveller is a registered trademark of Far Future Enterprises .
Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright (c) 2010-2013, Far Future Enterprises. All Rights Reserved.