2016-08-01 ~ FGx Crossfeed Dailies Replay R1 ~ Warning: Crashes your browser

[theo-armour]

@fgx/owners

Warning:

Crashes your browser after your second pass through the 31Mb of data
Open in Incognito mode or in separate window

And something very peculiar is happening with the lat/lon

But the main thing:

Thank you, Geoff, for making the data available.

See also: https://github.com/fgx/crossfeed-dailies

FGx Globe Crossfeed Dailies Replay

Change Log 2016-08-01 ~ R1

• First commit
• Add Read Me

[geoffmcl]

Hi Theo,

Warning: just uploaded some 23 more cflogs to crash your browser ;=))

... something very peculiar is happening with the lat/lon

There should not be any anomoly in the lat/lon, but in reading globe-crossfeed-replay-r1.html I can see you base the tracks seemingly on the CALLSIGN...

But over the course of the day, any particular CALLSIGN may have done several flights, and they could be at diverse globe locations, like KSFO, LEBL, YSSY, etc, etc... so you should not join all the dots simply according to CALLSIGN! But maybe I read it wrongly...

Each CSV line commences with a FID - a unique flight ID... You should only join-the-dots according to the FID...

And I would suggest you pre-process the CSV lines, and break them it up into FIDS... so you can cull the FID records even more... check if the last dist_nm is GTT 2 nm, and the last tsecs is GTT than 60 secs... otherwise skipped...

Now specifically looking at flights-2016-07-30.csv - it has 351,616 lines, but the first is the csv header...

It contains 1,760 (unique) flights, but after the above culling, I am left with just 697 valid flights, spread over some 225 CALLSIGNS, so you can see quite a few did multiple flights...

To see this in detail try - http://geoffair.org/tmp/flights-2016-07-30.htm#stat.alpha - it looks like most CALLSIGNS have more than one flight, during that day...

A special mention about the callsign callsig ... this person/machine does strange things ... You can see on this day he/she/it flew 15 models, for 27 flights ... and I have seen this on other days also... quite weird... he/she/it seems out to create some strange multiplayer records...

Anyway, back to your globe, I think you should consider FID as the difinitive, unique, flight marker, dot joiner... even if you do not get into culling...

Regards,
Geoff.

[theo-armour]

@geoffmcl

R3 is up

Change Log 2016-08-01 ~ R2/R3

• R3 adds
  • Select file memu item, uses GitHub API to obtain list of files
  • More detailed flights menu
  • Flights based on FID not call sign. Thanks Geoff
  • No longer crashes browser (Or maybe just not as quckly ;-)
  • Objects appear at correct lat/lon
  • Better object management
• BUT trails still causing big slowdown /havimg memory issues? and not working as intended
• R2 was a dud

[theo-armour]

@fgx/owners

FGx Globe Crossfeed Dailies Replay R4

Change Log 2016-08-02 ~ R4

Back to creating objects foe each callsigns. Means fewer objects to update 60 times per second. ;-)

Will need to look at monitoring both callsigns and FIDs. Still too many zombie trails.
And there still appear to be some memory and performance issues tat should be cleaned up prior to adding features.

• Linking meshes and trails to callsigns
• Add more metrics
• Simplify mesh creation and trails tacking

[geoffmcl]

Hi Theo,

What you choose the cf csv logs to be is your choice... I can only tell you what they are ;=))

Simply, the crossfeed.freeflightsim.org/flights.json is fetched, each 5 seconds, and stored in a by FID hash... and each flight written to a daily csv... so starts just as a mirror of the first json received, which can have min seens of 3, up to to a max seen of 54 records... so far...

Now when the next json arrives, 5 secs later, first flights that have left are removed... from the hash... that FID will never again appear in the csv... the existing flights are updated with later info, and any new flights, FIDS, added, and appended to the daily csv...

So the csv list is not a list of continuous records... it is in fact a set of 5 second json blocks... daily, so each contains up to 17,280 blocks (x 5 = 24 hrs), provided there were no comms breaks etc... during the 24 hour period...

The blocks can be separated by comparing the upd with the upd of the previous record... as indicated, this will only change up to 17,280 times per day... and although the count in each block varies, that gives a csv file with some 400,000 lines...

Back to creating objects foe each callsigns. Means fewer objects to update 60 times per second. ;-)

Well, that depends on what you mean...

If the objects were based on FID, then when you read the next json block, you could eliminate, remove, delete ended FIDS, thus at eny one time you would have a maximum of 54 objects to update... sometimes a lots less, down to 3...

This seems a better display - show only FID flights still active - gives a better, clearer, sort of by hour usage, world wide, rather than the display being massivly clutter with dead zombi flights, that will have no update...

And lines between where a callsign ended the last FID flight, and started up a new FID flight, maybe half-a-world away, are very confusing, to say the least... and must be avoided IMO...

And when you wrap the file, everything should be deleted, and the globe starts a new day... maybe you could add a Sun object, and move it around the globe... you already show a current UTC time... ie the CSV upd value...

Your present crude next set of record reading seems to warp forward an hour about each 10 seconds... that file reading could in fact be based on a user time warp - Show 24 hours in [ 5] minutes... etc...

Now I am sure you can sustain up to a maximum of 54 flight (FID) full tracks at a time... at 60 frames per second...

Just an idea...

Regards,
Geoff.

[theo-armour]

@geoffmcl

Still a lot to learn here. Will get there eventually, I hope. ;-)

More tomorrow or so.

Got me some pretty maps to finish up...

[theo-armour]

@fgx/owners

FGxCrossfeed Dailies Replay R5

Change Log 2016-08-03 ~ R5

Still takes way to long to reload or close the page. This has to do with the trails - Three.ls lines.

• Generally being less of a memory hog
• Updates on tot_secs
• Still working on memory issues
• Still not deleting zombie flights

[pedromorgan]

@geoffmcl @theo-armour

Can we help each other out here..
What seems to be happening is that..

Cut + paste Coding

Theo is creating custom pages and layout and custom "theo style" cos it worked (cut+paste programming, like I used to..
What we need is some base templates, and a base javascript, and then the "custom stuff in the middle etc"..
Keep going theo.. its fantastic.. ;-)) and mean that....

To Future and making it perfect

We need to get theo to start using git.. and having a "server" running..
mainly because we need theo to see the way its structured and working on real server..

Updating locally and commiting will update live server also and alike...

So can I suggest we make a decision on what "platform to use".. Cos we can use these dynamically as virtual machines..
I got freeflightsim.org (ffs) domain so we can create
theo.ffs < next branch and latest dev
map3d.ffs < master branch and tested and cool

We also need to automate creation of csv files and alike also..
even perl script are welcome to post data, and points..

And this is same stuff as before and the flight tracker... ie marking points in space etc..

My Opinion..

Server Lang ?
For me golang, Geoff would like it cos it smells like c, and theo cos it easier javascript..
(note I hate coding go and javascript together cos my hair/teeth get curly braces)

and I would prefer to go with revel.github.io, as a webframework, cos it simple to use imho.. biased cos a developer... but defiately easier that a lof of other "frame-works"..

Next Step

@theo-armour create and openshift account and a "php5 webapp".. Were not using php, but for statics...
Then share it with geoff and myself..

[theo-armour]

@pedromorgan

You raise a lot of really good and interesting points. Thank you for taking the time to write these down

Please do remember that I am a designer and not a programmer. I happen to use a computer language as my design medium but that's like saying one person might use a sheet of paper to write a poem and another to add up a column of figures.

I have jotted down some of my thoughts here:

Sandbox Coding Style

I am always open to new ways of thinking and working - so let's continue the discussion...

Theo

[geoffmcl]

@pedromorgan, wow a whole mish-mash of things... thanks for taking the time... very educational, for me...

And thank you Theo, for adding the Sandbox link, also quite edifying...

Continuing the discussion... I hope ;=))

Some points observed -

1. Can we help each other out here... YEAH! All for it...
2. Cut + paste Coding - YAY, that is primordial in the understanding process...
3. To Future and making it perfect - wow, mish-mash of ideas here...
4. automate creation of csv files - absolutely, see scripts
5. Server Lang - But Theo is using client lang, javascript... only needs http server...
6. Next Step - openshift, or other - very open to discussion...

Pete, like all programmers I suppose, you moved on from cut-paste like I used to.., for some things... stupidly, for me, that is still the first cut way to do things... and then move it on to new things... it works...

As you begin to understand the process, you begin to create some base templates... libraries of repeated code... and I think Theo has understood this process... showed it in other endeavors...

Keep going theo.. its fantastic.. ;-)) and mean that....

A big YAY to that!

As you point out it is an ongoing process... get one thing displayed, and then see if you can make it better...

I am presently using my RPI2 to capture the 5 second crossfeed json feeds, and write the results to a daily csv file... then manually do the next steps, publish it... the latest of this script is cfjsonlog.pl...

It is not very robust, and can abort, but do not want to install say supervisor in my RPI2 machine...

But it would be easy to add to that script an sfpt/scp copy of the last csv, to another location, on each file change... maybe as a shell/fork to avoid disturbing the current json flight 5-secs collection...

Pete, any thing you can do to help this would be very, very welcome...

But come on Pete Geoff would like it cos it smells like c!!! Go never smells like C ;=))

Me embrace Go LOL... It smells like Google, oh wait, it is Google ;=)) Oops, nothing wrong with Google... thank you Google for all... and may some continue free...

But you could host say a cfdaily.freeflightsim.org, that displayed the daily cf csv logs collected above, on a global 3D map... that would be something...

WHOH! As always, just IMO ;=))

Regards,
Geoff.

[theo-armour]

@fgx/owners

Just a reminder. This GitHub issues thing is like a forum. It's wise to stick to the thread.

This issue is: #37 - with subject: ' blah blah FGx Crossfeed Dailies Replay blah blah'

There is also issue: #34 - with subject: 'Is FGx Moribund? If so, what to do'

If you ask me, we now have a STAR pattern with two issues that have converged into a path to a single runway.

In other words the two issues are in the process of a mid-air collision.

So, Pete, you are really good with starting fresh repos. Can you do the same with issues?

[pedromorgan]

@theo-armour not yet.. am busy busy..

BUT I want your space as a website to stuff..
so inmho I kind want
3d.freeflightsim.org online = master
3d-next.freeflightsim.org online = next branch

[theo-armour]

@pedromorgan

There's likely to be two kinds of 3D: globes and maps

I see you like master' and 'next'. I've been using 'build' and 'dev' or 'latest' if it's oriented to non-coding peeps.

@geoffmcl

What do you like?

All makes no difference to me as long as we all agree...

[geoffmcl]

@theo-armour, I too do not care about specific branch names, but unsure where branches fits in here...

Your creation of a HTML update, by increment the r? number, adjusting the index.html to default to latest, while it took me quite a while to get the swing of, is fine with me...

So for either the actual repo source, which as far as I have seen are usually all one branch...

And for the fgx.github.io html pages, where you are using version iteration, then again, branches does not seem to apply... so I am lost as to what we discuss here ;=()

What I like about this particular globe display, is that the set of available csv files automatically grows, as each new one is pushed... using the interesting urlAPITreeContents - https://api.github.com/repos/fgx/crossfeed-dailies/git/trees/gh-pages?recursive=1 - in json no less ;=))

And this is from a html file in sandbox\globe-crossfeed-replay\globe-crossfeed-replay-r6.html ... this is great use of cross-repo fetching...

It is a little disappointing that the trails only lead back a certain distance... I assume this is to keep the vertices from getting enormous, or something...

But after watching it for quite a while, one can get to like this like global game of golf ;=))

It takes some time to read one file. It looks like I guess at ?? lines per frame, say 60 frames per second, so that is many minutes, or more, to process a single CSV file of some 400,000 plus records... so one wish would be able to speed this up... but that would make each flight shorter... so maybe not so good... but if their full trail remained... and have not re-checked exactly, but a flight trail must be on a FID basis, not callsign...

But maybe a time-warp slider - you assume daily CSV - ie 24-hours of flight records - to be player over what period of time? You could set appropriate upper and lower limits...

Anyway, just an idea after watching many file replays, and not a single browser crash ;=))

Also, while you show the current Update: date, would also like to see an elapsed time... and the vertices: count, you should subtract the sort of fixed base scene count, of over 3 million, before any flights added, and thus show yourself the time increment occurring... but just little ideas...

Waiting for the next r7 here ;=))

[theo-armour]

branches so I am lost as to what we discuss here ;=()

And well you should be.

Currently there is no branching. I was waffling about as and when there could be branching.

I find that - if you are the only one contributing - that committing from my local to GitHub host is quite sufficient. But the moment there start to be pull requests then I should and would switch to build and dev or whatever branches.

What I like about this particular globe display, is that the set of available csv files automatically grows,

Yup. the GitHub API enables many fun things to happen

It is a little disappointing that the trails only lead back a certain distance... I assume this is to keep the vertices from getting enormous, or something...

Good assumption. There is somewhere that the code is preventing proper garbage collection. I think trails is part of the issue - but this is a work in progress. One day I want to see the full day of trails.

Thus creating FlyingSpaghettiMonster Type 2 (Type 1 being the STAR system )

It takes some time to read one file.

??? Really? For me it's almost instantaneous. All the issue for me seem to relate to dumping the previous day

All the other bits: we will get there...