GISDoctor.com 2016 Year in Review!

It’s 2017, so let’s talk about 2016.

Back in January of 2016 I wrote a blog post about my goals for the up coming year. I had a few goals I wanted to accomplish during the past 12 months. Unfortunately, I didn’t learn Mandarin Chinese (didn’t even really start), but I did become a better runner (and check out my runBENrun posts!). My main goal for the year was to become better at what I do and what I do is geo.

The first step to achieve my main goal was to reactivate my github account. I started several new repositories including uploading the code from my dissertation, adding a couple projects I reference often for  Spatial SQL and PostGIS queries, and runBENrun, a code base where I took my raw Nike+ data and built tools to analyze and visualize my running data.

Posting on GISDoctor.com was more active throughout 2016 with 10 new posts to be exact.  Not a lot, but enough to keep me motivated and active.  I hope for more posts in 2017! As always, I have plenty of ideas. Finding time to write them up is a totally different challenge.

Being an active OSM contributor was another goal for 2016 and early in the year I craft-mapped a ton. I mapped almost everyday in January, bringing some sweet craft mapping skills to some under-mapped areas. Perhaps I’ll do another OSM-mapathon sometime in early 2017.

Python, PostgreSQL/PostGIS, Node/Javascripting/Leaflet, and QGIS were my tools of choice for 2016 (and probably for most of you as well). I expanded my skills in each, keeping current to new trends and tech. I wanted to get better, and I think by taking the time (a few hours here and there over nights and weekends throughout the entire year)  I was able to learn new skills and technologies.

Now, why do I do all this extra work?  I do work in a job where I get to do a lot of very technical geospatial work, where I continually get to push my skills. However, due to the nature of the work, I don’t and can’t share it here. It was through these “at home” projects and posts where I pushed myself to continue to learn more, expand my skills, and share them with you.

There was one thing I wanted to do in 2016 that I totally missed out on.  I wanted to get more involved in the geo-community.  I didn’t. I will try again in 2017.  One good thing about our community is that there are always plenty of opportunities to get involved and make a difference.

The stats of 2016

The pageviews from GISDoctor.com were down this year compared to 2015. I think this is mostly due to the fact that in 2015  I had a post get on HackerNews that lead to a ton of traffic.

The top ten viewed pages for the past year are seen below.  Many of these posts are actually pretty old, but they all have long comment histories or have been posted in other locations leading readers back to the site.

gisdoctor2016

What’s on tap for 2017?  I have a few plans, but that is another post!

Posted in GIS | Comments Off on GISDoctor.com 2016 Year in Review!

runBenrun – These aren’t Heat Maps

I’ve gone back into my running data from 2014 and 2015 to build some density maps to compare to what I have run so far in 2016. Building a 10m grid for the region, I did some simple aggregations based on the GPS points captured by my Nike+ watch and processed through my runBENrun project (see it here on github).

These aren’t heat maps.  These are simple density maps.  There is a difference.

<start rant>

Please stop calling every single choropleth map a heat map.

</end rant>

From my running data, I can see some pretty clear patterns in where I ran.  In 2014, I kept my runs in Winter Hill, but ventured out into Cambridge and Boston a few times. A couple races in Boston show up, but the blue color range is only for a couple points per pixel.

2014 Run Density

2014 Run Density

In 2015, I changed the geography of my runs. I stopped with my Winter Hill routes and went out to the Minuteman Bikeway, venturing out as far as Lexington. The darker reds indicate where most of my runs were. Again, a race in Boston stands out as a single run, as do a couple runs into Medford and the southern reaches of Somerville.

2015 Run Density

2015 Run Density

My 2016 run density map to date is much different than the previous two years.  Firstly, I have put on a lot more miles this year than in past years, but almost all my miles were on the Minuteman Bikeway! I did run quite a bit into Cambrigde and Boston, mostly on my long Sunday runs as I prepared for my marathon. Like 2015, a vast majority of my runs were in Somerville and Medford, along the bike path.

2016 Run Density

2016 Run Density

When I combine all years I get a view of my running history that I have developed quite the habit for running close to home! The runs along the Minuteman Bikeway radiate red, as I have logged hundreds of miles along the route over the past couple years.  Even my adventures into Cambridge and Boston start to stand out, as I tend to use the same routes down Mass Ave, Boylston Street, and back into Somerville and Medford along Broadway in Cambridge.

All Run Density Map

All Run Density Map

This exercise didn’t reveal anything new to me, but it was a good exercise in thinking about different ways to display the data collected from my Nike+ watch through my runBENrun project.

Posted in GIS, GIS Analysis, Open Source GIS, Spatial Analysis | Comments Off on runBenrun – These aren’t Heat Maps

runBENrun Part 3 – Starting with PostgreSQL and Running Analysis

Kernel Density Analysis Performed in ArcGIS

Red – more runs – Blue – less runs

Run Ben, Run! Data Analysis!

Finally! I am at a point with my Nike+ app data transformation process from the raw TCX files to data I can work with.  It is now time to now build the data foundation for further analysis. For this phase of the project I decided take all the run data created in my text parsing code and load it into a PostgreSQL database.  I work with SQL scripts all the time, but professionally I work in a Microsoft environment, so it is nice to branch out and work with PostgreSQL.

The biggest PostgreSQL challenge I ran into was remembering to add the semicolon to the end of all my queries!  Otherwise, any other difference in syntax/code editors/software between Transact SQL and PostgreSQL/PostGIS were easy to learn.

The first step was to design and build a simple database to store the data.  The database is built around three tables:

  • rbr_AllRuns_2016_Points – table where I upload the points and attribute data built in the TCXtoText.py script. The  table will also store the geometry point objects and the geometry line segments between each point for a given run. To tie the individual runs to other tables I added a runid field, which was added to the input dataset in the TCXtoText.py script.
  • rbr_AllRuns_2016_ID – table where each run’s characteristics are stored, including date, runid, descriptive info about the run, total time, average pace, fastest mile in minutes,  and the fastest mile (which mile in the run).
  • rbr_AllRuns_2016_MileSplits – table that stores the runid, mile marker, and time (in minutes) I completed that specific mile.  The time data was calculated in the TCXtoText.py script and imported into the rbr_AllRuns_2016_Points table.

There are also several “temp” tables that are built to support the three main tables.  These tables were built to clean values, generate the line geometries, add the mile markers, and mile splits.  I call these “temp” tables, but I wrote them all to the database.  There only “temp” in the sense that I won’t use them (probably) for analysis.  Everything I need from them is in the main tables.

The code for to generate the required tables and populate the necessary data can be found on my github account – rBr_ProcessNikeGPSData.sql 

If you check my code on github, my table naming isn’t very consistent for these temp tables.  I will clean it up.

Early Analysis Results

I have started thinking about the analysis I want to start to build and I have played with a few ideas.  Some early queries have included classifying my runs by distance and speed, finding my fastest and slowest miles, and comparing mile splits across runs and distances.

  • To this point in 2016, my GPS has logged 219 runs and 442,174 GPS points, which account for 117 hours, 39 minutes and 14 seconds of running and 1126.78 miles. My marathon, for whatever reason, won’t export out of Nike+.
  • The 442,174 GPS points sometimes create interesting patterns.  For example, when zoomed into a street where I run every day, I get an interesting stripping of points. Without seeing the individual runs, it is tough to see if this is just noise or a real pattern. I know my GPS takes a reading every 0.97 seconds. Since I run the same routes so much, I believe the pattern gets amplified, creating the striping. It’s neat to see.

pointsample

  • Not tracked in my data – the three pairs of running shoes I have gone through this year. Adidas Supernova Sequence.
  • I built a Run Type field in my ID table, where I pseudo categorize my runs by distance and speed.  This categorization needs work, but so far I have more Awesome runs and Ehh runs. I’ll post the details on how I categorize these runs later.

Run Type

Total Runs

OK 83
Intervals 59
Awesome 34
Great 33
Ehh 10
  • My fastest mile that I ran that wasn’t in a race or during intervals was on April 13 at a 5:48 pace, cruising down the bike path in Somerville.

fastestmile

  • My slowest mile was on July 31 at an 8:08 pace, but I didn’t map that!

What’s Next

Now that I have my data in a format that I can quickly query the deeper analysis will now follow.  There are some data cleaning steps I need to add in during the loading process (like how to deal with pauses and breaks in the GPS data) and refining how I measure distance.

Feel free to check out the code on github and shoot me any suggestions/comments/ideas through twitter @GISDoctor.

Posted in GIS, Open Source GIS, Spatial Analysis | Comments Off on runBENrun Part 3 – Starting with PostgreSQL and Running Analysis