I took today off from work to catch up on a couple things, including installing Postgres 9.1 and PostGIS 2.0 on my home machine.  Lately, I’ve been neglecting how I manage my geo-data on my home machine for my personal projects and I told myself it was time for an upgrade. The recent release of PostGIS 2.0 was good motivation for this project as well!

I was planning a couple hours from start to finish for this project, and did I over estimate the time it would take!  After I downloaded, installed and configured Postgres 9.1 for a 64 bit Windows machine and ran the StackBuilder to install PostGIS 2, I had everything configured within a hour (and that includes me watching several random YouTube videos!).

I then used the handy shapefile loader to batch load a few global datasets and voila, PostGIS 2.0 data in Quantum:

It has been a really good year for the OpenGIS community so far, and the release of PostGIS 2 should really keep the momentum going.  I haven’t been a Postgres or PostGIS power user for a couple years now, but I hope to get back on track soon, exploring all the new features of PostGIS 2.0.

We should be very thankful to the community of developers who create and maintain these amazing tools and applications, who then give them away for free!  If you are a GIS aficionado and you don’t have PostGIS, Quantum, TileMill or any other open source GIS tools go and install a couple of them and test them out.

For many years Microsoft provided a WMS that included USGS topos and US aerial photography.  Popularly known as TerraServer (or Microsoft Research Maps -MSRMaps), the service no longer exists as of 5/1/2012, meaning that the Google Maps mash-up I created using the data no longer work.  For more info on TerraServer’s death check out the post on Directions Magazine’s All Points blog.

No worries from my end, as the use of that pages was nearly non-existent!  People are way more interested in my OpenLayers Google Mash-Up .  I’ll update my links accordingly, again, not that it matters because people weren’t interested in Microsoft’s WMS service.  Thankfully today there are many, many more options for streaming high quality basemap data through web mapping applications.

ArcGIS 10.1 will be released soon and according to Esri, ArcGIS 10.1 will be the company’s biggest release ever.  Plenty has been published by Esri about the 10.1 release, including an overview of  10.1 from the Resource Center, a number of videos from the 2012 FedCon and a 10.1 “What’s Coming” site.  From what I have read and seen I am looking forward to a number of improvements including those for spatial analytics,  saying goodbye to ArcSOC and ArcSOM, improved database management functionality (which there looks to be a lot of), and generating buttons with ArcPy scripts.

However, if you have visited this blog before you have probably have seen this post or this post and have read the comments about user experiences with previous versions of ArcGIS.  I’m sure these users have many items they would like to see in the next release of ArcGIS and in their spirit here is my ArcGIS 10.1 wishlist.

My Unsolicited ArcGIS 10.1 wishlist!

64 bit ArcMap:  It’s not going to happen with this release, although ArcGIS Server will be.  However, I can still dream about lighting fast geoprocessing until then.

Provide a Consistent Product: One of the biggest items on my 10.1 wishlist is software consistency.  Too often users encounter geoprocessing errors, screens of spatial death, or the map refresh freeze.  Problems like this become major choke points when using the software when it is a critical component of a business process.

Simplify the Product:  I believe this is where the competition is killing Esri.  Look at Quantum, MapBox, OSM, or SpatiaLite.  Simply and elegant products with big time returns.  I know that many, from the once a week user to all-day-every-day user believe that ArcGIS is too big.  I consider myself a power user and I probably only use 20 to 30 percent of the product.  If Esri could produce a lightweight, stripped down version that still gets the job done (and is bug free!) I think the user base would be very, very happy.

Improve the Map Draw/Refresh in ArcMap: Am I the only person who is not a fan of the way ArcMap draws and refreshes the map?  Have you ever waited and waited watching a map load on screen?  I find it frustraing that I can visualize vast numbers of points faster in an html5 application than I can on-screen in ArcMap.  Now, this may only be me, but I would love if someday Esri could improve the map draw/refresh rate within their product.  Esri has had the best mapping tools in the game for a long time and improving the draw/refresh rate would make the already strong set of cartographic tools even better.

Get Rid of ArcCatalog: Since ArcGIS 10 I have probably used ArcCatalog 10 percent as much as I did during the ArcGIS 9.X days.  The major components of what I need are now embedded in ArcMap, which is great.  Roll the rest of the functionality into ArcMap and send ArcCatalog out to pasture where it can go hang out with ArcView 3.x.

No More 999999 Errors:  There are too many general errors that occur on a regular basis.  From the comments in this post I know this happens to others as well.  If geoprocessing tools have limits to them let the user know what those limits are so they can work around it.  I run a very clean analysis machine, I don’t run third party extension (I write my own code)  and I get unexplained 999999 errors running mundane and simple tasks.  Tell me what I am doing wrong!

If Esri could make my wishlist come true I think their user base would be very, very, very happy.  I know that this wishlist is a little late for 10.1, but Esri developers and product teams are free to wrap all these suggestions into 10.2 (thanks!).  Now when will I be installing 10.1?  Probably sometime after the first service pack is released (September, October, November?).

On that note, what would you, the user, like to see in 10.1?  What would be on your wishlist?

Disclaimer: I’m not a beta tester for ArcGIS 10.1, as I am too busy using the current version to test the next version.  What I know about ArcGIS 10.1 comes from what I have seen on the interwebs.

Guido Stein, the official social chair of the greater Boston GIS community, has started a Meet-up group, Avid Geo, for anyone interested in “geospatial”.

Check it out here: http://www.meetup.com/avidgeo/. This should be pretty cool.  Groups like this can be a great way to get connected to others in the geospatial community.

So, Boston geo-nerds, check out the link and sign-up!

The concept of topology isn’t something that every spatially enabled person fully understands.  That is OK, because I too had to learn (and relearn) how spatial topology works over the years, especially early on back in the ArcView 3.X days.  I think this experience is fairly typical of someone who uses GIS.  If one is taking a GIS course or a course that uses GIS it is not very often that the concept of spatial topology is covered in-depth or at all.  Spatial topology also may not be something that people are overly concerned about during their day-to-day workflow, meaning they may let their geospatial topology skills slide from time to time.  As a public service here is a basic overview of geospatial topology.

First question: What is topology?

You have probably heard the term topology before, whether it was in a GIS course where the instruction lightly glazed over the topic, or in a geometry/mathematics course.

Technically speaking, topology is a field of mathematics/geometry/graph theory, that studies how the properties of a shape remain under a number of different transformations, like bending, stretching, or twisting.   The field of topology is well established within mathematics and far more complicated than I wish to get in this post.

Second question: How does topology relate to GIS and spatial analysis?

Spatial analysis is at its core an analysis of shapes in space.  Geospatial topology is used to determine and preserve the relationships between shapes in the vector data model.

The GIS software we use for analysis and data storage incorporates a set of “topological rules” to define how vector objects are stored and how they can interact with each other.  These rules can dictate how nodes interact within a network, how the edges or faces of polygons coexist, or how points are organized across space.

Back in the “olden-days” (which was before “my time”) GIS users, particularly ArcInfo users, were well versed in geospatial topology because of the coverage.  The coverage data model, a precursor to today’s ubiquitous shapefile format, was unique in that topology was stored within the file.  This data format allowed users a certain set of controls to the spatial relationships within the dataset that later went away with the shapefile.  The shapefile is not a topologically valid dataset, as geometric relationships are not enforced.  For example, how may of you have downloaded (or bought) a shapefile from a data provider and it was FULL of slivers? In the Esri world geospatial topology came back with the geodatabase, and has been incorporated into a number of other geospatial data formats including spatial databases supported by Oracle, PostGIS (2.0) and SpatiaLite.

Today, topology is important in geodatabase design (for those who pay attention to it!), and data creation/editing.  By understanding the set of geospatial topology rules and creating topologically sound data, the user can have a level of trust in their data during analysis.

 Additional Resources:

Esri white paper on GIS topology 

PostGIS Topology

PostGIS 2.0 Topology Support

Oracle Topology Data Model

Vector topology in GRASS

Esri Coverage Topology

Esri Geodatabase Topology

Real topology

Vector topology cleaning with Quantum and GRASS – youtube vid

Google is pretty well know for their April Fools’ Day jokes, and today we are treated to another one, albeit on March 30th (maybe that’s a part of the joke too):

Google Maps 8bit for NES

Here is a quick screen grab of Boston…

Zooming in a little bit a couple landmarks are displayed.  I like the how MIT and the approximate location of the Google offices in Kendall Square are highlighted in Cambridge.

The best part, even StreetView is NES ready!  Sweet…

Good job Google.  This is pretty cool.  For next year, how about you flip your maps “upside down” for the day.

Get to Google Maps now and check this out before it is too late!

p.s. I love April Fools’ Day.

Anyone who knows me and works with me on a regular basis knows that I am not a developer.  I am a geographer who develops code for visualization and analysis applications that will hopefully work.  In our line of work knowing how to write and understand code is critical and on my quest to become a better programmer I am continuously searching for the next resource to add to my collection of how-to guides and programming resources.  Some of my favorite resources are spatially focused programming blogs.  Usually the bloggers are facing the same problems I am dealing with and they are using jargon that I understand.  These two factors make it much easier to follow their examples and ideas.

Here are three blogs (from people who know what they are doing) that I follow (do people still follow blogs, or is that so 2007?) and have referenced in my quest to improve my marginal programming skills(clear overuse of () in a sentence).  Check these blogs out when you get the chance:

GeoChalkboard - A number of Esri javascripting posts, which is great for me, since I am doing a lot of that type of work lately.  Also, professional courses are made available through the site.

Guerilla GIS – I’ve referenced this blog before, mostly because I like two things about it.  The variety of code examples and the GIS snark.

odoenet - A number of GIS programming examples, along with a number of tips a tricks.  For those of you new to GIS programming  should check out the two blog posts about simplifying GIS development in ArcGIS.  Not a bad read.  Also, this post is very true.

I know there are many more blogs like this out on the interwebs.  If you know of one or have a favorite spatial programming resource post it in the comments section!

Often, when performing spatial analysis, one may need to execute some type of sampling across space.  For example, one may need to sample locations across a geographically continuous surface (think soils, anything weather related, etc.).  A spatial random sample can be used to select locations without bias.  With a simple python script one can develop a spatial random sample with relative ease.  In this post I will cover a few definitions, provide a code sample, and discuss some additional points.

First, a few definitions:

Random Number: A number chosen as if by chance from some specified distribution such that selection of a large set of these numbers reproduces the underlying distribution.

Statistical Randomness: A numeric sequence is said to be statistically random when it contains no recognizable patterns or regularities; sequences such as the results of an ideal dice roll, or the digits of π exhibit statistical randomness.

Simple Random Sample: A sample in which every element in the population has an equal chance of being selected.

Second, what is a spatial random sample?

Spatial Random Sample: Locations obtained by choosing x-coordinates and y-coordinates at random (p. 58). Any points that do not intersect the landform will be dropped from the list of random points.  

Third, give me some python code to do this!

import os, random
from time import strftime

f = open("C:\\Data\\output\\spatial_random_sample.csv", 'w')

#How many points will be generated
numpoints = random.randint(0,1000)

# Create the bounding box
#set longitude values - Y values
minx = -180
maxx = 180

#set latitude values - X values
miny = -23.5
maxy = 23.5

print "Start Time:", strftime("%a, %d %b %Y %H:%M:%S")
#Print the column headers
print >>f, "ID",",","X",",","Y"
for x in range(0,numpoints):
print >>f, x,",", random.uniform(minx,maxx),",",                      random.uniform(miny,maxy)
f.close()

print "Script Complete, Hooray!", numpoints, "random points generated"
print "End Time:", strftime("%a, %d %b %Y %H:%M:%S")

This quick, dirty and very simple script does a few things. First, it creates a csv file in a local directory, and by using the ‘w’ mode the file will be created if it doesn’t exist and will be overwritten every time the code is run (so be careful).

Next, the code  selects a random number of points to be generated. In this case it will be a random integer between zero and 1,000. The user will then set the bounding box for which the points will be contained by. If using ArcPy and ArcGIS the user could easily set the bounding box to that of a particular layer. In this example, it is simply 180,-180 and the approximate Tropic of Cancer and Tropic of Capricorn.

The next block of code will generate the random number of points in the specified ranges and print them to a csv file.  The output is fairly straight forward: three columns, an ID field and X and Y. The user can open the file in OpenOffice as they could any other csv file.

Well, that’s great.  With this data the user can easily visualize it in Quantum using the Add Delimited Text Layer tool from the Layer menu. Since the output was formatted with X and Y fields the tool will populate itself:

Once the user clicks OK the points will be added to the map.  From there the user can export the data to any number of formats and perform their analysis.

As you can see it is pretty easy to generate random points with the script.  In fact, ArcMap and Quantum have tools that will do this, but both run much slower than just creating a simple spatial random sample as demonstrated here, as they have many more options than this simple script.  Also, the Arc version will only work if the user has ArcEditor or the spatial analyst extension.  The folks at SpatialEcology also have a tool that will do this within ArcMap as well, and I am sure there are other tools out there as well.

But before we wrap this up, here are a couple notes:

• This is a simple example, and not intended to be an “end-all, be-all example”.
• Python generates psuedo-random values
• The points that are generated have an equal chance of being created, meaning that whatever is being sampled with those coordinates has an equal chance of being selected as well.
• The script presented here does not check against any boundaries, only a bounding box.
• The above code can easily be extended to work within ArcPy and ArcGIS.  I can post the code later on if there is interest.

There once was a well know GIS blog post that compared geographic information systems to word processors.  No matter what you think about the post we will always need people who are skilled at “writing” and have something to “write” about.

As I have said before, and will say again, if you are using GIS technologies you should have a grasp on the fundamentals.  You wouldn’t write a paper or a report without a grasp on the basics of the topic or without a knowledge of writing in general.  So, to improve the world’s GIS grammar (or at least my own), I will be posting a number of spatial analysis related topics over the course of the next few months.  Here are a few of the topics I will cover:

• Data classification schemes
• Understanding spatial random samples
• Topology, from a spatial point of view
• The basics of projections
• Avoiding false accuracy
• Using root mean square
• Geary’s c and Moran’s I
• The First Law of Geography
• Spatial autocorrelation
• and many more…

I’ll use a variety of software, data, and problems to explain these topics, in order to expose the reader to the broad language of GIS.

Two quick articles for your reading pleasure:

OSM Movement Gaining Steam Due to Google’s Fees – NY Times

Esri Throwing more Support Behind OSM – Directions Magazine

Now, do your part and start contributing.  If everyone who reads this post takes an half hour to improve OSM in their home town the world will be a better (mapped) place.

What are you waiting for! Go map!