Sunday, March 24, 2013

Hands Off Data Collecting With Huey, Dewey and Louie

A few weeks ago I posted a blog discussing how my organization is testing the newest release of ArcGIS for Windows Mobile 10.1.1.  I've started to use the combination of the ArcGIS mobile software and my Trimble Juno to collect walking trail data on a fairly regular basis.  I call it 'development'.  My wife calls it 'playing around.'

Anyway there are three occupants of my house that think my only job in life is to take them outside and entertain them.  They shall rename nameless, but they each have four legs and a tail.  Whenever they see me putting on a jacket or lacing up my boots or tossing things into a backpack they go nuts jumping around, barking and causing general mayhem.  "It's walkie time!"

Now, I don't mind having them along on a walk or hike.  They are generally well behaved.  Unless they see a cat.  Or a deer, squirrel, rabbit, duck, goose, bird, grasshopper, snail, ant or, heaven forbid, another dog.  Other than that they're fine.  Great company.

But managing three dogs and a data collector is all but impossible unless you're an octopus.  I was in a quandary; how do I take these three amigos along on a data collection hike and get everything done?  Well today I had one of those 'duh' moments, as in 'duh, why didn't I think of this before?'  Just launch the collection job on the Juno, toss it into the zippered compartment in the lid of the rucksack (where it sits up high and the internal GPS receiver should have a fairly good 'view' of the sky), hitch up the dogs and start walking!

Launch the data collection job on the Juno and go!

When I started today's walk I wasn't 100% sure how this would work out.  I've carried other GPS receivers in pockets on the shoulder straps of rucksacks before, but I wasn't collecting data with those units.  As I walked the trail my mind was working, thinking up all sorts of configurations and contraptions I might use to improve signal reception.  By the end of the walk I had myself convinced that what I needed was some sort of pole mounted external antenna.  I had it all figured out in my mind - about 4' of quarter inch PVC with a metal plate epoxied to one end and a magnetic patch antenna (which Trimble makes for the Juno) stuck to that, with the antenna cord running down the tubing and connecting to the Juno inside my rucksack.  I could lash the PVC tubing to the side of the pack using the compression straps.  Absolutely, perfectly Rube Goldberg-esque!  Why it was so clever I'd be the envy of all the neighborhood GPS data collecting kids!

Rides nice and high in the pack where signal reception is pretty good!

At the end of the hike I pulled the Juno out of the bag and realized I didn't need all the fancy gadettry I was dreaming up.  The Juno did just fine collecting data while sitting snugly inside the rucksack lid.  It's not a perfect setup by any means; the GPS signal still has to penetrate the bag material and my big fat noggin' blocks a lot of the signals, and it's impossible to stop collecting streaming GPS data to collect point data.  But for hikes like these where I'm just after trail alignments it works fine.  The Juno isn't a survey-grade instrument anyway so some GPS track shift is to be expected, particularly under heavy tree canopy.

"Hey, we really like this data collection thing!"

So how did we do?  Not bad.  I need to clean the data up somewhat and I know that in the future if I'm set up for GPS data post processing I'll get better accuracies, but for now it'll work.  As was said in that classic movie about chronic over-achievement, "That'll do pig."


Sunday, March 17, 2013

My Data Is More Accurate Because It Got Here First

Earlier this month Eric Gagstatter wrote a great little article for Geospatial Solutions Monthly titled "Nightmare on GIS Street: Accuracy, Datums, and Geospatial Data".  Anybody who's worked in the GIS field for more than a week has experienced the kind of issues Eric discusses.  Simply put, it is a rare event when data pulled from multiple sources fits together with any semblance of accuracy or precision.

For a small scale project (let's say 1:20,000 or smaller) data fit is less important - at those smaller scales 'eyeball close' is often good enough.  The problem we face is that with modern GIS software the user is not stuck with a fixed scale like they were when everything was based on paper maps.  We live in the era of Google Earth, the era of high resolution satellite imagery, where everybody expects to be able to read the address number on their mailbox from space.  This new found ability to zoom to any scale with just the scroll of a mouse wheel has highlighted a problem that the general public and, to be frank, many Geospatial and civil engineering professionals, were not aware of: the data doesn't fit.

Eric highlights the most important factor impacting this issue - the emergence of high precision GPS-based field data.  In the past 10 years or so GPS data, that data collected by survey grade or SBAS* augmented GPS units, has dramatically exposed the errors embedded in decades of historical geospatial data.

It's not that this old data was collected badly - most of it was collected to established standards using the best resources and techniques available at the time.  In the old days it was paper maps, scaled aerial photos, compass headings, pace counts (or odometer readings for really long distances) and field notebooks.  Mapping grade accuracy was the accepted norm.  When you were using 1:24,000 USGS topo sheets as your project base an error of +/- 24 meters (the approximate National Map Accuracy Standard for those map sheets) was good enough.  Formal surveys were expensive and time consuming, and only done if there was a strong business justification - usually to establish legal boundary definitions, accurately map out small project areas, or precisely position critical features.

Today a Geospatial professional can collect data with handheld GPS units that easily achieves accuracies of +/- 15 feet with just SBAS augmentation, and centimeter level accuracies with survey-grade RTK (real time kinematic) equipment.  Accuracy has improved by several orders of magnitude and the cost of acquiring that data had dropped dramatically.

While Eric focuses on the issues of datums and datum transformations, my experience is a little different.  I work at a major airport that has terabytes of historical CAD data and a warehouse full of historical project plans on paper, mylar or linen that go back to the early 1940s.  Virtually all of this data is referenced to a local grid system that was first established as a construction grid back in 1948.  At the time this grid was established it was never formally defined in reference to the local State Plane coordinate system.  In fact, the surveyors who laid it out committed the cardinal sin of not establishing a central meridian that is oriented to true north.  The entire grid is rotated a few degrees off of true north and that angle of rotation was never defined when the grid was established.  For years this was not a problem.  The airport was happy to exist as an 'island', floating on the face of the earth within its own little grid system.  However, when the airport started to expand dramatically in the 1960s the engineers realized they needed to start tying into properly defined coordinate systems like State Plane.  USGS and USC&GS survey control was extended onto the airport and several monuments were defined in both the local grid system and State Plane.  This allowed project engineers and surveyors to 'extend' State Plane control onto their project sites if required, but all design and construction work was continued in the local grid system.  To this point all design work was done using old manual drafting methods, so the levels of error inherent in these processes were acceptable for the time.

In the 1980s CAD (computer aided design and drafting) systems started to be used on more and more projects at the airport. Since our local grid is a simple x,y grid based on distances in feet measured from an origin point it was easy to lay out in CAD.  No need to worry about that pesky rotation.  Or, for that matter, grid-to-ground mismatches over long distances (like say on a 9,000' runway).  But very soon some serious folks with serious money, like the Federal government, began asking for airport data in a 'real' coordinate system like State Plane.  A number of attempts were made to try to define the local grid as a true spatial coordinate system (with a tie to a known coordinate system, an origin point and a rotation and scale factor) but with no success.  As a result some very sloppy work-arounds were developed, most using a 'local fit' method  - an engineer or CAD technician would snap local project data from one coordinate system to known good data in the other coordinate system; building corners, grid tics, manholes, whatever they could find.  The problem was that a lot of 'known good' data turned out to be not so good.  Errors propagated and started to become uncontrollable.  The engineering staff worked around this by referencing in local project data (for example, a new taxiway segment) against a small subset of the overall CAD basemap for the airport.  This method tended to keep the the coordinate system shift error within acceptable limits for the small project area, but when the data was added to the larger CAD basemap grid shift errors of up to 15' were common.

When my Geospatial group came on board in 2007 the coordinate system transformation issue quickly became one of our biggest headaches.  We were faced with creating an enterprise geospatial database from scratch using this legacy CAD data.  We desperately needed a proper spatial definition for this local grid system, something that would work in both our CAD and GIS systems.  Our engineering staff was happy to dump the issue in our lap.  In fact, when I interviewed for the job one of the senior engineers told me that if I was hired the first thing he wanted me to do was to "fix this damned State Plane thing."

As we started talking with the engineering staff about the problem it became apparent they all had an institutional distrust of State Plane, or any spatial coordinate system for that matter.  They placed the entire blame for the data fit issues on 'inaccuracies' in the State Plane system - inaccuracies they couldn't articulate.  In their minds all data prepared in their local grid system was golden.  After all, the local grid system was known.  It was proven.  It was simple.  They had built an entire world-class airport on it.  This goofy State Plane thing just got everybody confused and besides, when they did move their CAD data to State Plane it 'got shifted all around' and didn't work anymore.  It might fit OK at one corner of the airport, but didn't fit too well at the other.

We eventually got the grid transformation issue solved.  We attacked it from several directions and ended up with a very accurate local grid system projection file for use in both AutoCAD and ArcGIS, and a best-fit definition for use in Blue Marble (for bulk coordinate point conversions).  All of these definitions are based on the same survey data so errors are consistent and controllable from system to system.  We can hold transformation errors to about 0.2' across the airport property.  And yet our engineering staff still retained a latent distrust of State Plane-based data.  The old institutional bias remained.  The perception that ran deep is that the old 'known' CAD data in the local coordinate system is somehow better, more accurate, than any newly collected GIS data.  There is a natural distrust of geospatial data; few civil engineers understand what geospatial data is, how it differs from CAD data and how geospatial data can be incorporated into planning and design projects.  If the data file doesn't have a .dwg at the end of it they don't like it.

We decided to approach the perception issue from two directions.  The first was a current high resolution, high accuracy orthophoto of the airport.  Using our newly developed projection file we were able to reproject the aerial from State Plane to the local grid system for use in AutoCAD.  For the first time ever the engineers and CAD staff had a single continuous coverage aerial image in their grid system that could be used as a base for project planning and drawing development.  Next, we acquired RTK-based data collectors that are capable of centimeter level accuracy.  We launched on an aggressive project to collect photo identifiable data - manholes, paint markings, slab joints, airfield lights - and provide the data in both grid systems as a tool to check current and historical data against.  From this we created a 'trusted' CAD file,  one the engineering group verified using their own sources.  Ever so slowly some of the doubters started to come around.  Once they started matching their legacy data against these new sources and saw the problems for themselves they began to do more aggressive data checks and not take CAD data, old or new, at face value.

Yet we continued to have perception problems.  The old-line engineering staff retained a deeply embedded distrust of GIS data in State Plane and our insistence that all legacy data to be closely checked and adjusted if necessary.  Their reasoning actually sounded pretty good - "We spent decades building a world class airport with this CAD data and it all came together.  How can the data be wrong?"

Our GIS group didn't really have a good response until some of the long time CAD staff complained that "it's impossible to get as-builts around here."  Our antennae went up and we started to do some digging on the issue.  Very quickly the problem revealed itself.  Our engineering staff rarely received true as-builts from the contractors that do construction on the airport.  The as-built delivery requirement is written into most contracts but is rarely enforced.  Contractors would regularly walk away from the as-built requirement and eat the contract penalty because they were too busy on other airport projects or the cost of developing the as-builts exceeded the monetary penalty.  If a contractor did deliver what they labeled as 'as-built' drawings they were seldom, if ever, checked for accuracy and completeness by the project manager.  The data was accepted at face value and often recycled for use on the next project.  Errors in spatial accuracy or attributes (pipe sizes, slab thicknesses, etc.) were unknowingly propagated from project to project as the project planners and designers used the same inaccurate data over and over again.  Down the line some errors became so glaringly obvious (like a stormwater line flowing uphill) that the engineering staff would hire engineering firms to go to the field and conduct existing condition surveys.  It was not unusual for the airport to hire the same firm that originally delivered the bad data to go back out and field verify what they should have originally delivered years before in the project as-builts!

But this only addresses half of the question.  The fact remains that this airport got built, and got built pretty darned well.  Was it all built on sloppy CAD data and it's just a happy accident that everything fits?  Well, once we understood the as-built issue the rest of the story fell into place.  The engineering staff at this airport only does planning and initial design.  The final design work and construction drawings are done by contracted engineering firms.  Construction drawings are based on a number of sources - initial design, existing condition surveys and final design plans.  Construction drawings are what the project engineers and tradesmen take to the field to actually build against  These are the drawings that get marked up as modifications are done in the field and it's these drawings that should be used to generate the as-builts.  These engineering firms do a very good job of making sure everything fits within the designated project space, and any ties to existing systems - utility lines, roadways, buildings, etc. - are adjusted for in the final design or in the field.  But we are back to the old as-built issue.  Much of what was actually constructed in the field never makes it back into the airport's master CAD drawing.

So the reality is that the airport got built, but the airport doesn't have a complete and accurate record of what got built.

But I do get the sense that we are over the hump.  In the last two years we've noticed an improvement in the consistency of the spatial accuracy of the CAD data being delivered.  We still find a good number of attribute data issues (stormwater manholes labeled as sewer manholes, that sort of thing), but as far as spatial accuracy things seem to be greatly improved.  I put it down to our engineering staff's increased use of known good data as a quality control check, increased emphasis on as-built delivery, a willingness to let us be part of the quality control check process, increased dialog between the CAD and GIS analysts and an increased dependence on our RTK data collectors to do quick field verification.  In fact, our engineering staff is now the #1 hands-on  user of our RTK systems.  The GIS group also has tight relationships with many of the major construction contractors doing work at the airport and we provide the coordinate system definition files and verified base data for use in project planning.  We also offer ourselves up as the data conversion experts and will help contractors verify that their data has been properly moved from one grid system to the other.  Over time our insistence on spatial accuracy has 'leaked into' the engineering business processes and workflows here at the airport.

We've shifted the paradigm just a bit and the momentum is in our favor.  Geospatial engineering 1, bad data 0.  That's the way it should be.


*SBAS = Space Based Augmentation System.  There are several SBAS systems in use around the world.  These are satellites in geosynchronous orbit that transmit correction data for the US GPS satellite constellation.  The Wide Area Augmentation System (WAAS) is a set of satellites transmitting correction data over the US and the eastern Pacific and is maintained by the FAA and DOT.  If your GPS unit can receive and use these signals they will roughly double the accuracy of the position fix your unit provides.  You can read more about WAAS and SBAS on Wikipedia.

Wednesday, March 13, 2013

Busting Brush with ArcGIS for Windows Mobile

Huh?  What?

OK, at work we are testing a new software package - ArcGIS for Windows Mobile.  The name is a mashup of two software package names - ArcGIS (ESRI) and Windows Mobile (Microsoft).  Yes, it is a cumbersome name.  Really, really cumbersome.

It is nothing more than a lite version of ESRI's ArcGIS software designed specifically to run on the Windows Mobile OS.  Never heard of Windows Mobile?  Don't worry, you haven't missed much.  It's an operating system that saw its widest use on mobile phones.  Notice I didn't say 'smart' phones, because Windows Mobile was (and still is) an awful operating system that made every piece of hardware it touched dumber.  Same for its users.  I'm sure when Steve Jobs was yelling at his software engineers during the early stages of Apple's iOS development he forced them to use Windows Mobile phones so they clearly understood what  iOS would not end up looking like.

Microsoft has moved on and introduced their new Windows Phone OS that is based on the Windows 8 platform.  However, their Windows Mobile OS hangs on in a few interesting places.  It's used a lot in 'embedded' applications, computers running inside of other things that don't look like computers.  For example, the Microsoft Sync system that controls just about everything in my new Ford F-150 is a version of Windows Mobile.

Another place Windows Mobile has achieved a lot of market penetration is in the surveying and GPS-based data collection market.  These are highly customized hardware systems that are more than mobile phones but less that full-fledged computers, robust devices dedicated to a narrow set of field data collection tasks.  Virtually all manufacturers of surveying and GPS-based field data collection systems use Windows Mobile - there's really nothing else available today that meets the need.

Two of the devices in this picture run on Windows Mobile.
The rest are easy to use.

Because of this ESRI still develops a lot of software to run on the Windows Mobile OS (now up to version 6.5 and renamed Windows Embedded Handheld).  ArcPad, ESRI's flagship field data collection package, has been running on the Windows Mobile platform for almost a decade.  A few years back ESRI released a version of its server software (called, naturally, ArcGIS for Server) that allowed the user to develop GIS applications that run on Windows Mobile handhelds and consume map services hosted on a local ArcGIS for Server instance.  We tested this at work but came away with the impression that the whole system required more care and feeding than we were able to provide.  In addition, our IT department was never willing to cooperate and provide a way to authenticate these mobile devices (Trimble Junos) on our organization's domain so they could 'see' our internal GIS software servers.

Instead, we've spent the last year developing mobile GIS applications to run on Apple iOS devices - iPads and iPhones - and leveraging the new hosted map service concept available through ESRI's ArcGIS Online cloud service.  The entire system works amazingly well and our users like the idea of collecting GIS data using an iPad (plus they can play Angry Birds in their down time).  But as we tested and developed in this environment we quickly bumped up against another roadblock - again, our IT department.  They repeatedly refuse to approve the purchase of iPads for the user base.  No real explanation - they say no just because they can.  We were stuck and desperate for an alternative.

This all changed in late February with the release of ArcGIS for Windows Mobile 10.1.1.  The #1 change with this new version is the ability of the software to connect to an ArcGIS Online subscription account and use a hosted (cloud) feature service as a data layer.  We no longer are forced to use our local instance of ArcGIS for Server as the data source.  Since the data is hosted in the cloud all we need is a wi-fi connection and an ArcGIS Online subscription account to get to our data.  The need to have our mobile devices authenticate on our organization's network is gone.  Our dependency on our local IT department is severed.

The other big benefit that ArcGIS for Windows Mobile brings is the ability to do disconnected editing when there is no wi-fi signal available.  This was always a concern with the iOS devices, which require an 'always on' internet connection when being used to collect data.  ArcGIS for Windows Mobile works differently in that the application places a copy of the map layer's database directly on the mobile device when it is first downloaded from ArcGIS Online.  In the field a wi-fi connection to the internet is not necessary - all the newly collected data and edits are stored on the mobile device.  When the user get back under wi-fi coverage they can do a synchronization of this new data with the master database stored in the cloud on ArcGIS Online.  The new data is pushed up to the master database via a wi-fi connection to the internet - any wi-fi connection to the internet; in the office, in Starbucks, in McDonalds, wherever they can get a signal.  Simple, slick and robust.

But why Trimble Junos?  While our IT department balks at the purchase of iPads, they have no problem with us purchasing Junos even though the current generation of Junos cost about twice as much as an iPad!.  IT views them as dedicated field survey devices and allows us to buy as many as our budget can support.  Over the past two years we've purchased seven of them so we have plenty of hardware available to put into the hands of our users.

The Trimble fairy barfed on my desk

So like any good Geospatial weenie I figured I needed to test my applications before unleashing them on the unsuspecting public.  To acquaint myself with the workflows embedded in ArcGIS for Windows Mobile I set up a project to collect data during one of my favorite activities - hiking.  Then it was time to take ArcGIS for Windows Mobile to the wilds of suburban Atlanta for the ultimate test: can a befuddled 56 year old make sense of this mobile thing and actually collect useful data?

Everything I need to survive - water, food, first aid kit and
ArcGIS for Windows Mobile on a Trimble Juno

So how did everything work?  Pretty darned good.  In fact, better than I expected.  The simplified workflows in the ArcGIS for Windows Mobile interface make collecting data almost foolproof.  I only had a chance to capture about two miles of trail information and some points of interest, but it was enough to convince me this mobile GIS interface will work just fine for most of our user base.  Of course it's an imperfect world, and so is this application.  I'd love the ability to collect photos while remaining in the GPS data streaming mode, and being the GPS geek that I am I'd like a better GPS performance interface (similar to what you get with ArcPad or Trimble's TerraSync), but I also understand this package is designed for simplified data collection by non-GIS personnel, so I can live with the lack of GPS performance data.

Yellow SO clashes with my woods gear!

Of course all this simplification also serves as a straightjacket.  What you give up with the ArcGIS for Windows Mobile interface is the ability to make on-the-fly changes to your project - add new data types, change symbology, adjust GPS performance parameters, do complex searches, buffers, etc.  It's a trade-off;  reduced complexity =  fewer options.  It's a trade-off my organization can live with.

ArcGIS for Windows Mobile up close and personal.
A simple interface that works well on devices with small screens.

The REAL elephant in the room is the overall cost of this capability.  ArcGIS for Windows Mobile is an enterprise level solution for enterprise level projects.  As configured this project relies on an ArcGIS Online subscription account as the data hosting platform and ArcGIS Mobile deployment licenses tied to a very expensive ArcGIS enterprise license.  This equates to about $55,000 in licensing costs (toss in another $1,200 or so for the Trimble Juno).

Obviously this is not for the little guy.  But it should be, and it can be!  Let's say you are a Geospatial geek like me.  Right now, today, if you participate in the ArcGIS for Home Use Program you get one free user license for  ArcGIS for Windows Mobile.  The software package includes a toolset that allows you to stage all your data on your local computer instead of on ArcGIS for Server or in the ArcGIS Online cloud.  You don't get to wirelessly update your data - you have to connect the device to your computer via USB to do the synchronization process - but the rest of the workflows are the same.  A great (and cheap) way to test this capability for yourself.  You just have to go find a compatible Windows Mobile device (check eBay, there's plenty for sale out there).

Looking into the future I see ESRI opening the ArcGIS Online subscription program up to market segments that don't need and can't afford to buy into the service at the current enterprise-like levels.  It's almost an inevitability.  There's a lot of emerging competition in the cloud GIS services arena, and companies leveraging some of the better Open Source GIS tools will start to provide low cost cloud hosting services in direct competition with ESRI.  Of course ESRI has the market share and clout, but the 'cloud' is a huge space, there's a huge potential market, and ESRI can't control it all.  Price competition will soon have its inevitable impact.  Plus, ESRI is pushing 'the cloud' like it's the Second Coming and at some point they will realize their service availability will have to mesh with their message.  As they move more and more capability to the cloud ESRI will have to start offering low cost services for the little guy.

Maybe not this year, maybe not next year, but soon.

In the meantime use what's available and get moving on Mobile!


Saturday, March 2, 2013


I was testing the camera on my new iPhone this morning and decided to take some gratuitous cheesecake shots of one of my newest acquisitions:

K&E Pocket Transit

It's essentially a new-old stock (NOS) pocket transit made by K&E sometime in the early 60's (judging by the dates in the instructional brochure).

Unlike Ainsworth or Brunton, K&E never serial numbered their pocket transits so there's no way to firmly establish manufacturing dates.  It may be possible to determine an age range based on features, but the one source that discussed K&E pocket transit manufacturing date ranges, the outstanding William J. Hudson pages on the history of the pocket transit, have gone off line.  Let's hope this great resource will soon be back up and available.

This particular pocket transit pretty much follows the Ainsworth model feature for feature.  It uses an un-dampened needle setup with luminous points and offers the standard long level on the clinometer and bubble level for general leveling.  It uses the 360 degree 'reversed' compass ring that is adjustable for declination. It's immaculate inside and outside, and I doubt it had been handled much before I acquired it.

How well does it compare to an Ainsworth or Brunton product of the same age?  Pretty damned well!  K&E was a premiere American manufacturer of surveying and engineering equipment from the late 1800s right up into the early 1980s, and they knew how to make a quality product.  This pocket transit is equal to, and in some cases surpasses, the quality put out by either Ainsworth or Brunton.

All-in-all an excellent example of an American pocket transit.