Earlier this month Eric Gagstatter wrote a great little article for Geospatial Solutions Monthly titled "Nightmare on GIS Street: Accuracy, Datums, and Geospatial Data". Anybody who's worked in the GIS field for more than a week has experienced the kind of issues Eric discusses. Simply put, it is a rare event when data pulled from multiple sources fits together with any semblance of accuracy or precision.
For a small scale project (let's say 1:20,000 or smaller) data fit is less important - at those smaller scales 'eyeball close' is often good enough. The problem we face is that with modern GIS software the user is not stuck with a fixed scale like they were when everything was based on paper maps. We live in the era of Google Earth, the era of high resolution satellite imagery, where everybody expects to be able to read the address number on their mailbox from space. This new found ability to zoom to any scale with just the scroll of a mouse wheel has highlighted a problem that the general public and, to be frank, many Geospatial and civil engineering professionals, were not aware of: the data doesn't fit.
Eric highlights the most important factor impacting this issue - the emergence of high precision GPS-based field data. In the past 10 years or so GPS data, that data collected by survey grade or SBAS* augmented GPS units, has dramatically exposed the errors embedded in decades of historical geospatial data.
It's not that this old data was collected badly - most of it was collected to established standards using the best resources and techniques available at the time. In the old days it was paper maps, scaled aerial photos, compass headings, pace counts (or odometer readings for really long distances) and field notebooks. Mapping grade accuracy was the accepted norm. When you were using 1:24,000 USGS topo sheets as your project base an error of +/- 24 meters (the approximate National Map Accuracy Standard for those map sheets) was good enough. Formal surveys were expensive and time consuming, and only done if there was a strong business justification - usually to establish legal boundary definitions, accurately map out small project areas, or precisely position critical features.
Today a Geospatial professional can collect data with handheld GPS units that easily achieves accuracies of +/- 15 feet with just SBAS augmentation, and centimeter level accuracies with survey-grade RTK (real time kinematic) equipment. Accuracy has improved by several orders of magnitude and the cost of acquiring that data had dropped dramatically.
While Eric focuses on the issues of datums and datum transformations, my experience is a little different. I work at a major airport that has terabytes of historical CAD data and a warehouse full of historical project plans on paper, mylar or linen that go back to the early 1940s. Virtually all of this data is referenced to a local grid system that was first established as a construction grid back in 1948. At the time this grid was established it was never formally defined in reference to the local State Plane coordinate system. In fact, the surveyors who laid it out committed the cardinal sin of not establishing a central meridian that is oriented to true north. The entire grid is rotated a few degrees off of true north and that angle of rotation was never defined when the grid was established. For years this was not a problem. The airport was happy to exist as an 'island', floating on the face of the earth within its own little grid system. However, when the airport started to expand dramatically in the 1960s the engineers realized they needed to start tying into properly defined coordinate systems like State Plane. USGS and USC&GS survey control was extended onto the airport and several monuments were defined in both the local grid system and State Plane. This allowed project engineers and surveyors to 'extend' State Plane control onto their project sites if required, but all design and construction work was continued in the local grid system. To this point all design work was done using old manual drafting methods, so the levels of error inherent in these processes were acceptable for the time.
In the 1980s CAD (computer aided design and drafting) systems started to be used on more and more projects at the airport. Since our local grid is a simple x,y grid based on distances in feet measured from an origin point it was easy to lay out in CAD. No need to worry about that pesky rotation. Or, for that matter, grid-to-ground mismatches over long distances (like say on a 9,000' runway). But very soon some serious folks with serious money, like the Federal government, began asking for airport data in a 'real' coordinate system like State Plane. A number of attempts were made to try to define the local grid as a true spatial coordinate system (with a tie to a known coordinate system, an origin point and a rotation and scale factor) but with no success. As a result some very sloppy work-arounds were developed, most using a 'local fit' method - an engineer or CAD technician would snap local project data from one coordinate system to known good data in the other coordinate system; building corners, grid tics, manholes, whatever they could find. The problem was that a lot of 'known good' data turned out to be not so good. Errors propagated and started to become uncontrollable. The engineering staff worked around this by referencing in local project data (for example, a new taxiway segment) against a small subset of the overall CAD basemap for the airport. This method tended to keep the the coordinate system shift error within acceptable limits for the small project area, but when the data was added to the larger CAD basemap grid shift errors of up to 15' were common.
When my Geospatial group came on board in 2007 the coordinate system transformation issue quickly became one of our biggest headaches. We were faced with creating an enterprise geospatial database from scratch using this legacy CAD data. We desperately needed a proper spatial definition for this local grid system, something that would work in both our CAD and GIS systems. Our engineering staff was happy to dump the issue in our lap. In fact, when I interviewed for the job one of the senior engineers told me that if I was hired the first thing he wanted me to do was to "fix this damned State Plane thing."
As we started talking with the engineering staff about the problem it became apparent they all had an institutional distrust of State Plane, or any spatial coordinate system for that matter. They placed the entire blame for the data fit issues on 'inaccuracies' in the State Plane system - inaccuracies they couldn't articulate. In their minds all data prepared in their local grid system was golden. After all, the local grid system was known. It was proven. It was simple. They had built an entire world-class airport on it. This goofy State Plane thing just got everybody confused and besides, when they did move their CAD data to State Plane it 'got shifted all around' and didn't work anymore. It might fit OK at one corner of the airport, but didn't fit too well at the other.
We eventually got the grid transformation issue solved. We attacked it from several directions and ended up with a very accurate local grid system projection file for use in both AutoCAD and ArcGIS, and a best-fit definition for use in Blue Marble (for bulk coordinate point conversions). All of these definitions are based on the same survey data so errors are consistent and controllable from system to system. We can hold transformation errors to about 0.2' across the airport property. And yet our engineering staff still retained a latent distrust of State Plane-based data. The old institutional bias remained. The perception that ran deep is that the old 'known' CAD data in the local coordinate system is somehow better, more accurate, than any newly collected GIS data. There is a natural distrust of geospatial data; few civil engineers understand what geospatial data is, how it differs from CAD data and how geospatial data can be incorporated into planning and design projects. If the data file doesn't have a .dwg at the end of it they don't like it.
We decided to approach the perception issue from two directions. The first was a current high resolution, high accuracy orthophoto of the airport. Using our newly developed projection file we were able to reproject the aerial from State Plane to the local grid system for use in AutoCAD. For the first time ever the engineers and CAD staff had a single continuous coverage aerial image in their grid system that could be used as a base for project planning and drawing development. Next, we acquired RTK-based data collectors that are capable of centimeter level accuracy. We launched on an aggressive project to collect photo identifiable data - manholes, paint markings, slab joints, airfield lights - and provide the data in both grid systems as a tool to check current and historical data against. From this we created a 'trusted' CAD file, one the engineering group verified using their own sources. Ever so slowly some of the doubters started to come around. Once they started matching their legacy data against these new sources and saw the problems for themselves they began to do more aggressive data checks and not take CAD data, old or new, at face value.
Yet we continued to have perception problems. The old-line engineering staff retained a deeply embedded distrust of GIS data in State Plane and our insistence that all legacy data to be closely checked and adjusted if necessary. Their reasoning actually sounded pretty good - "We spent decades building a world class airport with this CAD data and it all came together. How can the data be wrong?"
Our GIS group didn't really have a good response until some of the long time CAD staff complained that "it's impossible to get as-builts around here." Our antennae went up and we started to do some digging on the issue. Very quickly the problem revealed itself. Our engineering staff rarely received true as-builts from the contractors that do construction on the airport. The as-built delivery requirement is written into most contracts but is rarely enforced. Contractors would regularly walk away from the as-built requirement and eat the contract penalty because they were too busy on other airport projects or the cost of developing the as-builts exceeded the monetary penalty. If a contractor did deliver what they labeled as 'as-built' drawings they were seldom, if ever, checked for accuracy and completeness by the project manager. The data was accepted at face value and often recycled for use on the next project. Errors in spatial accuracy or attributes (pipe sizes, slab thicknesses, etc.) were unknowingly propagated from project to project as the project planners and designers used the same inaccurate data over and over again. Down the line some errors became so glaringly obvious (like a stormwater line flowing uphill) that the engineering staff would hire engineering firms to go to the field and conduct existing condition surveys. It was not unusual for the airport to hire the same firm that originally delivered the bad data to go back out and field verify what they should have originally delivered years before in the project as-builts!
But this only addresses half of the question. The fact remains that this airport got built, and got built pretty darned well. Was it all built on sloppy CAD data and it's just a happy accident that everything fits? Well, once we understood the as-built issue the rest of the story fell into place. The engineering staff at this airport only does planning and initial design. The final design work and construction drawings are done by contracted engineering firms. Construction drawings are based on a number of sources - initial design, existing condition surveys and final design plans. Construction drawings are what the project engineers and tradesmen take to the field to actually build against These are the drawings that get marked up as modifications are done in the field and it's these drawings that should be used to generate the as-builts. These engineering firms do a very good job of making sure everything fits within the designated project space, and any ties to existing systems - utility lines, roadways, buildings, etc. - are adjusted for in the final design or in the field. But we are back to the old as-built issue. Much of what was actually constructed in the field never makes it back into the airport's master CAD drawing.
So the reality is that the airport got built, but the airport doesn't have a complete and accurate record of what got built.
But I do get the sense that we are over the hump. In the last two years we've noticed an improvement in the consistency of the spatial accuracy of the CAD data being delivered. We still find a good number of attribute data issues (stormwater manholes labeled as sewer manholes, that sort of thing), but as far as spatial accuracy things seem to be greatly improved. I put it down to our engineering staff's increased use of known good data as a quality control check, increased emphasis on as-built delivery, a willingness to let us be part of the quality control check process, increased dialog between the CAD and GIS analysts and an increased dependence on our RTK data collectors to do quick field verification. In fact, our engineering staff is now the #1 hands-on user of our RTK systems. The GIS group also has tight relationships with many of the major construction contractors doing work at the airport and we provide the coordinate system definition files and verified base data for use in project planning. We also offer ourselves up as the data conversion experts and will help contractors verify that their data has been properly moved from one grid system to the other. Over time our insistence on spatial accuracy has 'leaked into' the engineering business processes and workflows here at the airport.
We've shifted the paradigm just a bit and the momentum is in our favor. Geospatial engineering 1, bad data 0. That's the way it should be.
Brian
*SBAS = Space Based Augmentation System. There are several SBAS systems in use around the world. These are satellites in geosynchronous orbit that transmit correction data for the US GPS satellite constellation. The Wide Area Augmentation System (WAAS) is a set of satellites transmitting correction data over the US and the eastern Pacific and is maintained by the FAA and DOT. If your GPS unit can receive and use these signals they will roughly double the accuracy of the position fix your unit provides. You can read more about WAAS and SBAS on Wikipedia.
Showing posts with label gis. Show all posts
Showing posts with label gis. Show all posts
Sunday, March 17, 2013
Sunday, November 13, 2011
Brain Droppings
Naaah, I haven't been ignoring this blog. I've just been busy. It's just one of those times when one's personal and professional life get a little crowded and you get distracted.
So anyway, what have I been working on that is related to maps, mapping, topography, GIS, et. al.?
Actually, quite a bit, although the activity is more of a low murmur rather than a series of 'ta-da!' moments. Let's begin...
ArcGIS Online. Aaah, the software I hate to love. I can't discuss too much detail right now because we are involved in the beta test program (and that imposes some confidentiality rules on us), but it is starting to look like the new ESRI ArcGIS Online for Organizations program is shaping up to be something of a game changer for enterprise web mapping, particularly for small and medium organizations that can't afford or don't need heavy iron IT backbones to get their jobs done. The ArcGIS Online program is part of ESRI's cloud GIS strategy and I for one like the direction it's going. This program will allow many - perhaps a majority - of organizations that develop and deploy GIS web maps to leave their in-house IT departments behind. It is not a complete solution, but it takes the field a lot further down the path of IT independence than we've ever seen. Anything that helps destroy the perception that GIS is just another IT discipline is great.
By the way, you do have your ArcGIS Online accounts set up, don't you? And you are using the free ArcGIS Online web mapping tools right?
Mobile Mapping. This is loosely tied to the ArcGIS Online initiative. We are testing some new mobile solutions that utilize the ArcGIS applications developed for Apple's iOS (iPad and iPhone) and Google's Android OS (smartphones and tablets). While you will still need ArcGIS Server (but hey, that can reside in the cloud now, too), these new mobile apps make it easier to create and deploy lightweight apps that include a basic level of interactive field data creation and editing. I'm sure we'll see more of this at the next ESRI User's Conference. It's slick!
To The Cloud! Lately I've been absolutely fascinated by the concept of cloud computing and I've been poking around in that realm just to see what's possible, what's not possible and what will be possible in the months or years to come. Folks, this is the future of computing. As Steve Jobs would say, it's 'the next big thing'. Surprising then that Apple is still stumbling at getting their cloud computing programs up and running. The idea of an OS-agnostic computing environment that maintains all your information in a secure data store on the internet (i.e., the cloud) and can be accessed from any computer through a simple web browser is compelling. Apple, Google and Microsoft are in a mad race to stake out their territory in the cloud, but it's clear that Google has taken an early and substantial lead. That's not surprising, given that cloud computing has been at the core of Google's business strategy been for years. That's OK because Google's competitors are in a hurry to catch up, and the inevitable struggle for our attention (and dollars) will make this an exciting wrestling match. I'm gonna' go grab a tub of popcorn and a Coke and watch the show from the cozy comfort of my iPad.
Oh, and keep your eye on the new kid on the block - Amazon.
Well, that's it for now. What's up next? Probably something to do with CAD & GIS integration methinks.
Brian
So anyway, what have I been working on that is related to maps, mapping, topography, GIS, et. al.?
Actually, quite a bit, although the activity is more of a low murmur rather than a series of 'ta-da!' moments. Let's begin...
ArcGIS Online. Aaah, the software I hate to love. I can't discuss too much detail right now because we are involved in the beta test program (and that imposes some confidentiality rules on us), but it is starting to look like the new ESRI ArcGIS Online for Organizations program is shaping up to be something of a game changer for enterprise web mapping, particularly for small and medium organizations that can't afford or don't need heavy iron IT backbones to get their jobs done. The ArcGIS Online program is part of ESRI's cloud GIS strategy and I for one like the direction it's going. This program will allow many - perhaps a majority - of organizations that develop and deploy GIS web maps to leave their in-house IT departments behind. It is not a complete solution, but it takes the field a lot further down the path of IT independence than we've ever seen. Anything that helps destroy the perception that GIS is just another IT discipline is great.
By the way, you do have your ArcGIS Online accounts set up, don't you? And you are using the free ArcGIS Online web mapping tools right?
Mobile Mapping. This is loosely tied to the ArcGIS Online initiative. We are testing some new mobile solutions that utilize the ArcGIS applications developed for Apple's iOS (iPad and iPhone) and Google's Android OS (smartphones and tablets). While you will still need ArcGIS Server (but hey, that can reside in the cloud now, too), these new mobile apps make it easier to create and deploy lightweight apps that include a basic level of interactive field data creation and editing. I'm sure we'll see more of this at the next ESRI User's Conference. It's slick!
To The Cloud! Lately I've been absolutely fascinated by the concept of cloud computing and I've been poking around in that realm just to see what's possible, what's not possible and what will be possible in the months or years to come. Folks, this is the future of computing. As Steve Jobs would say, it's 'the next big thing'. Surprising then that Apple is still stumbling at getting their cloud computing programs up and running. The idea of an OS-agnostic computing environment that maintains all your information in a secure data store on the internet (i.e., the cloud) and can be accessed from any computer through a simple web browser is compelling. Apple, Google and Microsoft are in a mad race to stake out their territory in the cloud, but it's clear that Google has taken an early and substantial lead. That's not surprising, given that cloud computing has been at the core of Google's business strategy been for years. That's OK because Google's competitors are in a hurry to catch up, and the inevitable struggle for our attention (and dollars) will make this an exciting wrestling match. I'm gonna' go grab a tub of popcorn and a Coke and watch the show from the cozy comfort of my iPad.
Oh, and keep your eye on the new kid on the block - Amazon.
Well, that's it for now. What's up next? Probably something to do with CAD & GIS integration methinks.
Brian
Saturday, September 17, 2011
The Software I Hate To Love
In the Geospatial Engineering world there is one Big Dog software developer and a pack of miniature chihuahuas snapping at its heels. The Big Dog is ESRI, developers of the ArcGIS suite of software products.
ESRI dominates the GIS (geospatial information systems) software field in the same way Microsoft dominates the computer operating system field - there are competitors but nobody even comes close to the market share that ESRI developed and has held for decades.
But unlike Microsoft, ESRI didn't get to where it is by being predatory and imposing crushing licensing agreements on its clients. ESRI got it's market share the old fashioned way - by simply being the best product in the market for the target consumer group. ArcGIS is the software product that moved the traditional discipline of topography out of the paper map and overlay era and into the computer-based, analysis driven discipline of Geospatial Engineering.
ESRI was started by Jack Dangermond, someone I refer to as a "Birkenstock wearin', Volvo drivin', granola crunchin' hippie." In the late 1960s and early 70s, building on pioneer work that had been done on early GIS concepts and development in Canada (where the discipline of GIS got its start), Dangermond created a land cover analysis program called ArcInfo and released it as a commercial product in the early 1980s.
Early versions of ArcInfo were hindered by limited computer processing, storage and graphics capability. Geospatial analysis is very much a visual discipline - you're making maps, after all. Early desktop hardware simply didn't have the capability and capacity to bring the full visual mapping experience to the user. Up through the mid 1990s only expensive Unix workstations could handle that level of processing. This all changed around 1995 when desktop computing power started increasing exponentially with each new processor design while at the same time hardware prices dropped like a brick. Almost overnight inexpensive desktop computers appeared that could easily handle the processing and graphics demands a software package like ArcInfo placed on them. I was working as a GIS program manager for the US Army when this hardware revolution hit the field and watched as in less than two years inexpensive desktop PCs caught up with and then quickly surpassed the processing power of the Unix-based Sun, Silicon Graphics and HP systems we had been relying on. What also helped was Microsoft's release of WindowsNT at about the same time. Finally we had a serious network-ready enterprise operating system running on high capacity hardware that didn't make our budget guys weep every time we said we needed to do an upgrade.
ArcInfo is the flagship product of the ESRI line and is extremely powerful software. But in the 1980s ESRI realized that not everyone needed the processing power of ArcInfo (nor could they afford the nausea-inducing cost of an ArcInfo software license). ESRI introduced a lightweight version of ArcInfo that included most of the visualization capability of the high end package but left out the heavyweight analysis and data development functionality. They named it ArcView. It was priced right - something small organizations and even individuals serious about GIS could afford (if I remember correctly the GSA schedule price for a single ArcView license ran around $600 in 2000). The vast majority of today's GIS professionals cut their teeth on ArcView.
But ESRI's real contribution to the GIS profession is the development of data types that both support complex spatial analysis and can be shared across different software platforms. It is Dangermond's vision that GIS-based mapping and analysis solutions should not be a stovepipe, but a shared resource. This drove ESRI to develop the concept of the geodatabase. A geodatabase is a collection of data in a standard relational database management system (RDBMS) like Oracle or SQL Server, but the data has very unique spatial values (location in x, y and z coordinates) assigned to it. This means that GIS software can leverage the spatial values to relate the data in a location context and other RDBMS-based software systems can easily share their information with the geodatabase. The geodatabase only needs to store GIS-unique features and can pull and do analysis against associated data in another database.
ESRI also developed a version of the geodatabase that does not require a high powered relational database management system as it's foundation. About a decade ago ESRI introduced the concept of a file-based geodatabase designed for use by small organizations or groups. The file geodatabase is a simple to create yet powerful and extremely flexible data format that brings most of the power of the relational database and complex data analysis to the desktop machine and the individual user.
But what does the future hold? ESRI realized long ago that the Internet was the map content delivery vehicle of the future. Paper maps were headed to obsolescence and what Jack Dangermond describes as the 'rich web map' would quickly become the geospatial data visualization and analysis tool of the future. He's right, but only very recently has web technology started to catch up with his vision.
For the better part of a decade it was possible to hire professional web developers to create some very nice web mapping applications built on ESRIs early web technology called ArcIMS. The problem was that those applications were difficult to develop, difficult to maintain, and required a lot of heavy weight back-end web and database server technology. Only large enterprises and governments could support the hardware, software, development and maintenance costs. ESRI's web solutions were very much limited by the immature web development technologies available at the time. It is ESRI's vision that even the average geospatial professional working for a small business or local government should be able to develop, launch and maintain high quality web maps that bring value to the organization they support. ESRI started laying the groundwork for this vision back with their ArcGIS 9 series of software releases and the development of things like ArcGIS Server and the concept of Map Services. Two years ago they released ArcGIS 10 that brought a lot of maturity to the concept of integrated and streamlined web mapping using the Microsoft Silverlight and Adobe Flex web development environments, and the launch of ArcGIS Online with its peek into the future concept of 'cloud services' for hosting GIS data, services and web maps.
At it's recent worldwide user's conference ESRI announced the pending release of ArcGIS 10.1 with better integrated and streamlined web development tools. But ESRI also announced two new developments that are generating a lot of interest. The first is the announcement that ESRI has partnered with Amazon.com to host robust, enterprise-level cloud services for GIS web mapping, data hosting and application development. The idea is that an enterprise purchases an ArcGIS Server software license, passes that license over to Amazon and Amazon stands up and maintains the necessary database and web development environment for the enterprise. This is a huge development because it can free the GIS group supporting the enterprise from the often onerous and restrictive shackles placed on it by their local IT department.
The other announcement was the pending release of the ArcGIS Online Organizational Account program. The Organizational Account program appears to be targeted as smaller enterprises and groups that don't have the money or need to purchase full-up cloud services like those offered by Amazon. Under the Organizational Account concept an organization will be able to purchase data and web hosting services from ESRI on a subscription basis. It is still a 'cloud' model, but on a smaller, more tailorable scale that should allow small organizations to enjoy most of the capabilities of a full-up ArcGIS Server implementation.
The last good thing I need to discuss is another little-known program released this year - the concept of ArcGIS for home or personal use. ESRI's software licensing fees have escalated to the point that the geospatial professional simply can't afford a copy to use to keep his or her skills sharp. I noted above that the GSA price for an ArcView license used to run about $600 - a bearable cost if you were serious about GIS. However, the cost for an ArcView license now hovers around $1,600, far too much for even the serious home user. This year ESRI announced the ArcGIS for Home Use program. Anyone can purchase a 1-year license of ArcView for $100, a very reasonable price. Not only does this $100 include on-line software training and support, but you also get a very extensive suite of add-on modules like 3D Analyst, Spatial Analyst and Geostatistical Analyst. The total value of the software you get for your $100 subscription comes to over $10,000. One hell of a deal. Of course there are restrictions attached to this deal. The intent of the home use program is just that - you can only use it at home. You can also only use it for personal development/training purposes or non-profit use. Still, like I said, it's one hell of a deal.
Now, it's not all rainbows and unicorns when it comes to ArcGIS and ESRI's position in the GIS world. All this GIS goodness is of little use unless it's leveraged in an environment with clearly defined professional standards. Nor can you allow a professional discipline to be defined by a software application or be inexorably joined to a piece of software. This is where ESRI's has failed the geospatial community, and they have failed in ways they can't even visualize from where they sit.
Here's the reality: geospatial engineering is the discipline, the term geospatial information systems - GIS - merely describes the tools geospatial professionals use to do their job. Where ESRI has failed is in using its industry position and influence to help clearly delineate the difference between the two. As a result, far too many engineering professionals view geospatial professionals as little more than button pushing software monkeys, one step up from data entry clerks.
Part of the culture Jack Dangermond has fostered and progressed through ESRI is the idea that GIS is for everyone and nobody owns it. What he is effectively saying is that GIS is the discipline; the tools and the software drive the field, not the other way around.
While community ownership is a noble goal, ESRI's dominance of the field gives lie to that very philosophy. Effectively, ESRI 'owns' GIS; it is by far the world's largest GIS software developer. It has either developed or successfully implemented most of the recognized spatial analysis processes in use today. It's data management features have driven the development of most of the spatial data standards in use today. The vast majority of geospatial professionals worldwide learned their trade using ArcGIS.
What is lacking, however, is a clear and recognized definition of just what a geospatial professional is. Dangermond is correct when he claims it's not his role to define what a geospatial professional should be - that is the job of the geospatial field and industry as a whole. But Dangermond has been the biggest catalyst in the geospatial world for the last 30 years. He and the resources he commands through ESRI have been in the best position to cajole and coerce the private sector, academia and the government to establish the roles, practices and responsibilities that define Geospatial Engineering as a formal discipline. He should have been the single biggest champion of the concept of Geospatial Engineering as a professional discipline. Instead he's been pretty much silent on the whole issue.
It is only in the last few years that the US Department of Labor developed a formal competency model for GIS (GIS, not Geospatial Engineering), and the GIS Professional Certification program is just starting to get its feet on the ground (after a disastrous grandfathering period that allowed perhaps hundreds of clearly unqualified individuals to get a GISP certificate and do damage to the reputation of the geospatial profession that may take years to overcome). Great, but all this should have happened 20 years ago.
What this means is that Geospatial Engineering is not respected as a professional discipline. I can tell you from long personal experience that geospatial professionals are looked down upon by other disciplines such as civil engineering and surveying, in large part because there are no testable and enforced standards that define us as a 'profession'. Guess what - they are right!
Many readers are probably asking themselves "Huh? What's he getting at here?" I guess I'd ask the same question myself if I didn't understand the background issues.
I've been a topographer and geospatial engineer for over 30 years. A few months back I laid out my initial arguments in a post titled In Praise of the Old Topographer. In that post I made the argument that Geospatial Engineering is just a logical continuation of the older and much respected profession of Topographer. I also outlined my argument that geospatial information systems, including ArcGIS, are merely the tools that the Geospatial Engineer uses to do his or her job.
With this post my goal was to identify one of the main culprits that is keeping Geospatial Engineering from fully maturing into a recognized profession, a profession with it's own standards, roles and responsibilities.
ArcGIS is that culprit. On the one hand we have extraordinarily capable software that is almost single handedly responsible for bringing the discipline into the computer age and is poised to bring it fully into the age of world wide web. On the other hand, ArcGIS and it's parent company ESRI are almost single handedly responsible for holding the discipline back and keeping it from taking it's rightful place as a profession on par with other engineering disciplines.
For these reasons ArcGIS is the software I hate to love.
ESRI dominates the GIS (geospatial information systems) software field in the same way Microsoft dominates the computer operating system field - there are competitors but nobody even comes close to the market share that ESRI developed and has held for decades.
But unlike Microsoft, ESRI didn't get to where it is by being predatory and imposing crushing licensing agreements on its clients. ESRI got it's market share the old fashioned way - by simply being the best product in the market for the target consumer group. ArcGIS is the software product that moved the traditional discipline of topography out of the paper map and overlay era and into the computer-based, analysis driven discipline of Geospatial Engineering.
ESRI was started by Jack Dangermond, someone I refer to as a "Birkenstock wearin', Volvo drivin', granola crunchin' hippie." In the late 1960s and early 70s, building on pioneer work that had been done on early GIS concepts and development in Canada (where the discipline of GIS got its start), Dangermond created a land cover analysis program called ArcInfo and released it as a commercial product in the early 1980s.
Early versions of ArcInfo were hindered by limited computer processing, storage and graphics capability. Geospatial analysis is very much a visual discipline - you're making maps, after all. Early desktop hardware simply didn't have the capability and capacity to bring the full visual mapping experience to the user. Up through the mid 1990s only expensive Unix workstations could handle that level of processing. This all changed around 1995 when desktop computing power started increasing exponentially with each new processor design while at the same time hardware prices dropped like a brick. Almost overnight inexpensive desktop computers appeared that could easily handle the processing and graphics demands a software package like ArcInfo placed on them. I was working as a GIS program manager for the US Army when this hardware revolution hit the field and watched as in less than two years inexpensive desktop PCs caught up with and then quickly surpassed the processing power of the Unix-based Sun, Silicon Graphics and HP systems we had been relying on. What also helped was Microsoft's release of WindowsNT at about the same time. Finally we had a serious network-ready enterprise operating system running on high capacity hardware that didn't make our budget guys weep every time we said we needed to do an upgrade.
ArcInfo is the flagship product of the ESRI line and is extremely powerful software. But in the 1980s ESRI realized that not everyone needed the processing power of ArcInfo (nor could they afford the nausea-inducing cost of an ArcInfo software license). ESRI introduced a lightweight version of ArcInfo that included most of the visualization capability of the high end package but left out the heavyweight analysis and data development functionality. They named it ArcView. It was priced right - something small organizations and even individuals serious about GIS could afford (if I remember correctly the GSA schedule price for a single ArcView license ran around $600 in 2000). The vast majority of today's GIS professionals cut their teeth on ArcView.
But ESRI's real contribution to the GIS profession is the development of data types that both support complex spatial analysis and can be shared across different software platforms. It is Dangermond's vision that GIS-based mapping and analysis solutions should not be a stovepipe, but a shared resource. This drove ESRI to develop the concept of the geodatabase. A geodatabase is a collection of data in a standard relational database management system (RDBMS) like Oracle or SQL Server, but the data has very unique spatial values (location in x, y and z coordinates) assigned to it. This means that GIS software can leverage the spatial values to relate the data in a location context and other RDBMS-based software systems can easily share their information with the geodatabase. The geodatabase only needs to store GIS-unique features and can pull and do analysis against associated data in another database.
ESRI also developed a version of the geodatabase that does not require a high powered relational database management system as it's foundation. About a decade ago ESRI introduced the concept of a file-based geodatabase designed for use by small organizations or groups. The file geodatabase is a simple to create yet powerful and extremely flexible data format that brings most of the power of the relational database and complex data analysis to the desktop machine and the individual user.
But what does the future hold? ESRI realized long ago that the Internet was the map content delivery vehicle of the future. Paper maps were headed to obsolescence and what Jack Dangermond describes as the 'rich web map' would quickly become the geospatial data visualization and analysis tool of the future. He's right, but only very recently has web technology started to catch up with his vision.
For the better part of a decade it was possible to hire professional web developers to create some very nice web mapping applications built on ESRIs early web technology called ArcIMS. The problem was that those applications were difficult to develop, difficult to maintain, and required a lot of heavy weight back-end web and database server technology. Only large enterprises and governments could support the hardware, software, development and maintenance costs. ESRI's web solutions were very much limited by the immature web development technologies available at the time. It is ESRI's vision that even the average geospatial professional working for a small business or local government should be able to develop, launch and maintain high quality web maps that bring value to the organization they support. ESRI started laying the groundwork for this vision back with their ArcGIS 9 series of software releases and the development of things like ArcGIS Server and the concept of Map Services. Two years ago they released ArcGIS 10 that brought a lot of maturity to the concept of integrated and streamlined web mapping using the Microsoft Silverlight and Adobe Flex web development environments, and the launch of ArcGIS Online with its peek into the future concept of 'cloud services' for hosting GIS data, services and web maps.
At it's recent worldwide user's conference ESRI announced the pending release of ArcGIS 10.1 with better integrated and streamlined web development tools. But ESRI also announced two new developments that are generating a lot of interest. The first is the announcement that ESRI has partnered with Amazon.com to host robust, enterprise-level cloud services for GIS web mapping, data hosting and application development. The idea is that an enterprise purchases an ArcGIS Server software license, passes that license over to Amazon and Amazon stands up and maintains the necessary database and web development environment for the enterprise. This is a huge development because it can free the GIS group supporting the enterprise from the often onerous and restrictive shackles placed on it by their local IT department.
The other announcement was the pending release of the ArcGIS Online Organizational Account program. The Organizational Account program appears to be targeted as smaller enterprises and groups that don't have the money or need to purchase full-up cloud services like those offered by Amazon. Under the Organizational Account concept an organization will be able to purchase data and web hosting services from ESRI on a subscription basis. It is still a 'cloud' model, but on a smaller, more tailorable scale that should allow small organizations to enjoy most of the capabilities of a full-up ArcGIS Server implementation.
The last good thing I need to discuss is another little-known program released this year - the concept of ArcGIS for home or personal use. ESRI's software licensing fees have escalated to the point that the geospatial professional simply can't afford a copy to use to keep his or her skills sharp. I noted above that the GSA price for an ArcView license used to run about $600 - a bearable cost if you were serious about GIS. However, the cost for an ArcView license now hovers around $1,600, far too much for even the serious home user. This year ESRI announced the ArcGIS for Home Use program. Anyone can purchase a 1-year license of ArcView for $100, a very reasonable price. Not only does this $100 include on-line software training and support, but you also get a very extensive suite of add-on modules like 3D Analyst, Spatial Analyst and Geostatistical Analyst. The total value of the software you get for your $100 subscription comes to over $10,000. One hell of a deal. Of course there are restrictions attached to this deal. The intent of the home use program is just that - you can only use it at home. You can also only use it for personal development/training purposes or non-profit use. Still, like I said, it's one hell of a deal.
__________________________________________________________
Now, it's not all rainbows and unicorns when it comes to ArcGIS and ESRI's position in the GIS world. All this GIS goodness is of little use unless it's leveraged in an environment with clearly defined professional standards. Nor can you allow a professional discipline to be defined by a software application or be inexorably joined to a piece of software. This is where ESRI's has failed the geospatial community, and they have failed in ways they can't even visualize from where they sit.
Here's the reality: geospatial engineering is the discipline, the term geospatial information systems - GIS - merely describes the tools geospatial professionals use to do their job. Where ESRI has failed is in using its industry position and influence to help clearly delineate the difference between the two. As a result, far too many engineering professionals view geospatial professionals as little more than button pushing software monkeys, one step up from data entry clerks.
Part of the culture Jack Dangermond has fostered and progressed through ESRI is the idea that GIS is for everyone and nobody owns it. What he is effectively saying is that GIS is the discipline; the tools and the software drive the field, not the other way around.
While community ownership is a noble goal, ESRI's dominance of the field gives lie to that very philosophy. Effectively, ESRI 'owns' GIS; it is by far the world's largest GIS software developer. It has either developed or successfully implemented most of the recognized spatial analysis processes in use today. It's data management features have driven the development of most of the spatial data standards in use today. The vast majority of geospatial professionals worldwide learned their trade using ArcGIS.
What is lacking, however, is a clear and recognized definition of just what a geospatial professional is. Dangermond is correct when he claims it's not his role to define what a geospatial professional should be - that is the job of the geospatial field and industry as a whole. But Dangermond has been the biggest catalyst in the geospatial world for the last 30 years. He and the resources he commands through ESRI have been in the best position to cajole and coerce the private sector, academia and the government to establish the roles, practices and responsibilities that define Geospatial Engineering as a formal discipline. He should have been the single biggest champion of the concept of Geospatial Engineering as a professional discipline. Instead he's been pretty much silent on the whole issue.
It is only in the last few years that the US Department of Labor developed a formal competency model for GIS (GIS, not Geospatial Engineering), and the GIS Professional Certification program is just starting to get its feet on the ground (after a disastrous grandfathering period that allowed perhaps hundreds of clearly unqualified individuals to get a GISP certificate and do damage to the reputation of the geospatial profession that may take years to overcome). Great, but all this should have happened 20 years ago.
What this means is that Geospatial Engineering is not respected as a professional discipline. I can tell you from long personal experience that geospatial professionals are looked down upon by other disciplines such as civil engineering and surveying, in large part because there are no testable and enforced standards that define us as a 'profession'. Guess what - they are right!
_________________________________________________________
Many readers are probably asking themselves "Huh? What's he getting at here?" I guess I'd ask the same question myself if I didn't understand the background issues.
I've been a topographer and geospatial engineer for over 30 years. A few months back I laid out my initial arguments in a post titled In Praise of the Old Topographer. In that post I made the argument that Geospatial Engineering is just a logical continuation of the older and much respected profession of Topographer. I also outlined my argument that geospatial information systems, including ArcGIS, are merely the tools that the Geospatial Engineer uses to do his or her job.
With this post my goal was to identify one of the main culprits that is keeping Geospatial Engineering from fully maturing into a recognized profession, a profession with it's own standards, roles and responsibilities.
ArcGIS is that culprit. On the one hand we have extraordinarily capable software that is almost single handedly responsible for bringing the discipline into the computer age and is poised to bring it fully into the age of world wide web. On the other hand, ArcGIS and it's parent company ESRI are almost single handedly responsible for holding the discipline back and keeping it from taking it's rightful place as a profession on par with other engineering disciplines.
For these reasons ArcGIS is the software I hate to love.
Sunday, April 24, 2011
The US National Map
Earlier this month the US Geological Survey (USGS) released their latest version of The National Map Viewer.
The US National Map is not a map per se. You can't ring up the USGS and say "Send me a copy of the National Map." It doesn't exist as a single product. The US National Map is a collection of digital geographic and geospatial data that, when brought together, forms the foundational map of the United States. Here's how the USGS describes it:
"As one of the cornerstones of the U.S. Geological Survey's (USGS) National Geospatial Program, The National Map is a collaborative effort among the USGS and other Federal, State, and local partners to improve and deliver topographic information for the Nation. It has many uses ranging from recreation to scientific analysis to emergency response. The National Map is easily accessible for display on the Web, as products and services, and as downloadable data. The geographic information available from The National Mapincludes orthoimagery (aerial photographs), elevation, geographic names, hydrography, boundaries, transportation, structures, and land cover. Other types of geographic information can be added within the viewer or brought in with The National Map data into a Geographic Information System to create specific types of maps or map views. The National Map is a significant contribution to the National Spatial Data Infrastructure (NSDI) and currently is being transformed to better serve the geospatial community by providing high quality, integrated geospatial data and improved products and services including new generation digital topographic maps."
OK, like I said, it's a collection of digital geographic and geospatial data that forms the foundational map of the US. Geeze, I think government bureaucrats get paid by the word.
Here is the USGS's introduction to the National Map program and the National Map Viewer:
The National Map Viewer is the USGS's on-line portal to all the data that makes up the National Map.
The Viewer is pretty good (if you are at all interested, it is built on ESRI's ArcGIS Server technology) and offers some neat functionality. It will provide location information in a number of formats, including US National Grid coordinates, it has a pretty robust reverse geocoding feature (click on a building on the map and the map returns the street address for that location) and it will provide spot elevations from the national elevation dataset. You can do area and distance measurements, add text and simple graphics and even add data from external sources like a GoogleEarth KML file or a web mapping service. You can also bring up indexes for the USGS's standard map products like the US Topo series of maps and link to them for download as GeoPDF files. For advanced users the Viewer offers some pretty good search and query builder functionality, so you can find specific data that is embedded in the data layers.
There are some shortcomings, however. The print function is essentially useless and is perhaps THE major drawback of this Viewer. About all it does is grab a screen shot of your viewer and dumps it to a PDF file. The USGS needs to wake up and realize that people still want quality paper maps and with today's technology it should be easy to print a fully detailed paper map with things like a grid, scale indicator, geographic extents, legend, etc.
The Viewer also exhibits a common issue found in web-based maps - map content naming conventions can be pretty obtuse and downright confusing. While the Viewer does pretty good with the base data layer naming conventions, when you start using advanced features like the Query Builder you start to interact directly with the database field names. For example, if I'm building a query to identify all the wetlands in my county I'm presented with a list of 'Columns' (which are the database field names). Those column names are confusing and don't mean anything to most humans. We get to pick from selections named 'ATTRIBUTE' or 'OBJECTID' or 'SHAPE_Area'. There is an easy solution to this - the GIS professional building this map can establish what are called 'field alias' names - a human-friendly nickname for each of the information fields. ATTRIBUTE can be displayed as 'Wetland Attribute', OBJECTID can be displayed as 'Wetland ID' and SHAPE_Area can be displayed as 'Wetland Area'. This naming convention issue usually reflects the fact that GIS professionals with little cartography experience compiled the data for use in the Viewer. (If I seem to be nit-picking here it is because I build maps for a living using this same technology. I know these are issues that are easy to fix and should have been taken care of before the Viewer was opened up to the public.)
These shortcomings aside, the National Map Viewer is pretty darned good. I'd say the USGS gets a good solid 'B' for this effort. If they'd improve the damned printing issue I'd give them an 'A'.
Brian
 |
| US National Map View of Maumee, Ohio |
 |
| The same view of Maumee, Ohio with the aerial image background turned on |
The US National Map is not a map per se. You can't ring up the USGS and say "Send me a copy of the National Map." It doesn't exist as a single product. The US National Map is a collection of digital geographic and geospatial data that, when brought together, forms the foundational map of the United States. Here's how the USGS describes it:
"As one of the cornerstones of the U.S. Geological Survey's (USGS) National Geospatial Program, The National Map is a collaborative effort among the USGS and other Federal, State, and local partners to improve and deliver topographic information for the Nation. It has many uses ranging from recreation to scientific analysis to emergency response. The National Map is easily accessible for display on the Web, as products and services, and as downloadable data. The geographic information available from The National Mapincludes orthoimagery (aerial photographs), elevation, geographic names, hydrography, boundaries, transportation, structures, and land cover. Other types of geographic information can be added within the viewer or brought in with The National Map data into a Geographic Information System to create specific types of maps or map views. The National Map is a significant contribution to the National Spatial Data Infrastructure (NSDI) and currently is being transformed to better serve the geospatial community by providing high quality, integrated geospatial data and improved products and services including new generation digital topographic maps."
OK, like I said, it's a collection of digital geographic and geospatial data that forms the foundational map of the US. Geeze, I think government bureaucrats get paid by the word.
Here is the USGS's introduction to the National Map program and the National Map Viewer:
The National Map Viewer is the USGS's on-line portal to all the data that makes up the National Map.
The Viewer is pretty good (if you are at all interested, it is built on ESRI's ArcGIS Server technology) and offers some neat functionality. It will provide location information in a number of formats, including US National Grid coordinates, it has a pretty robust reverse geocoding feature (click on a building on the map and the map returns the street address for that location) and it will provide spot elevations from the national elevation dataset. You can do area and distance measurements, add text and simple graphics and even add data from external sources like a GoogleEarth KML file or a web mapping service. You can also bring up indexes for the USGS's standard map products like the US Topo series of maps and link to them for download as GeoPDF files. For advanced users the Viewer offers some pretty good search and query builder functionality, so you can find specific data that is embedded in the data layers.
There are some shortcomings, however. The print function is essentially useless and is perhaps THE major drawback of this Viewer. About all it does is grab a screen shot of your viewer and dumps it to a PDF file. The USGS needs to wake up and realize that people still want quality paper maps and with today's technology it should be easy to print a fully detailed paper map with things like a grid, scale indicator, geographic extents, legend, etc.
The Viewer also exhibits a common issue found in web-based maps - map content naming conventions can be pretty obtuse and downright confusing. While the Viewer does pretty good with the base data layer naming conventions, when you start using advanced features like the Query Builder you start to interact directly with the database field names. For example, if I'm building a query to identify all the wetlands in my county I'm presented with a list of 'Columns' (which are the database field names). Those column names are confusing and don't mean anything to most humans. We get to pick from selections named 'ATTRIBUTE' or 'OBJECTID' or 'SHAPE_Area'. There is an easy solution to this - the GIS professional building this map can establish what are called 'field alias' names - a human-friendly nickname for each of the information fields. ATTRIBUTE can be displayed as 'Wetland Attribute', OBJECTID can be displayed as 'Wetland ID' and SHAPE_Area can be displayed as 'Wetland Area'. This naming convention issue usually reflects the fact that GIS professionals with little cartography experience compiled the data for use in the Viewer. (If I seem to be nit-picking here it is because I build maps for a living using this same technology. I know these are issues that are easy to fix and should have been taken care of before the Viewer was opened up to the public.)
These shortcomings aside, the National Map Viewer is pretty darned good. I'd say the USGS gets a good solid 'B' for this effort. If they'd improve the damned printing issue I'd give them an 'A'.
Brian
Subscribe to:
Posts (Atom)