Remedial GIS

[remedialgis]

You don't need to care about 3D GIS. Yet.

3D maps, as they stand, have the same general usefulness as 3D graphs and charts. Yes, the maps look a little prettier sometimes. Often, the 3D effects are simply a distraction from the actual purpose of the map or application. At best, they are a wash.

Why? Because people who are not visual artists, cartographers, or developers looking to do something new are not looking at maps because they are pretty.

Replace a mapbook (digital or plain paper) that a utility crew or police use with an identical product that includes some 3D buildings. You've accomplished two things:

You've probably wasted quite a bit of time

You might have annoyed a few of them with the unfamiliar look

Maybe a handful look at them and go "neat"

The most popular, maybe the only 3D map application of any popularity is Google Earth. And lets be honest with ourselves, do you use it for a real purpose? It isn't the first thing you open for directions. It's unlikely that you pop in there to take a virtual city tour.

I have seen a total of two useful applications of 3D maps software. They've been in use for this purpose for quite a while and such maps have not expanded for other uses.

One is for visualization, particularly for planned construction and viewsheds. In the United States, the visual impact of construction is a legal requirement of an environmental impact statement/assessment. Viewsheds can and are often more useful as 2D maps with an overlay that shows what is visible from a given point.

The other is for air traffic.

Quite a narrow set of use cases, which is why it isn't shocking that 3D maps have taken off even as software and hardware has made them trivial to create and use.

This could change of course. Some augmented reality stuff looks promising and essentially requires 3D visualization, but that might not really catch on until the formfactor of mobile devices changes drastically (so you need not hold up a phone like a dork to use them).

It is interesting there are hints that cartography is going in the opposite direction, with map visualization becoming simplier, tighter, and more functional.

You can do a lot with 2D.

[remedialgis]

ok, so intern interviews are officially over 26 interns - 4 passed fizzbuzz all interns were in their final CS semester

Startup founder on IRC

[remedialgis]

ArcSde/ArcServer Tips

Trust error messages. Error messages arising from some software or human error are extensively well documented and always tell you the underlying problem. Take this message for instance:

"Server object instance creation failed on all SOC machines. The base table definition string is invalid. Bad User."

Obviously, the problem is the existing cache on the server with an identical name. You should manually delete the folder on that machine. All of the errors messages you get will be similarily helpful and to-the-point.

User experience is for end users, the software is supposed to be hard

A lot of tools exist to automate common database operations - many exist as simple command line scripts (often paired with SQL) and provide a good way of managing large datasets and effect complex model changes with proper testing and without human error. A lot of web frameworks now support a migration pattern, where schema changes can be made and reversed in a maintainable, testable, and reliable fashion (see Rails Migrations, Django South, and very soon EFCodeFirst from Microsoft).

But that junk is for scrubs. Far better for you to push some buttons in a wizard than have a clean config file paired with a command one-liner that can be easily integrated into maintenance scripts. Database migration and automation tools are for chumps who don't want the continued experience of pushing "next" to recreate the network. Otherwise what excuse would you have to bring the server down?

It's totally necessary to have some middleware between you and your database

Modern relations database management systems are really unreliable/untested and none of them really support spatial types. Better to abstract away that nonsense with some DCOM goodness.

It's especially useful to have a multiple ways to use the middleware to effect some database change - like deleting a domain - that silently fails if it isn't in the right program or uses/doesn't use a toolbox tool.

Eliminating complexity is for people who don't want job security

Use every possible complex ArcSDE feature, that is what they are there for! If you don't start using, say, a versioned geometric network feature class dataset, some DBA without any GIS training might get it in his/her fool head that they can do what you do. Nothing is more dangerous. ArcSDE helps you avoid this cruel fate by making a simple insert statement into a tremendous pain.

Fear your database tables

If you've just installed ArcSDE, you will notice it has convienently created a number of tables with obscure names. This is just it's gentle way of letting you know that you no longer really own this data or have much control over it. It's important to avoid the inclination to start using that database in the same manner you might other databases - creating convienent views or reoccuring jobs that better insure data accuracy and validity. If you want to do anything like that, go open up ArcMap or Visual Studio with an ArcObjects template.

Jump into the new version immediately

Don't let those 987,345 service packs on the previous version fool you. You can immediately deploy that hot new version with those nifty new features. Yes, version 10 has three service packs already, but 10.1 is going to come out production-ready, so don't wait.

[remedialgis]

You should avoid ArcObjects

Why?

ArcObjects: Beating the dead IHorseFactory

It probably isn't necessary to go out and start ripping on ArcObjects. Every indication is that AO is on its way out the door - a new .NET native client was previewed at the Devsummit that isn't supposed to replace ArcObjects but really does. Even if that didn't exist, the use cases for ArcObjects have been progressively chipped away at by an expanding Python site package (arcpy) and an increasing emphasis on doing all communication via HTTP.

But it doesn't matter. ArcObjects exists, it's pretty irritating, and I'm hoping to find it cathartic to whine about it.

What's a constructor?

Want an FeatureClass object? Of course you do. It's like the thing you basically always want. Well there are a few ways to get one, and all of them are terrible. You might be under the impression you could just do something like this:

FeatureClass fc = new FeatureClass(aPathName);

Think again. There are almost no actual constructors in ArcObjects. Everything is pulled as a reference to the current application or through some Factory methods. Speaking of which...

IFactoryFactory factory = (IFactoryFactory)AnotherFactory.Create()

Little more needs to be said about this that wasn't said here. That FeatureClass you wanted? Here's how to get it.

ESRI.ArcGIS.Geodatabase.IWorkspaceFactory workspaceFactory = new ESRI.ArcGIS.DataSourcesFile.ArcInfoWorkspaceFactoryClass(); ESRI.ArcGIS.Geodatabase.IWorkspace workspace = workspaceFactory.OpenFromFile(databaseOrDirectory, 0); ESRI.ArcGIS.Geodatabase.IFeatureWorkspace featureWorkspace = (ESRI.ArcGIS.Geodatabase.IFeatureWorkspace)workspace; ESRI.ArcGIS.Geodatabase.IFeatureDataset featureDataset = featureWorkspace.OpenFeatureDataset(featureClassName); ESRI.ArcGIS.Geodatabase.IFeatureClass fc = (IFeatureClass)featureDataset;

Unless you wanted to go through an attached application. Then you have more options. Which brings me to my next point.

All the options in the world, none of them good.

You can get a FeatureClass object using the above, or by grabbing selected objects out of ArcMap itself, or via the table of context or by the ffffffffuuuuuuuuu

Maybe it is the Python guy in me, but if you have one relatively good way of doing things, a convention if you will, it helps maintenance, readability, and guides new programmers to proper practices. And the GIS/spatial dev field is awash in new programmers - many who are just new to spatial stuff and a whole lot that are new to software development in general.

The FeatureClass thing might be forgivable if it wasn't part of a pattern. Do you want to look up the features in a feature class with IEnumFeature OR IFeatureCursor? Wait, why are we writing up iterators like this?

Oh, .NET has a standard library? Full of data structures?

List, HashSet, Dictionary. These things work, people like them, and they are build in. Cool stuff like LINQ extends them and builds in all sorts of powerful features. People shouldn't have to find themselves doing this:

public static IEnumerable Features(IFeatureCursor featureEnum) { IFeature f = featureEnum.NextFeature(); while (f != null) { yield return f; f = featureEnum.NextFeature(); } }

Or variants on that theme. Making a commercial application SDK easier to use and integrate it with the platform you've chosen is really your job.

This isn't too shocking. Until recently, the Python API/SDK wasn't any better about using built in stuff. But it has improved greatly and continues to improve. It's sad to see that really isn't the case with ArcObjects yet. There is a reason that the Devsummit talk on the new SDK was a full house. People need to use a lot of custom features in ArcGIS products and they don't like ArcObjects. I'm happy it's likely death is imminent, and it is particularly wise they are handling this as an "addition" rather than a replacement (which freaks out some developers).

[remedialgis]

Projections, Datums, Transformations

It's possible they did glance over this stuff in school. Honestly it is pretty boring, but you'll need to at the very least know the difference between a datum and a projection.

As important is knowing how to fluidly move between them. Yes, if you have ArcMap there is a tool that will do this, but you don't or suppose it doesn't work (which happens). Once again, you can fall back to the holy trinity of open GIS: GDAL, Proj, and OGR. You can even use proj4js to reproject stuff on the fly in the browser. Seriously, for almost any given mundane GIS task you can save yourself a lot of time by going through these steps:

Does GDAL/OGR do this?

Has someone already wrote code that does this? Ask a mailing list/gis stackoverflow/google/twitter/quora/whatever.

Go do some research.

[remedialgis]

Open Data

You should know how to find data, manipulate it, and serve it up online for zero cost (besides your time and some hardware). Know what a webserver does, the basics of how HTTP works, and how to cache.

This will get you started. This book isn't bad either. This tech is a moving target though, so learn how to research.

[remedialgis]

Automate everything

Your default position should be to try to automate your (and others) work. All of it.

The GIS/spatial data/neogeography/cartography/map-huckster field is positively RIFE with stuff done by hand for no damned reason. Finding big, ugly Access "databases" that have been meticulously updated by hand for years - with the spatial data in the form of a description field like "west of elm street next to the Stuckeys" is a common occurrence.

And that's not the worst case - at least in the above example they had the presence of mind to record something about where the thing is.

In such cases, it is likely the very same data already exists somewhere else, can be calculated from existing data, and/or is less error prone when in the hands of a machine.

You want to automate everything. But you need to decide on priorities. It is best to ask yourself these questions:

How often does whatever business need/process come up?

How long will it take to automate?

How much human error is likely if it is done by hand?

Is anyone going to thank you for automating whatever this process is? Lack of praise might be an indication that no one cares about this thing and you can spend your time better elsewhere.

All of the stuff you do right now will probably be automated or made easier by some automation in 5-10 years. You'll want to be part of that bandwagon. You don't need to be a developer to do this - learn the command line (particularly cron and at).

It's healthier for you anyway.

[remedialgis]

database design

There's a relatively good chance you know nothing about databases, outside messing with ESRI Personal Geodatabases while coasting through some worthless tutorials in this book. You might have even noticed that these showed up as Microsoft Access files and you could open them up. You'll almost certainly have had issues with a PGDB being locked for no goddamned reason.

Going into detail on how to do good database design is outside the scope of an angerblog, but I'll tell you what you need to know:

What is a server? What is a database?

When do you use a relational database?

What is a foreign key? What is a primary key?

What is normalization? Costs/benefits?

When should you divide a table, combine multiple tables?

How is spatial data stored? How can you get it out

It's a lot to learn. Here is my recommendation: you should get some database software, PostGIS + Postgres or MSSQL Express are good choices, and play around in PgAdmin or MS SQL Studio Express. Make your own database tracking something of interest - your favorite music, vacation spots, whatever. Alternatively, pull some geodata from gov sources (parks, roads, boundaries) and try to import them into PostGIS.

How you expand such a thing in a modular way?

What happens when you need to add a new dataset?

How do the spatial and nonspatial data interact?

Is it easy to query?

Are there a bunch of null values that would never be filled by design?

The answers may not be obvious at first. You will want to read up on the subject. I'll get you started.

[remedialgis]

SQL

You can leave a GIS program without knowing SQL. This is actually a good way to gauge if you were screwed by your school.

This isn't some kind of fad/esoteric language meant only for nerdlords.  It is used for every relational database on the planet. There's a fairly good chance it is older than you are - and will outlive you. It isn't even hard! If you plan on doing anything non-trivial in GIS you have to know this. If you plan on calling yourself a GIS professional without inciting laughter, you should know this.

And actually, if you came from one of the schools that just worked through some ESRI exercises and handed out GIS certificates like slimey guys hand out 'special discount' tickets in Las Vegas, you've probably been doing SQL without knowing it. Check out this (probably) familiar screen:

Look at the bit just before you put in the text

Yeah, that's all there is to a SELECT query. Now go learn the rest from here or here.

[remedialgis]

Basic web stuff

There's just no excuse these days not to know the basics of HTML/CSS/Javascript. The online resources for it are fantastic.

Google Code University with a brain-dead simple video series (intro above).

W3Schools.com

HTMLDog

Basic web service stuff from Yahoo (just a bit more advanced).

Yes. It is possible to leave a four year university and not have any idea of how the internet works. If I am describing you, ask for your money back.

And don't be stupid. It is, right now, just barely possible to do something relevant in GIS and not know at least what is said in that above intro video. Three years from now that won't be the case.

[remedialgis]

Init

This is a blog of anger and disappointment.

These feelings are directed at those training or were just trained in Geographic Information Systems (GIS).

Many GIS programs are good, undoubtedly many of of the people coming out of them are good, but I've been personally shocked at the ignorance on display by people who share a similar educational background. I was personally shocked at my own ignorance.

But I've become tired of this anger, and this blog will be one of the ways I try to remedy this problem. Welcome to Remedial GIS.