Data Visualization, Maps

Dorky Data Visualization Pumpkin: Minard’s Graph of Napoleon’s March

Happy Halloween! This is the dorkiest pumpkin I’ve ever carved. For those of you into data visualization or mapping, maybe you can recognize it:

napoleon_pumpkin_1

This is a pumpkin representation of Charle’s Minard‘s visualization of Napoleon’s march into Russia in 1812. This graphic is considered by some (ie Edward Tufte) to be the “best statistical graphic ever drawn.” The graph shows the size of Napoleon’s army as they marched to and from Moscow. You can see how the army shrank as they approached Moscow. Once they reached Moscow they found the city had been abandoned and burned. Then they marched back home, except it was through a brutal Russian winter and nearly killed the remaining army. By the time they return home you can see the size of the army is just a small trickle.

minard_small

Beyond just the two charts of the march to and from Moscow, the graphic also serves as a map, with the paths indicating where the troops were geographically. And below the map is a temperature chart that visualizes how severe the winter weather was, which correlates with some of the major drops in troops on the way home.

The carved pumpkin ended up being very hard to take a photo of because the graph wraps around over half the pumpkin’s circumference. So I tried to take a few pictures to get the different sides. I carved the march to and from Moscow, as well as the temperature chart along the bottom.

napoleon_pumpkin_2

napoleon_pumpkin_3

Hope everyone has a great Halloween tonight!

Standard
Maps, SpatialKey

New SpatialKey Crime Example for San Francisco

We just posted a new example of using SpatialKey to visualize crime in San Francisco. We load in 90 days of crime data from the city, then filter down to only include sales of heroin, crack cocaine, and methamphetamine within 1,000 feet of a school. Why those particular crimes around schools? The SFPD just launched a new initiative called “Operation Safe Schools” that specifically targets these drug crimes. If you’re caught dealing crack, heroin, or meth around a school while the school is in session you can get extra prison time.

Check out the video below and read the full article on the SpatialKey blog.

Read the whole article on the SpatialKey blog to see how we put this together and learn more about the SFPD’s “Operation Safe Schools.” You can also watch the full resolution video on YouTube

Standard
Flex/Flash/Actionscript, Maps, SpatialKey

Drinking at 10am and geeking out on SpatialKey and Flex

James Ward and Jon Rose just published the latest episode of Drunk on Software that features members of the SpatialKey team, including myself, Tom Link (CTO of Universal Mind), and Brandon Purcell (Director of Technology for UM).

As a brief disclaimer in case I slur any words near the end: I was in Denver for a short trip and we squeezed in a time to meet with Jon and James right before I had to head to the airport to fly home. The only problem was that we had to meet at about 10am in the morning. And since the show is called Drunk on Software we obviously had to be drinking. So by the time I got on my flight I was probably 6 beers down 🙂

A big thanks to Jon and James for making the time to have us over (that’s the living room of Jon’s house). And thanks for the beer guys!

Standard
Maps, SpatialKey

SpatialKey featured on the cover of ComputerWorld

The most recent issue of ComputerWorld magazine features a cover story called “Can Web 2.0 Save B.I.?” that features a case study on SpatialKey. In the article they interview Chief Jon Greiner of the Ogden Police Department in Utah. Ogden is the first installation of the enterprise version of SpatialKey Law Enforcement Dashboard (see the press release), and Universal Mind has been working closely with the Ogden PD to use the SpatialKey platform to develop what we think is a game-changing crime mapping product.

Here’s an excerpt from the article talking about SpatialKey:

Today, the officers are using the new BI tools to perform geographic profiling of crimes and analysis of police data “in seconds,” he says. Before, it could take days for the department’s single crime analyst to fulfill a report request. An added bonus is that experienced police officers with extensive street experience are now able to apply their firsthand knowledge to crime analysis.

“You have practitioners asking the what-if questions, which has changed the way we police,” Greiner says.

And here’s the cover of this month’s print edition of the magazine:

SpatialKey on cover of ComputerWorld magazine

That’s our heatmap! Yeah!

Standard
Maps, SpatialKey

Announcing SpatialKey – Geographic Information Without Limits

SpatialKey LogoToday I’m proud to announce the launch of SpatialKey, the geospatial information visualization product I’ve been working on with our fantastic team at Universal Mind. I’ll make a bold statement that I stick by: this is the best web-based mapping product in existence. Today we’re releasing a “technology preview” that gives you a little glimpse at what we’ve been working on (just to whet your appetite until we release the full product).

Quick links
Before I explain what SpatialKey is I wanted to give a few quick links because I know a lot of you are going to have your ADD act up before you read the rest of the post.

  • SpatialKey Gallery – lists a few dataset/template pairs that we think tell great stories. Read the descriptions of the datasets and then launch the app to play with the data yourself.
  • screenshot067.jpg
    San Antonio Prostitution hotspots

    San Antonio Prostitution Crimes – This link will jump you straight into exploring the prostitution crimes in San Antonio from Jan 2006 – July 2007. Check out how clearly the heatmap points out the corners that are the hotspots in the city.

  • Growth of Walmart – This link will load the Walmart dataset into a playback template that lets you click play and watch Walmart take over America.



Beyond points on a map

screenshot068.jpg
Overwhelmed with markers

We’ve been seeing the same tired approach to web-based mapping for years now. Everyone throws markers on a map. You want to track crime? Throw a bunch of markers on a map. Little pin markers work fine if you’re showing a few data points. Want to see the location of Starbucks within a 3 block radius of your house? Use markers. But what if you want to see the total sales of all Starbucks worldwide? Or all crimes for the past 10 years? For the whole country?

SpatialKey uses some of the most advanced visualization renderings for geospatial data that have ever been seen on the web. The focus here is on aggregate renderings: heatmaps, thematic grids, graduated circles. 1,000 markers all piled on top of each other doesn’t help anyone. What you want to see is density or sum total value. SpatialKey focuses on rendering aggregate data in meaningful ways. We can show you a heatmap of the entire country and let you visualize any number of data fields. You want to see the heatmap represent total sales of all stores in the region? No problem. You want to see average house price over the past 10 years? We can do that.

Heat Map Heat Grid
SpatialKey Heat Maps SpatialKey Heat Grids

We haven’t seen innovative technology in this industry since Google let you drag the map. (I actually vividly remember that moment when I first dragged a Google map and my mouth started to water). It’s time to move beyond points on a map.

Your data doesn’t have limits
Try adding 10,000 data points to a Google Map. I dare you. What happens? If you’re using the “My Maps” feature of Google Maps, you’re limited to only show 200 points at a time, then you have to page through your data. And to top it all off you’re limited to a whopping total of 1,000 data points in the entire data set. So you get to page through 5 pages of data and only see 1/10th of your total data set anyway. If you create your own application with the Google Maps AJAX API you’re going to have serious performance problems when you get up into a few hundred markers. We think that’s ridiculous.

SpatialKey breaks through the limits of previous mapping technology in two ways. First, we’re simply faster. Flash can process and render data far faster than JavaScript. We can render 10,000 data points in a matter of milliseconds. You simply can’t do that with any JavaScript API out there. Second, we’re smarter. We aggregate data to produce heatmaps instead of just trying to overlay markers one on top of the other. Fundamentally, a massive dataset is an information visualization problem, not a technical one. You need better renderings to convey massive amounts of data, and that’s what SpatialKey delivers.

This is just the beginning
This is a technology preview. That basically means we’re showing you some cool stuff, but we’ve got way more up our sleeve. We’re looking for feedback on what we’ve got, and we’re hoping to get you excited about what we’ll be rolling out. We’ll be releasing new versions of SpatialKey Personal that will let you easily import your own data (if you’ve got an Excel file with addresses you can drop it right in). We’re also going to be releasing SpatialKey Enterprise, which lets you load a data set of any size (millions and millions of points). And then we’ve got a third product that we’re launching called SpatialKey Law Enforcement Dashboard, which is an enterprise version of SpatialKey specifically targeted toward police departments (includes special law enforcement reporting templates). And in the meantime we’ll be rolling out some more example datasets for you to play with, so keep an eye on the SpatialKey blog.

So go check out the SpatialKey Gallery and play with some data. We’re looking for feedback during this phase, so if you have any suggestions or (god forbid!) you run across bugs, please let us know by emailing feedback@spatialkey.com.

Standard
Maps

Tracking my location – forever

I have recently started tracking my geographic location, with the intent of keeping an automatic log of where I am, ideally for the rest of my life. I have no idea how long I’ll realistically be able to keep this updated, but I figure “forever” sounds like a good goal.

Method
firefoxscreensnapz011.pngI just got my very first Blackberry (the Curve), which has built-in GPS. The first thing I wanted to figure out (after getting gmail on the phone) was how to store position reports automatically. There are a few pieces of software that sort of do this, although some applications store the position logs locally on the SD memory card, which you then download to your computer. There are also some subscription-based services catered more toward enterprise-level fleet management that do way more than I need (TeleNav Track, Accutracking) . The SD card approach is no good for me. I’m lazy, and I know there’s no way I’m ever going to transfer the position log from my phone to my computer. The subscription service is a bit more realistic (Accutracking even has a REST API), but I’m not excited about having to keep a subscription paid and active and rely on someone else for my reports. The only real workable solution was to find an application that automatically sent the position information to a web server, and ideally to my own webserver so I can have full control over the data.

firefoxscreensnapz012.pngEnter Mologogo. Mologogo is a free application and service for GPS-enabled phones that does exactly what I wanted. It runs automatically every few minutes (mine’s set to run every 5 minutes) and sends your position report to the Mologogo online service, which keeps track of your points. That’s halfway to what I needed. The icing on the cake is that the Mologogo application has an awesome feature that lets you set your own URL that the service will also send the position report to. Bingo. This lets you save your position reports in your own database to do whatever you want. I grabbed some of the sample PHP code from the Mologogo wiki and installed it on my personal web server. After configuring the Mologogo Blackberry app I was getting position reports logged in my database.

Why?
This is not about showing where I am in real-time. This is not about “life-streaming” or “life casting” or whatever the current buzzwords are for letting strangers watch every moment of your life. I do not plan on showing my current location on my blog, or letting people see exactly where I am as I’m there. I’m not going to do that for a few reasons, and privacy is one of them I guess, although I’m not really freaked out about the idea of having my current location be public knowledge. The main reason is that I think that stuff is boring as hell. I watched Justin.tv for all of about 5 minutes, and the entire draw of that kind of lifecasting is the hope that the person you’re watching is going to do something stupid, which usually happens specifically because they’re living to entertain their audience. Other people (in fact most of the people who use Mologogo I assume) like the concept of simply letting people see their latest position reports. But why? I don’t find the idea of viewing where some random guy (or even a good friend) is at the current instant compelling. I suppose if all my friends automatically told me where they were that could be kind of cool and helpful for meeting up. But whatever, that’s semi-useful and still boring.

This is about analyzing historical data. It’s about the visualization of years, the visualization of decades. It’s about being able to see the geographic context of my life. I probably won’t do anything with this data for a year or two. It only becomes truly valuable after extended periods of time.

To try to think about how much data we’re talking about here’s the approximate math:

1 report every 5 minutes = 12 reports an hour = 288 reports a day = 105,120 reports a year = 1 million reports a decade

So theoretically let’s say I can somehow figure out how to actually do this constantly until I’m 80 years old. That would be about 5 and a half million records that would catalog my movement around the world for most of my life. That amount of data is already easy to store online. I can store millions of records like this in an inexpensive hosted database, ideally duplicated in multiple locations for redundancy. So the challenge isn’t about how to store the data, it’s about what to do with it, and that’s where it starts to get fun.

Visualizing the geospatial context of life
The important things in your life have a geographic context. Moving to a new city, changing jobs (even if that just means a slightly different commute), traveling on vacation, etc etc. The geographic movements of your life are directly tied to meaningful experiences. Being able to visualize those movements is like having a picture of a certain time of your life.

I wish I had the data for the past few years. It would have shown me in college, then moving to San Francisco. I could have seen the daily commute I did for a few years, and then the change to working from home. Imagine a heatmap on a map showing where I spend my time. While I was commuting to work you would have been able to make out the route of the train I took every day, and the amount of time I spent simply getting to and from the office. Compare that with my current lifestyle, which now involves either working from my home or my girlfriend’s place, which would also show a kind of “commute”, but this one is very different. This commute represent a shift in where I spend my time, and even beyond showing a change in my work life, it also signifies the strengthening of our romantic relationship. This is a visual representation of a very meaningful part of my life. Being able to visually see this movement is a way to quantify important changes. I imagine creating some amazing looking visual images of this type of data and hanging them on the wall. A single image would represent so much.

Challenges
The biggest challenge is going to be keeping new data updated and old data accessible as technology changes. In another year or two I will have a different mobile device, at which point I’ll have to figure out how to use the new device to keep the position reports updated. In another ten years who knows what kind of device I might be carrying around in my pocket. There’s just no way to predict that kind of stuff. So every few years I’ll have to figure out how to keep my reports updated using new technology. I’ll also have to adapt to changes in how I store that data. Storing it in a database works well now, but inevitably I’ll have to migrate that data from one database to another, or even move from a relational database to some other form of storage. This means I’ve got to either stay geeky enough to know how to implement this stuff for the rest of my life, or I’ve got to figure out a solution that literally requires no changes for the next few decades (I don’t think this is at all possible).

There are also some more immediate challenges. I don’t know how to keep my positions logged as I travel internationally yet. Even as I’m in the US I’ve got to ensure that the logging software is always running on my phone. My phone needs to be on at all times for the reports to come in. Ideally I’m going to try to write my own custom software to do exactly what I want, just so I know how it works under the hood in case something breaks or I upgrade devices. I’m not stoked about the fact that I’m tied to software written by someone else. I’ve also obviously got to keep the webserver that receives the reports up and running at all times. I’ll probably try to move to more of a database in the cloud approach using Amazon SimpleDB and EC2 so I can rely on Amazon instead of my current web host (I imagine I’ll always be relying on some company to handle this kind of stuff).

I’m excited about the possibilities for using this data in a few years, I just hope I can keep the position feed coming in.

Standard
Flex/Flash/Actionscript, Maps

Not Your Mamma’s Maps

My most recent work at Universal Mind has been focused on developing data analysis tools for geospatial and temporal data. Basically that means making sweet Flex mapping apps. Our current project has been dubbed LaunchPAD, and you can read more on the Universal Mind website.

Analysis through space and time
The time element of a dataset is an important piece that we’re focusing on. In nearly all mapping applications I see on the web the time aspect is completely ignored. There are some exceptions to this (the Oakland Crimespotting site being a great example), but overall time gets the shaft. We’re trying to expose time as a powerful method to drill down into a dataset. About halfway through the video below I talk about what we’re doing with the timeline in LaunchPAD.

Data Analysis Framework
We’re building a framework to analyze data in space and time. We’re not building an application to analyze police data (although the tools we’re developing work really well to analyze police data). The demo video mostly focuses on the police scenario because that’s one that we’re most actively pursuing, but the idea is that LaunchPAD can apply to any dataset that has spatial and temporal aspects. Near the end of the video I show an example of pulling in current earthquake data from a GeoRSS feed and using the same toolset to analyze that data.

AS3 Mapping APIs
I’ve been working with a few of the AS3 mapping APIs, in my previous post about MapMyPix I mention working with the MapQuest API. I’ve also been working heavily with the ESRI API, and this is what we’ve used to create LaunchPAD.

Video Demonstration
I did a video demo walking through some of the features we’ve been working on. It’s kind of long, close to 6 minutes, so feel free to skim it and jump around if your ADD kicks in (I added some bookmarks to the video so you can jump to specific parts). And yes, I know I had the mic too close to my mouth, I was going for the sensual phone-sex operator voice, but I don’t think that turned out quite right.

Live Demo
You can play with a live demo version of the software. So if you’re too cool for school to watch explanatory videos, just pull up the demo app and start playing around. You’ll figure it out. Make sure to turn on the heatmap option 🙂

Standard
Flex/Flash/Actionscript, Maps

What I’ve been working on: MapMyPix

universal_mind.jpg

One of the recent projects I have been working on with a team at Universal Mind is the MapMyPix AIR application that allows you to take data from a handheld GPS device and mash that together with pictures from a digital camera and automatically geocode your photos.
128.png

What it does
You drop a GPX file (which is what you get from your GPS device) onto the application and it shows your route. Then you drop a bunch of pictures onto the route and it automatically analyzes your pictures and figures out where along the route they were taken. Now you’ve got a geocoded photo set that you can view on the map and upload to Flickr to share. When the photoset is uploaded to Flickr the geocoded data is uploaded as well, so you can view your photos on a map. You can share the uploaded photoset with other people by sending out a link, or you can embed your mapped photos in a blog post or web page.

mapmypixscreensnapz002.jpg

Who this is for
Primarily this application is for someone who has a handheld GPS device and likes taking pictures on hikes, or ski trips, or other outdoor activities. The application also has broader implications in the business realm where large sets of photos need to be combined with GPS data (scenarios like real estate brokers, insurance investigators, etc come to mind). If you’re a backpacker and you take pictures along a hike and you carry a GPS device, then this app is right up your alley.

If you don’t have a GPS device you can still use the application to geocode and upload your photos. You can manually drag individual photos to set their geo locations. Further enhancements to the application are in the works to make the application more friendly to people without GPS devices, since for this initial release we were really focusing on users who have GPS data.

mapmypixscreensnapz003.jpg

Why this is cool
There are other ways to geocode your pictures. Flickr allows you to geocode pictures that you’ve already uploaded. But the process involves dragging each individual picture onto a location on the map. When you’ve got a hundred pictures from a hike you took this is painful. Not to mention that you’ve got to upload your photos first from your computer, then do all this manual gecoding one by one after the photos are online. Lame.

With MapMyPix we’ve combined the geocoding and uploading process into a smooth interface. Since this is an AIR app you get all the slick mapping capabilities that you can get with a Flex app. And since it’s a desktop application we allow you to simply drag a whole folder full of hundreds of images straight from your computer onto the map and boom, they get geocoded. Sweet.

mapmypixscreensnapz004.jpg

Some technical details
The app is a Flex application using the MapQuest AS3 API. Universal Mind made the MapQuest AS3 API (you can read the businessy sounding case study here). I’ve worked with both the ESRI AS3 API and the MapQuest API. Without making any comparisons I will say that the MapQuest API is great to work with. It’s fast, simple, and does everything I’d expect a map API to do (I’m not trying to kiss ass here, it actually is a very solid API).

The core application is a desktop AIR app. However, we also used almost entirely the same code base to create an online viewer application that is used to display photosets once they’ve been uploaded. After a photoset is uploaded we provide a unique URL and embeddable code that uses a Flex application to display the images on a map for sharing. Since the whole thing is a Flex app it took almost no work to spin off the online viewer after we had made the AIR desktop version.

Where we had issues

  • Time
    We had 3 developers. We thought we had 3 weeks. We had a blank canvas (ie we were able to brainstorm and come up with the ideas behind the app from the ground up). Awesome. We brainstormed like mad and came up with far more ideas than we could implement in 3 weeks. So we identified core functionality and then stuff that we hoped to have time for (got a bunch of that in there) and stuff that we would like to see (got some, but most of that is still waiting for the next version). Then after a week of work it turned out that there had been a mistaken assumption about the deadline and we lost 2 days. That’s like 10% of the total time. So that meant that the last week and a half were kind of a mad rush to accomplish as much as possible. I actually enjoyed this a lot, but it was definitely a bit stressful.
  • AIR suckiness
    This was Flex 3 Beta 1 and that meant we had a completely asynchronous SQLLite implementation. That sucked. Lots of time was spent coming up with ways of doing complex database transactions in an asynchronous environment. That wasn’t my job on this project so my life wasn’t too bad, but if you ever meet John Yanarella ask him how much he likes asynchronous SQLLite work. In addition to the DB stuff, I ran into lots of AIR memory issues. Our app dealt with very large images (multi-megabyte photos) and lots of them. We wanted the user to be able to quickly browse between hi-res images. But it turns out that if you load a few hi-res images in an AIR app one after the other the garbage collector decides to go on strike and you hit memory errors and the app crashes. Great. Adding in the localConnection GC hack fixed this.
  • Timezones! Argh!
    So GPS units save their time in UTC time, which is awesome because that means we don’t have to worry about time zones. If we have a date from a GPS we always know what universal time that means. The problem comes when you add in a camera. Cameras typically store images in JPEG format with EXIF data to store metadata. One field in the EXIF data is the data and time the photo was taken. Cool. But what ISN’T stored in the EXIF data is a friggin timezone to let you know the real time. So we have to make some assumptions about the camera timezone. We allow the user to adjust the UTC offset for their camera (which means which timezone the camera is set for) and then the syncing with the GPS can work as it should. If only the stupid EXIF data told us timezone…

A few more pretty pictures:

mapmypixscreensnapz005.jpgmapmypixscreensnapz006.jpg
mapmypixscreensnapz007.jpgmapmypixscreensnapz008.jpg

Links:

And here’s an example of an embedded MapMyPix geocoded photoset that was uploaded using the app:

Standard