SpatialKey

SpatialKey launches private beta

All of us at Universal Mind who have been hard at work on SpatialKey are incredibly proud to announce that we have begun a private beta program.

To really launch this product in style, we’ve put together this video to give an overview of the SpatialKey system. Even after watching it a dozen times I get blown away every time I watch it. Make sure to check it out in all its HD full-screen glory!

We’ve also got a whole series of videos that go into much more detail of all the different features. If you spend a few minutes watching some of those we’re betting you’ll start brainstorming ideas about how you can use SpatialKey with your own data.

This private beta period means that you fill out the form requesting access and we will be adding accounts on a first come first served basis over the next few weeks. We will add accounts as fast as we can over the following weeks and then after this private beta period we will open the system up to a full public beta, at which point anyone will be able to sign up for a trial account. SpatialKey will be a software as a service product that you’ll pay for with a monthly subscription based on your usage and data needs (pricing details will be coming very shortly, for now all accounts are a free trial with a cap of 10,000 rows per dataset).

I’ll be posting much more about SpatialKey and what you can do with the software, but until then go check it out and let us know what you think!

SpatialKey has been a long time coming and it wouldn’t be possible without the fantastic work of everyone involved. A personal thank you (in no particular order) goes out to every talented member of the SpatialKey family: Darron, Ben, Andy, Zach, Reggie, John, Tom G, Anthony, Francisco, Robert, Mike, Brandon, and Tom L.

Maps, SpatialKey

SpatialKey featured on the cover of ComputerWorld

The most recent issue of ComputerWorld magazine features a cover story called “Can Web 2.0 Save B.I.?” that features a case study on SpatialKey. In the article they interview Chief Jon Greiner of the Ogden Police Department in Utah. Ogden is the first installation of the enterprise version of SpatialKey Law Enforcement Dashboard (see the press release), and Universal Mind has been working closely with the Ogden PD to use the SpatialKey platform to develop what we think is a game-changing crime mapping product.

Here’s an excerpt from the article talking about SpatialKey:

Today, the officers are using the new BI tools to perform geographic profiling of crimes and analysis of police data “in seconds,” he says. Before, it could take days for the department’s single crime analyst to fulfill a report request. An added bonus is that experienced police officers with extensive street experience are now able to apply their firsthand knowledge to crime analysis.

“You have practitioners asking the what-if questions, which has changed the way we police,” Greiner says.

And here’s the cover of this month’s print edition of the magazine:

SpatialKey on cover of ComputerWorld magazine

That’s our heatmap! Yeah!

Maps, SpatialKey

Announcing SpatialKey – Geographic Information Without Limits

Today I’m proud to announce the launch of SpatialKey, the geospatial information visualization product I’ve been working on with our fantastic team at Universal Mind. I’ll make a bold statement that I stick by: this is the best web-based mapping product in existence. Today we’re releasing a “technology preview” that gives you a little glimpse at what we’ve been working on (just to whet your appetite until we release the full product).

Quick links
Before I explain what SpatialKey is I wanted to give a few quick links because I know a lot of you are going to have your ADD act up before you read the rest of the post.

SpatialKey Gallery – lists a few dataset/template pairs that we think tell great stories. Read the descriptions of the datasets and then launch the app to play with the data yourself.
San Antonio Prostitution hotspots

San Antonio Prostitution Crimes – This link will jump you straight into exploring the prostitution crimes in San Antonio from Jan 2006 – July 2007. Check out how clearly the heatmap points out the corners that are the hotspots in the city.
Growth of Walmart – This link will load the Walmart dataset into a playback template that lets you click play and watch Walmart take over America.

Beyond points on a map

Overwhelmed with markers

We’ve been seeing the same tired approach to web-based mapping for years now. Everyone throws markers on a map. You want to track crime? Throw a bunch of markers on a map. Little pin markers work fine if you’re showing a few data points. Want to see the location of Starbucks within a 3 block radius of your house? Use markers. But what if you want to see the total sales of all Starbucks worldwide? Or all crimes for the past 10 years? For the whole country?

SpatialKey uses some of the most advanced visualization renderings for geospatial data that have ever been seen on the web. The focus here is on aggregate renderings: heatmaps, thematic grids, graduated circles. 1,000 markers all piled on top of each other doesn’t help anyone. What you want to see is density or sum total value. SpatialKey focuses on rendering aggregate data in meaningful ways. We can show you a heatmap of the entire country and let you visualize any number of data fields. You want to see the heatmap represent total sales of all stores in the region? No problem. You want to see average house price over the past 10 years? We can do that.

Heat Map	Heat Grid

We haven’t seen innovative technology in this industry since Google let you drag the map. (I actually vividly remember that moment when I first dragged a Google map and my mouth started to water). It’s time to move beyond points on a map.

Your data doesn’t have limits
Try adding 10,000 data points to a Google Map. I dare you. What happens? If you’re using the “My Maps” feature of Google Maps, you’re limited to only show 200 points at a time, then you have to page through your data. And to top it all off you’re limited to a whopping total of 1,000 data points in the entire data set. So you get to page through 5 pages of data and only see 1/10th of your total data set anyway. If you create your own application with the Google Maps AJAX API you’re going to have serious performance problems when you get up into a few hundred markers. We think that’s ridiculous.

SpatialKey breaks through the limits of previous mapping technology in two ways. First, we’re simply faster. Flash can process and render data far faster than JavaScript. We can render 10,000 data points in a matter of milliseconds. You simply can’t do that with any JavaScript API out there. Second, we’re smarter. We aggregate data to produce heatmaps instead of just trying to overlay markers one on top of the other. Fundamentally, a massive dataset is an information visualization problem, not a technical one. You need better renderings to convey massive amounts of data, and that’s what SpatialKey delivers.

This is just the beginning
This is a technology preview. That basically means we’re showing you some cool stuff, but we’ve got way more up our sleeve. We’re looking for feedback on what we’ve got, and we’re hoping to get you excited about what we’ll be rolling out. We’ll be releasing new versions of SpatialKey Personal that will let you easily import your own data (if you’ve got an Excel file with addresses you can drop it right in). We’re also going to be releasing SpatialKey Enterprise, which lets you load a data set of any size (millions and millions of points). And then we’ve got a third product that we’re launching called SpatialKey Law Enforcement Dashboard, which is an enterprise version of SpatialKey specifically targeted toward police departments (includes special law enforcement reporting templates). And in the meantime we’ll be rolling out some more example datasets for you to play with, so keep an eye on the SpatialKey blog.

So go check out the SpatialKey Gallery and play with some data. We’re looking for feedback during this phase, so if you have any suggestions or (god forbid!) you run across bugs, please let us know by emailing feedback@spatialkey.com.

Uncategorized

I’m a full-time employee with Universal Mind

I have recently decided to make the move from an independent subcontractor to a full-time employee with Universal Mind. This is a big decision for me, one that I’ve been debating for a while now. I’ll try to explain my thinking a little. My official job title will be Principal Software Engineer, pretty pimp huh?

Building the team
We’re building a badass team to dominate the Flex world. Honestly, we’re bringing together the best developers, no question about it. One reason I decided to come on full time is so I can have more influence over that team. One of the best parts of my job is getting to work with some of the smartest guys I know. Guys who teach me new stuff every day on a project. The full time move gives me more pull to attract developers I want to work with. It shows I’m committed to what we’re doing and that there’s a reason to come work with us rather than some other Flex company. Now I really get to build this team. I have a say in who we hire. The biggest draw for me is the chance to work with rockstar developers, and at one point I thought maybe the best option was to branch out independently and get a kick-ass group of guys together to do our own thing. Then I started to realize that we already have a pretty badass team at UM, and in the last month we’ve brought on some ridiculous talent. So if I want to work with the best guys, we seem to be pulling them into UM already, so I might as well take that and run with it.

LaunchPAD The product and intellectual challenges
We’re going to be raising the bar for mapping applications. If you think the ability to plot points (oooh, different colors even!) on a Google map is the definition of awesome mapping technology then we’re going to blow you away. I’ve been working as a subcontractor for the past 8 months and a lot of my time has been spent on a project dubbed LaunchPAD. We’re taking mapping to a whole new level. I made a video a month ago showing off some of the stuff I’ve been working on. That’s just scratching the surface. You ain’t seen nothing yet.

One of my biggest concerns about taking a full time job is that the day to day work will get stale. The consulting world offers a unique opportunity to work for short periods on varied projects. Some people like this, some people don’t. I found that it was a great fit for me mentally. I got to go into a random company, learn about the cool stuff they’re doing, and help them out. Then a few weeks later I got to go to a completely new project in a completely different field and solve different problems. If you ever get bored with a project you only have to deal with it for a few weeks or months, then you get to leave the poor suckers who work full time on the project to deal with it for the rest of their lives while you go off and do something fun (I’m not actually speaking from experience here, I loved all my projects, but I always viewed this aspect as a big plus for consulting).

The full-time move with UM is a bit more than a simple shift in employment status. It also represents a shift from being a normal consultant (ie jumping project to project) to focusing primarily on the mapping stuff we’re doing. When I was first considering this my immediate concern was that I would get bored working on the same project for more than a few months. Now, after digging in deeper in the online mapping/GIS space I’ve become confident that there’s enough breadth within this area to hold my interest and keep me challenged. Figuring out how to leverage mapping for a police department is quite different than for a farming company, or a public health organization. There are a lot of fundamental technical challenges (like how to draw client-side heatmaps, or how to visualize millions of data points) that are common to many of the scenarios, but each implementation brings unique things you can’t anticipate. And the fact that my software is going to be used in a police department just makes me feel badass.

And I won’t be spending 100% of my time on the same project. I forget the proposed split, but it’s understood that I’ll have a certain number of weeks allocated to consulting for clients (gotta help UM pay the bills somehow…). So I’ll get to focus on mapping, but still jump on fun consulting projects throughout the year.

The lifestyle and commitment issues
The move to full time won’t actually have much of an effect on the lifestyle I’ve grown to love while being a subcontractor. I’ll still be working from home in San Francisco. I’ll still have flexibility in setting my daily schedule. One big difference is that I’m now expected to work a certain amount throughout the year. I have allocated vacation time, which is good because I get paid for doing nothing, but bad because I’m expected to not take much more than the time I get. As a subcontractor I could decide to take time off between projects for as long as I wanted. Sure, I wouldn’t get paid for these breaks, but I had the flexibility to take a few weeks off and travel, or just chill out.

One of the biggest mental hurdles for me was the shift in thinking about time. The time scale for a decision like this is in years, as opposed to weeks or months when I was a subcontractor. Before, I would go from one project to another, but often wouldn’t know what I would be doing next after my current project was over. I might have some time off, I might jump straight onto another project. There was a certain unknown aspect to that lifestyle that appealed to me. I don’t have kids, I don’t have a mortgage. I don’t need to know that I have work lined up for the next few years. I don’t like planning. For most things in my life I don’t plan beyond a month or two. I just bought my plane tickets to Europe for Christmas last week. I knew I was going for months. I just don’t like planning. I won’t try to psychoanalyze myself here (my girlfriend is a psychologist, so that’s her department), but I’ll just say that the shift in time scale was a difficult mental challenge for me.

So what does this mean?

It means Universal Mind is badass. We got some of the best Flex talent in the world, no question (and I’m not being an egotistical ass, I’m referring to everyone else working with UM). I won’t even tell you how ridiculous the Flex job market is right now. If you know Flex you can get a bazillion jobs anywhere. If you’re the best you can come work with us.
It means Flex mapping is going to explode. We’re going to make some crazy shit.
It means you can call me a sell-out for taking a full time job, but you have to do so to my face, not like a wuss in the comments.
It means the emails about new jobs or consulting gigs can stop coming in now… thanks. I’m spoken for.

Flex/Flash/Actionscript, Maps

Not Your Mamma’s Maps

My most recent work at Universal Mind has been focused on developing data analysis tools for geospatial and temporal data. Basically that means making sweet Flex mapping apps. Our current project has been dubbed LaunchPAD, and you can read more on the Universal Mind website.

Analysis through space and time
The time element of a dataset is an important piece that we’re focusing on. In nearly all mapping applications I see on the web the time aspect is completely ignored. There are some exceptions to this (the Oakland Crimespotting site being a great example), but overall time gets the shaft. We’re trying to expose time as a powerful method to drill down into a dataset. About halfway through the video below I talk about what we’re doing with the timeline in LaunchPAD.

Data Analysis Framework
We’re building a framework to analyze data in space and time. We’re not building an application to analyze police data (although the tools we’re developing work really well to analyze police data). The demo video mostly focuses on the police scenario because that’s one that we’re most actively pursuing, but the idea is that LaunchPAD can apply to any dataset that has spatial and temporal aspects. Near the end of the video I show an example of pulling in current earthquake data from a GeoRSS feed and using the same toolset to analyze that data.

AS3 Mapping APIs
I’ve been working with a few of the AS3 mapping APIs, in my previous post about MapMyPix I mention working with the MapQuest API. I’ve also been working heavily with the ESRI API, and this is what we’ve used to create LaunchPAD.

Video Demonstration
I did a video demo walking through some of the features we’ve been working on. It’s kind of long, close to 6 minutes, so feel free to skim it and jump around if your ADD kicks in (I added some bookmarks to the video so you can jump to specific parts). And yes, I know I had the mic too close to my mouth, I was going for the sensual phone-sex operator voice, but I don’t think that turned out quite right.

Live Demo
You can play with a live demo version of the software. So if you’re too cool for school to watch explanatory videos, just pull up the demo app and start playing around. You’ll figure it out. Make sure to turn on the heatmap option 🙂

Flex/Flash/Actionscript, Maps

What I’ve been working on: MapMyPix

One of the recent projects I have been working on with a team at Universal Mind is the MapMyPix AIR application that allows you to take data from a handheld GPS device and mash that together with pictures from a digital camera and automatically geocode your photos.

What it does
You drop a GPX file (which is what you get from your GPS device) onto the application and it shows your route. Then you drop a bunch of pictures onto the route and it automatically analyzes your pictures and figures out where along the route they were taken. Now you’ve got a geocoded photo set that you can view on the map and upload to Flickr to share. When the photoset is uploaded to Flickr the geocoded data is uploaded as well, so you can view your photos on a map. You can share the uploaded photoset with other people by sending out a link, or you can embed your mapped photos in a blog post or web page.

Who this is for
Primarily this application is for someone who has a handheld GPS device and likes taking pictures on hikes, or ski trips, or other outdoor activities. The application also has broader implications in the business realm where large sets of photos need to be combined with GPS data (scenarios like real estate brokers, insurance investigators, etc come to mind). If you’re a backpacker and you take pictures along a hike and you carry a GPS device, then this app is right up your alley.

If you don’t have a GPS device you can still use the application to geocode and upload your photos. You can manually drag individual photos to set their geo locations. Further enhancements to the application are in the works to make the application more friendly to people without GPS devices, since for this initial release we were really focusing on users who have GPS data.

Why this is cool
There are other ways to geocode your pictures. Flickr allows you to geocode pictures that you’ve already uploaded. But the process involves dragging each individual picture onto a location on the map. When you’ve got a hundred pictures from a hike you took this is painful. Not to mention that you’ve got to upload your photos first from your computer, then do all this manual gecoding one by one after the photos are online. Lame.

With MapMyPix we’ve combined the geocoding and uploading process into a smooth interface. Since this is an AIR app you get all the slick mapping capabilities that you can get with a Flex app. And since it’s a desktop application we allow you to simply drag a whole folder full of hundreds of images straight from your computer onto the map and boom, they get geocoded. Sweet.

Some technical details
The app is a Flex application using the MapQuest AS3 API. Universal Mind made the MapQuest AS3 API (you can read the businessy sounding case study here). I’ve worked with both the ESRI AS3 API and the MapQuest API. Without making any comparisons I will say that the MapQuest API is great to work with. It’s fast, simple, and does everything I’d expect a map API to do (I’m not trying to kiss ass here, it actually is a very solid API).

The core application is a desktop AIR app. However, we also used almost entirely the same code base to create an online viewer application that is used to display photosets once they’ve been uploaded. After a photoset is uploaded we provide a unique URL and embeddable code that uses a Flex application to display the images on a map for sharing. Since the whole thing is a Flex app it took almost no work to spin off the online viewer after we had made the AIR desktop version.

Where we had issues

Time
We had 3 developers. We thought we had 3 weeks. We had a blank canvas (ie we were able to brainstorm and come up with the ideas behind the app from the ground up). Awesome. We brainstormed like mad and came up with far more ideas than we could implement in 3 weeks. So we identified core functionality and then stuff that we hoped to have time for (got a bunch of that in there) and stuff that we would like to see (got some, but most of that is still waiting for the next version). Then after a week of work it turned out that there had been a mistaken assumption about the deadline and we lost 2 days. That’s like 10% of the total time. So that meant that the last week and a half were kind of a mad rush to accomplish as much as possible. I actually enjoyed this a lot, but it was definitely a bit stressful.
AIR suckiness
This was Flex 3 Beta 1 and that meant we had a completely asynchronous SQLLite implementation. That sucked. Lots of time was spent coming up with ways of doing complex database transactions in an asynchronous environment. That wasn’t my job on this project so my life wasn’t too bad, but if you ever meet John Yanarella ask him how much he likes asynchronous SQLLite work. In addition to the DB stuff, I ran into lots of AIR memory issues. Our app dealt with very large images (multi-megabyte photos) and lots of them. We wanted the user to be able to quickly browse between hi-res images. But it turns out that if you load a few hi-res images in an AIR app one after the other the garbage collector decides to go on strike and you hit memory errors and the app crashes. Great. Adding in the localConnection GC hack fixed this.
Timezones! Argh!
So GPS units save their time in UTC time, which is awesome because that means we don’t have to worry about time zones. If we have a date from a GPS we always know what universal time that means. The problem comes when you add in a camera. Cameras typically store images in JPEG format with EXIF data to store metadata. One field in the EXIF data is the data and time the photo was taken. Cool. But what ISN’T stored in the EXIF data is a friggin timezone to let you know the real time. So we have to make some assumptions about the camera timezone. We allow the user to adjust the UTC offset for their camera (which means which timezone the camera is set for) and then the syncing with the GPS can work as it should. If only the stupid EXIF data told us timezone…

A few more pretty pictures:

Links:

Download the MapMyPix AIR application from http://www.mapmypix.com
Read more about MapMyPix on Tom Link’s blog post here
Check out the MapQuest AS3 API
Read about Universal Mind working with MapQuest to make their AS3 API

And here’s an example of an embedded MapMyPix geocoded photoset that was uploaded using the app:

Doug McCune

Data Artist

Tag Archives: Universal Mind