Author Archives: Doug

Art, Maps, SpatialKey

Night Vision Maps of the WikiLeaks Iraq Casualty Data




In 1990 I was an eight year-old kid. And like most eight year-olds I spent a lot of time in front of my TV. But the summer of 1990 was different. Instead of cartoons I was watching the first Gulf War.

The television media coverage of the war was everywhere. Except these weren’t the gruesome images of the Vietnam era. These were images that looked more like videogames. We had cameras attached to bombs that used night vision and targeting scopes as they dove into buildings. All the images were a bit fuzzy, a bit grainy, either tones of gray or green, and overall void of emotion.

But we were watching people die.

The disconnect between the emotionless images shown on TV and the reality that they represented has always stuck with me. The fact that we could (and still do) present something so horrible in such a clinical, disconnected way makes my head spin.

WikiLeaks Iraq data

I’ve been experimenting with mapping the recently released data from WIkileaks that documents deaths in Iraq. All told the data documents 108,365 deaths, which we assume are just a fraction of the true casualty count from this war. Of those deaths, 65,641 were civilians.

I’ve used SpatialKey to produce some heatmaps of these deaths by recreating the aesthetic of the night vision images we’ve grown so used to seeing. I downloaded the data from the compiled spreadsheet published by the Guardian. Each image has a high resolution version available (2,474 pixels by 1,419 pixels).

A view of the entire country


High resolution version

A closer look at the area of Baghdad


High resolution version

More details of Baghdad


High resolution version

Why?

These images are meant to be a bit provocative. Every tiny blurred dot represents someone dying. And yet it’s all presented in a way that everyone is comfortable with. When you glance at these images you don’t immediately think of killing. We’re so used to seeing emotionless, blurry images of rockets exploding and precision bombs targeting buildings that we disconnect the image from the reality. These are images of death. And the fact that we’re comfortable looking at them should give us pause.

Standard
Data Visualization, Maps, SpatialKey

Take the Tangent – Video of my 360|Flex Keynote

I was honored to be asked to give a keynote presentation at 360|Flex in DC last month. All the sessions were recorded, and John Wilker was gracious enough to let me post the full video of my keynote.

This keynote was a bit different. I went out on a limb a bit and talked about the experimental projects that I’ve been working on, and my belief in the importance of pursuing fun experiments to stay invigorated and passionate about our work. It covers a number of mapping and data visualization projects I’ve been playing with, but the point was really that we all need to pursue what we’re passionate about. For me that happens to be maps right now, but everyone has their own unique areas of interest.

If you’re interested in mapping work then the projects I talk about should be right up your alley. But even if you’re not a map geek, I think the presentation is still interesting and (I hope!) inspirational.

You can also see the slide deck on its own, but I think the video gives much better context to the slides.

Standard
SpatialKey

SpatialKey on ABC News in Salt Lake City

I just found out a local news story in Salt Lake City featured SpatialKey and the work we’re doing with the Ogden Police Department. Pretty sweet seeing your code come to life on TV 🙂

Here’s the video:

Standard
Art, Data Visualization, Maps

If San Francisco Crime were Elevation

I’ve been playing with different ways of representing data (see my previous night lights example) and I decided to venture into 3D representations. I’ve used a full year of crime data for San Francisco from 2009 to create these maps. The full dataset can be download from the city’s DataSF website.

A view from above

This view shows different types of crime in San Francisco viewed directly from above. The sun is shining from the east, as it would during sunrise.

top_500

I love how some of the features in these maps are pretty consistent across all the crime types, like the mountain ridge along Mission St., and how some of the features only crop up in one or two of the maps. The most unique map by far is the one for prostitution (more on that further down).

An alternate view

Here’s the same data but from a different angle, which helps show some of the differences.

UPDATE: Whoops, I screwed up originally and had a duplicate image. The original graphic showed the same map for Vandalism and Assault (both were the Vandalism map). This updated graphic has the correct map for Assault.

right_500_2

Many of the maps have peaks in the Tenderloin, which is that high area sort of in the north-east center area of the city. Some are extremely concentrated (narcotics) and some are far more spread out (vehicle theft).

My favorite map is the one for prostitution (maybe “favorite” is the wrong choice of words there). Nearly all the arrests for prostitution in San Francisco occur along what I’m calling the “Mission Mountain Ridge”, which runs up Mission St between 24th and 16th.

EDIT: I’ve been corrected. Upon closer inspection the prostitution arrests are peaking on Shotwell St. at the intersections of 19th and 17th. I’m sure the number of colorful euphemisms you can come up with that include the words “shot” and “well” are endless.

I love the way the mountain range casts a shadow over much of the city. There’s also a second peak in the Tenderloin (which I’m dubbing Mt. Loin).

prostitution_500

Drug crimes are also interesting to look at, since so much of the drug activity in San Francisco is centered in a few distinct areas. We can see Mt. Loin rising high above all the other small peaks. The second highest peak is the 16th St. BART peak.

drugs_500

There are other consistent features in these maps, in addition to Mt. Loin and the Mission Range. There’s a valley that separates the peaks in the Mission and the peaks in the Tenderloin, which is where the freeway runs (Valley 101). You’ll also notice a division in many of the maps that separates the southeast corner. That’s the Hunter’s Point Riverbed (aka the 280 freeway).

Disclaimer

These maps were generated from real data, but please don’t take them as being accurate. The data was aggregated geographically and artistically rendered. This is meant more as an art piece than an informative visualization.

Standard
Uncategorized

Nate Beck’s Birthday Surprise at 360|Flex

Payback’s a Queen!

Nate got a special surprise in the middle of his session at 360|Flex. This should teach you a) don’t fuck with me and b) don’t do a presentation on your birthday.

Happy birthday Nate!

P.S. Apologies for the shaky camera work, there were plenty of other video cameras in the room recording (including the official tripod camera), so there will be a bunch of copies of this video up soon I assume.

Standard
Uncategorized

What do you do with a giant head?

You make it vacuum your floor of course.

Inspired by this fantastic piece of work by Eric Testroete, my friend and I created my very own paper craft giant head (my buddy Todd did all the hard work of the 3D modeling and texturing). Of course, once you have such an amazing giant head, you need to figure out what the hell to do with it. And so boredom on a Friday night plus a few beers plus a giant head plus a roomba equals a magical vacuuming head!

A few more pictures of the head in action:
giant_head

gianthead2
(*not my baby)

As I figure out more shenanigans to get into with my giant head I’m sure I’ll post more ridiculous photos and videos.

Standard
Data Visualization

Recreating William Playfair’s Import/Export Charts

I’ve been reading William Playfair’s Commercial and Political Atlas, in which he invented the line chart. In the book, Playfair examines the imports and exports between Britain and various countries. To illustrate these trade relationships, Playfair created the first ever line charts that show the change in trade over time.

The Inspiration

Each section of the book covered a different country, and each one contained a chart that showed the imports and exports like this:

playfair_north_america_trade2
playfair_ireland
Two line series are shown, one for imports and one for exports, and shading is used to show when there was a “balance in favor of England” (when there were more exports than imports).

My Recreation

I’ve been captivated by these charts and wanted to recreate them, but with modern data. You can find tons of US trade data at the US Census Bureau’s website, including a spreadsheet that has all the data in one place. I downloaded that data and put together a little application to create Playfair-esque charts.

Click this screenshot to play with the app yourself:
playfair_app_screenshot
View source is enabled.

The app displays all the countries that the US has trade data for, month by month going back as far as 1985. Each country is displayed in the list on the left with a sparkline chart of the trade data. A red fill indicates we are importing from a given country more than we are exporting, and a light green fill indicates we are exporting more than we are importing.

Exploring the data

The charts tell some really interesting stories. Some of the charts show a nearly identical relationship of imports to exports, both growing at the same rates, like these charts of the UK and Guatemala.
united_kingdom
guatemala

While some other charts show different relationships. Notice how exports to Hong Kong have been steadily increasing, but imports from Hong Kong have been declining.
hong_kong

Or we can see what imposing sanctions on a country looks like, as illustrated by sanctions on Burma that were put into place in 2003:
burma

Or what a coup in Haiti looks like:
haiti

Or what a massive tsunami can do to a place like the Maldives:
maldives

We can see the massive growth of China (and notice how interestingly seasonal each year is, peaking in October):
china

And one final one that I find very interesting, isn’t a country, but the import and export of what is classified as “Advanced Technology Products“, which includes things like biotech and advanced electronics products. Notice how up until the early 2000s we were exporting more of these products than we were importing, but by 2002 that balance shifted and the gap continues to increase:
advance_tech_products

I had fun creating this app, but one thing I didn’t expect was how much fun researching the charts was going to be. The charts that stuck out with trends that were abnormal all had interesting stories to tell about the history of the country.

In closing, I’ll end with a quote from Playfair in which he describes the concept of displaying numeric values in a line chart (remember, he was the first person to actually do this):

As the eye is the best judge of proportion, being able to estimate it with more quickness and accuracy than any other of our organs, it follows, that wherever relative quantities are in question … this mode of representing it is peculiarly applicable; it gives a simple, accurate, and permanent idea, by giving form and shape to a number of separate ideas, which are otherwise abstract and unconnected.

Well said, Mr. Playfair, well said. Your charts are just as effective nearly 200 years later.

Standard
Flex/Flash/Actionscript

Reflections on the first 360|Flex

The next 360|Flex conference is coming up in March in San Jose. I got to thinking about my history with 360|Flex and wanted to reflect on how important the conference has been for me. This post is about the very first 360|Flex conference that was in March 2007. It’s self-indulgent, but I hope it doesn’t come off as egotistical. My goal is to explain my personal experience and my path, not to brag about how cool I am.

I attended the first 360|Flex conference in San Jose simply as an attendee. I barely knew anyone (at least personally) in the Flex world. I had been attending the Flex user group meetings in San Francisco for just a bit before this conference and so I had met a few people very briefly like Tom Ortega (conference organizer) and Ted Patrick (Adobe evangelist). I had been blogging for literally only 3 months. I was still very inexperienced (even if I tried to mask that) and I wasn’t at all confident in myself (I really tried to mask that).

Opportunity #1:

The night before the first keynote of 360|Flex I got an email from Mark Anders from Adobe (who I had never met) telling me he was going to be showing a demo app of mine during the keynote. In the email he said:

“Make sure you attend the keynote, because I’m going to feature your RSS reader in it as a demo!  I hope that’s ok! 🙂  I’ll also be asking you to stand up and take a bow, so make sure you look great!  See you tomorrow!”

anders_rss_reader_360flexHope it’s OK? Fuck yeah, it’s awesome. Completely unexpected. I had butterflies in my stomach all night until the next morning, I was like a little schoolgirl. I got there early the next day, I wore a black sweater that I thought looked kind of professional in a dorky tech way, sort of Steve Jobs inspired. I realize now that I probably looked like a tool, but hey, it worked alright for me. Keynote came, the demo was shown, people seemed impressed and clapped and I did a little stand up wave kind of awkward thing. Super nervous, totally weird, but people saw my face. And after the keynote I met Mark and chatted with him and some fellow Adobe employees a bit (at the time I was seriously considering trying to get a job at Adobe, so I was sucking up pretty big time I think).

Opportunity #2:

On the second day of the conference Ted Patrick approached me and asked if I could put together a demo of FlexLib components to be shown at the keynote on the last day of the conference. I think this was in the afternoon and he was going to show it the next morning. I proceeded to go sit in the back of the last session of the day and crank out a little demo explorer app to showcase some of the components. To spice things up I threw in an extra water eyecandy effect that I was in the middle or working on. I threw the demo together fairly quickly and got it over to him for the next day. The demo was just a small part of his keynote, but during it he did get my name in there and point me out in the crowd, and people seemed to dig the water effect nonsense 🙂

So with a whole lot of luck I had managed to get my name mentioned in both the keynotes of 360|Flex. I’m grateful for these opportunities, they felt like they came out of nowhere and suddenly people were coming up to me to talk (as opposed to me sheepishly approaching people in the awkward tech-dork way).

Opportunity #3:

I remember sitting in the back of some session (can’t for the life of me remember anything about the session) and I happened to be sitting next to Tom Link and Brett Cortese from Universal Mind. I wasn’t at all tapped into the Flex consulting scene at that time, so I didn’t know the name Universal Mind. But I was sitting there in the back and Tom was reviewing some app that UM was working on. It was a sort of business dashboard app that had a slick UI that let you customize various charts. I can’t remember the exact conversation, but I remember looking over his shoulder and being impressed with the app and chatting about it a bit. He mentioned Darron Schall’s name at some point. I had been collaborating with Darron a bit on FlexLib and so I got online and started asking him about UM. Turns out he had only good things to say (and he apparently said good things about me to UM as well). After 360|Flex I left Tom with one of my ridiculous business cards. We ended up getting in touch shortly after and following up and the rest is history. I just passed my two year anniversary working for UM and I’ve had a hell of a good time.

Swing and miss

At the conference Ted Patrick introduced me to a guy from Yahoo. He was interested in hiring a Flex contractor for an internal project. If I recall correctly the basic plan was to have this Flex app that let Yahoo employees create different layouts and combinations of various Yahoo widgets, which would eventually turn into Yahoo web pages. It was sort of a prototyping tool for internal use. I had never done Flex consulting work before. I had never done any independent consulting whatsoever. I didn’t have the utmost confidence in my own abilities and I certainly didn’t feel comfortable asking for a lot of money. But I figured, fuck it, what’s the worst that can happen?

So with no prior consulting experience I quoted him $125 an hour. Pulled straight out of my ass. My logic was that I didn’t actually need the work yet (I was happily unemployed) but that if it came through and I was making over a hundred dollars an hour I’d would dive in head first (and be scared shitless in the process). I was far more scared of actually getting the gig than not, since then I’d have to actually be worth what I saw as such a ridiculous rate. I ended the email like this (parentheses are my thoughts, not in the actual email): “I’m happy to discuss the project further. I find the concept very interesting (I didn’t really), so even if I’m not involved in the development I’d be more than happy to give my opinions on the project (this was true, since I had no experience whatsoever I figured that any involvement would help me). Let me know how you want to proceed, and regardless let’s keep in touch about work that Yahoo is doing with Flex (having contacts inside companies like Yahoo can never hurt).” Send. And then I never even got a response. Ego bruised, but not badly.

The Jesse Warden validation

warden_thumbThis is funny because it’s a story I’ve never told Jesse (or anyone for that matter) and it showcases just how insecure I was at the time. I had been talking to Tom from UM about consulting and he had mentioned that Jesse Warden had been doing a bit of consulting for them as well, and that I should talk to him to see what UM was like. Jesse (who I had never met) was sitting against some wall working on a laptop and I walked over and said some awkward introduction like “Hey Jesse, it’s nice to meet you, I’m Doug McCune.” After being caught completely off guard with Jesse’s caffinated million-word-a-minute style of speech we got to talking a bit and then he said something like, “Where the hell did you come from? Like all the sudden here’s this Doug McCune guy coming out of nowhere making our computers blow up with this crazy fire and shit.” (I had recently released a fire effect demo). It’s funny how that interaction sticks so vividly in my mind. I probably laughed it off and said some kind of thanks, but inside I was beaming with pride. Jesse fucking Warden knew who I was and thought my shit was cool. That’s actually the exact moment that I knew that I was on some kind of path here, that this was actually going to work out.

This conference was huge for me

I had no expectations going into the conference, I figured I’d attend like I had attended various conferences before, sit in the back of sessions, take notes and write down links, and hopefully learn good material. I had no idea I’d get the kind of exposure in the keynotes that I ended up getting, or that I’d make the connections that I did. I ended up meeting Jesse, Deepa, Ben Stucki, Renaun Erickson, Mark Anders, Ely Greenfield, and countless other people. And these people that I met actually knew who I was. I couldn’t fucking believe it. My head was literally spinning at the end of it all.

Standard
Data Visualization

My Data Visualization Presentation from RIAdventure (full video and slides)

On the RIAdventure conference I gave a presentation about the past, present, and future of data visualization as I see it (fun side note: RIAdventure is the only conference I can say I “went on”). Luckily, the organizers filmed the entire thing, and we now have the video of the whole presentation that you can watch. This presentation covered a brief history of the field of data visualization, with the focus on the invention (in the not too distant past) of many data visualization techniques we take for granted. The point of the historical exercise was to point out that new opportunities with new data that we have before us present new opportunities for invention. I talked about new trends I see emerging in the data itself (massive datasets, city data, you life data, stream data) and what those trends mean for us as data visualization software engineers (I also argue that everyone will be a “data viz” engineer to some degree in the future).

I hope you enjoy the presentation, it was a lot of fun to create and to present. I learned a ton from the research and it was exciting thinking about the future of the field. Below is the full video (low resolution streaming from vimeo, or you can find higher resolution streaming form screencast here, or you can even download the full video file). Also embedded below are the slides that go along with the presentation, and you can always download the slides as a PDF.

Also check out some of the other presentations from RIAdventure.

Video:

Slides:

Standard
Data Visualization

The American Credit Crisis (of 1772) Visualized

One of the first time-series line charts ever drawn was a visualization of the great American credit crisis (but probably not the credit crisis that comes immediately to mind). If you were to look at this chart today you might even mistake it for the charts of the housing credit crisis of the past few years.

playfair_north_america_trade2
(see image credits below for image details)

This chart was created by William Playfair and published in 1786 in The Commercial and Political Atlas. In that work Playfair literally invented the line chart. This particular chart shows the imports and exports between Britain and America between 1700-1800. The red line is the line for exports from Britain to America, and the lighter yellow line is the imports from America. You can see the relationship of imports to exports stays relatively constant for the first 50 years (1700-1750) and then the exports start shooting up dramatically, at a rate much greater than the increase in imports.

Compare that with this chart of housing prices, created by the New York Times.
nyt_chart_cropped
(image from the New York Times)

I guess we know what a credit-driven catastrophe looks like. And it’s not only the image itself that looks similar, at times his words sound as if he’s writing today about our current financial mess.

Between 1750-1772 there was a rapid increase in exports from Britain to America. These exports were the result of many new merchants hoping to strike it big by shipping goods to the new settlers. But the reason things got out of control has to do with credit. Merchants started lending and borrowing on credit to finance their get-rich-quick schemes of selling stuff to America. Playfair writes (all emphasis added is mine),

Ever since the invention of paper credit, trade has had a latitude it did not before enjoy, and its progress being less natural, has become more intricate. That bound set and preserved by the nature of things was removed, when paper credit was first invented; previous to which, nothing represented wealth that was not wealth itself, or that was not physically worth the sum it represented; and in order to give credit in business, it was absolutely necessary either to possess, or to have borrowed capital.

And because of this new credit, people started making business decisions that were insane. They started shipping products to America before they knew they could sell them. Since the money was free they took irrational risks. And if your business venture failed miserably you could always just hide from your creditors in that new land of opportunity.

Of the eventual crash, Playfair writes,

For the first fifty years, we observe the simple and regular growth, from poverty to wealth, of a new country; during the succeeding twenty years, we are astonished at the extent and operation of a mad mercantile speculation carried on by our own country; and the period which succeeds, shews the catastrophe that so airy and so ill-founded a project was likely, sooner or later, to experience. There is not any branch of trade, which, from the nature of its progress, affords so much instruction as this. It merits equally the attention of the philosopher, the politician, and the merchant; for it throws light upon all the three different objects of their pursuits.

Isn’t that beautiful? Almost the same words could apply to the current financial crisis. And one final quote that I like, which also made me think of our current crisis:

Upon the manner in which business is conducted, depends something more than merely the gaining or losing a little money. The happiness of numbers of innocent individuals is frequently depending upon the success of projects, with the formation of which they had no concern. What numbers have been ruined, and how many more deprived of fortune, by our ill-conducted trade with America?

What numbers have been ruined indeed.

I’ve been reading the works of Playfair to understand the history of data visualization (in this same work he also invented the bar chart, and in a successive work he invented the pie chart). I wanted to make sure I understood the history of statistical charts, since as they say, those who cannot remember the past are condemned to repeat it. I didn’t realize that phrase would also apply so perfectly to the text accompanying the images.

* Image Credits
The first image above is from William Playfair’s Commercial and Political Atlas, 3rd edition, published in 1801. The scan is of a copy contained in the University of Pennsylvania’s Annenberg Rare Book and Manuscript Library. It was reproduced in a publication by Cambridge University Press entitled The Commercial and Political Atlas and Statistical Breviary, published in 2005, which was compiled by Howard Wainer and Ian Spence (and if you want to be even more technical the image above is a reproduction from a Google scan of the Cambridge University scan). As was decided in Bridgeman vs Corel Corp (full text), a reproduction of a work of art in the public domain is not protected by copyright. As was stated in that verdict: “While it may be assumed that this required both skill and effort, there was no spark of originality — indeed, the point of the exercise was to reproduce the underlying works with absolute fidelity. Copyright is not available in these circumstances.” I am reproducing the image here with that legal precedent in mind, and with the best of intentions. I would highly recommend that if you are interested in Playfair’s work you buy the reprint by Cambridge University Press. It contains full-color reproductions of the charts, and the introduction contains great biographic information about Playfair.
Standard