Payback’s a Queen!

Nate got a special surprise in the middle of his session at 360|Flex. This should teach you a) don’t fuck with me and b) don’t do a presentation on your birthday.

Happy birthday Nate!

P.S. Apologies for the shaky camera work, there were plenty of other video cameras in the room recording (including the official tripod camera), so there will be a bunch of copies of this video up soon I assume.

del.icio.us:Nate Beck's Birthday Surprise at 360|Flex digg:Nate Beck's Birthday Surprise at 360|Flex spurl:Nate Beck's Birthday Surprise at 360|Flex wists:Nate Beck's Birthday Surprise at 360|Flex simpy:Nate Beck's Birthday Surprise at 360|Flex newsvine:Nate Beck's Birthday Surprise at 360|Flex blinklist:Nate Beck's Birthday Surprise at 360|Flex furl:Nate Beck's Birthday Surprise at 360|Flex reddit:Nate Beck's Birthday Surprise at 360|Flex fark:Nate Beck's Birthday Surprise at 360|Flex blogmarks:Nate Beck's Birthday Surprise at 360|Flex Y!:Nate Beck's Birthday Surprise at 360|Flex smarking:Nate Beck's Birthday Surprise at 360|Flex magnolia:Nate Beck's Birthday Surprise at 360|Flex segnalo:Nate Beck's Birthday Surprise at 360|Flex gifttagging:Nate Beck's Birthday Surprise at 360|Flex

You make it vacuum your floor of course.

Inspired by this fantastic piece of work by Eric Testroete, my friend and I created my very own paper craft giant head (my buddy Todd did all the hard work of the 3D modeling and texturing). Of course, once you have such an amazing giant head, you need to figure out what the hell to do with it. And so boredom on a Friday night plus a few beers plus a giant head plus a roomba equals a magical vacuuming head!

A few more pictures of the head in action:
giant_head

gianthead2
(*not my baby)

As I figure out more shenanigans to get into with my giant head I’m sure I’ll post more ridiculous photos and videos.

del.icio.us:What do you do with a giant head? digg:What do you do with a giant head? spurl:What do you do with a giant head? wists:What do you do with a giant head? simpy:What do you do with a giant head? newsvine:What do you do with a giant head? blinklist:What do you do with a giant head? furl:What do you do with a giant head? reddit:What do you do with a giant head? fark:What do you do with a giant head? blogmarks:What do you do with a giant head? Y!:What do you do with a giant head? smarking:What do you do with a giant head? magnolia:What do you do with a giant head? segnalo:What do you do with a giant head? gifttagging:What do you do with a giant head?

I’m in love with the New York Times data visualization/infographics division. They consistently put out some of the most amazing visualization pieces (both in print and online) that I’ve ever seen. Their recently geographic analysis of Netflix ratings was absolutely superb. And we all probably saw their election maps (either for 2008 or 2004). They produce stunning displays that convey amazing amounts of information in a way that only interactive graphics can do. And they’re all done in Flash.

nyt_netflix
A Peek Into Netflix Queues


nyt_thanksgiving_recipes
What’s Cooking For Thanksgiving
nyt_swineflu
Swine Flu Cases Map


nyt_unemployment_explorer
The Jobless Rate for People Like You
nyt_vtech
Virginia Tech Shooting


nyt_parkingtickets
Map of Parking Tickets in New York City
nyt_how_people_spend_their_day
How Different Groups Spend Their Day


And for even more check out the NYT’s selected infographics list or simply do a Google search for “interactive graphic” on the New York Times website.

flashSo when you see images showing the missing plugin icon on the New York Times website on the iPad or iPhone, that’s not just some annoying ad that’s not playing or a streaming video. That’s some of the most cutting edge visualization work that’s being produced today. And without Flash it simply doesn’t exist.

Sure, you might be able to recreate some of these without using Flash (I’d argue that many you simply would never be able to do, but that’s for another debate). But the point isn’t whether or not you could eventually do it without Flash. The point is that the New York Times does them all with flash. So we need to ask why. It’s not an accident or an arbitrary technology choice. Newspapers operate on a schedule and a budget (and one that is getting tighter and tighter). The simple truth is, creating amazing visualizations like you see on the NYT website is possible and easy with Flash. They use the tools that get the job done most efficiently and produce the best end result. This isn’t an argument about whether it’s theoretically possible to create these types of visualizations without Flash, it’s about whether it’s being done. And save for handfuls of examples, it’s not (for every one good JavaScript visualization I’ll show you ten good Flash ones). Taking away the New York Times’ ability to use Flash is setting their data visualization department back 5 or 10 years. And it would mean that we, as readers and citizens, would be missing out on some of the most important journalism being produced today.

The New York Times (like all newspapers) is in crisis. They are trying to reinvent themselves in an online form. And as a news organization they are one of the most progressive and experimental out there. They are embracing the new medium by doing some of the best damn interactive graphic work I’ve ever seen. They make things that convey news and information in ways that draw people in and keep them coming back for more.

But without Flash they’re just a newspaper. And we all know newspapers are dying.

del.icio.us:The New York Times Without Flash digg:The New York Times Without Flash spurl:The New York Times Without Flash wists:The New York Times Without Flash simpy:The New York Times Without Flash newsvine:The New York Times Without Flash blinklist:The New York Times Without Flash furl:The New York Times Without Flash reddit:The New York Times Without Flash fark:The New York Times Without Flash blogmarks:The New York Times Without Flash Y!:The New York Times Without Flash smarking:The New York Times Without Flash magnolia:The New York Times Without Flash segnalo:The New York Times Without Flash gifttagging:The New York Times Without Flash

I’ve been reading William Playfair’s Commercial and Political Atlas, in which he invented the line chart. In the book, Playfair examines the imports and exports between Britain and various countries. To illustrate these trade relationships, Playfair created the first ever line charts that show the change in trade over time.

The Inspiration
Each section of the book covered a different country, and each one contained a chart that showed the imports and exports like this:

playfair_north_america_trade2
playfair_ireland
Two line series are shown, one for imports and one for exports, and shading is used to show when there was a “balance in favor of England” (when there were more exports than imports).

My Recreation
I’ve been captivated by these charts and wanted to recreate them, but with modern data. You can find tons of US trade data at the US Census Bureau’s website, including a spreadsheet that has all the data in one place. I downloaded that data and put together a little application to create Playfair-esque charts.

Click this screenshot to play with the app yourself:
playfair_app_screenshot
View source is enabled.

The app displays all the countries that the US has trade data for, month by month going back as far as 1985. Each country is displayed in the list on the left with a sparkline chart of the trade data. A red fill indicates we are importing from a given country more than we are exporting, and a light green fill indicates we are exporting more than we are importing.

Exploring the data
The charts tell some really interesting stories. Some of the charts show a nearly identical relationship of imports to exports, both growing at the same rates, like these charts of the UK and Guatemala.
united_kingdom
guatemala

While some other charts show different relationships. Notice how exports to Hong Kong have been steadily increasing, but imports from Hong Kong have been declining.
hong_kong

Or we can see what imposing sanctions on a country looks like, as illustrated by sanctions on Burma that were put into place in 2003:
burma

Or what a coup in Haiti looks like:
haiti

Or what a massive tsunami can do to a place like the Maldives:
maldives

We can see the massive growth of China (and notice how interestingly seasonal each year is, peaking in October):
china

And one final one that I find very interesting, isn’t a country, but the import and export of what is classified as “Advanced Technology Products“, which includes things like biotech and advanced electronics products. Notice how up until the early 2000s we were exporting more of these products than we were importing, but by 2002 that balance shifted and the gap continues to increase:
advance_tech_products

I had fun creating this app, but one thing I didn’t expect was how much fun researching the charts was going to be. The charts that stuck out with trends that were abnormal all had interesting stories to tell about the history of the country.

In closing, I’ll end with a quote from Playfair in which he describes the concept of displaying numeric values in a line chart (remember, he was the first person to actually do this):

As the eye is the best judge of proportion, being able to estimate it with more quickness and accuracy than any other of our organs, it follows, that wherever relative quantities are in question … this mode of representing it is peculiarly applicable; it gives a simple, accurate, and permanent idea, by giving form and shape to a number of separate ideas, which are otherwise abstract and unconnected.

Well said, Mr. Playfair, well said. Your charts are just as effective nearly 200 years later.

del.icio.us:Recreating William Playfair's Import/Export Charts digg:Recreating William Playfair's Import/Export Charts spurl:Recreating William Playfair's Import/Export Charts wists:Recreating William Playfair's Import/Export Charts simpy:Recreating William Playfair's Import/Export Charts newsvine:Recreating William Playfair's Import/Export Charts blinklist:Recreating William Playfair's Import/Export Charts furl:Recreating William Playfair's Import/Export Charts reddit:Recreating William Playfair's Import/Export Charts fark:Recreating William Playfair's Import/Export Charts blogmarks:Recreating William Playfair's Import/Export Charts Y!:Recreating William Playfair's Import/Export Charts smarking:Recreating William Playfair's Import/Export Charts magnolia:Recreating William Playfair's Import/Export Charts segnalo:Recreating William Playfair's Import/Export Charts gifttagging:Recreating William Playfair's Import/Export Charts
The next 360|Flex conference is coming up in March in San Jose. I got to thinking about my history with 360|Flex and wanted to reflect on how important the conference has been for me. This post is about the very first 360|Flex conference that was in March 2007. It’s self-indulgent, but I hope it doesn’t come off as egotistical. My goal is to explain my personal experience and my path, not to brag about how cool I am.

I attended the first 360|Flex conference in San Jose simply as an attendee. I barely knew anyone (at least personally) in the Flex world. I had been attending the Flex user group meetings in San Francisco for just a bit before this conference and so I had met a few people very briefly like Tom Ortega (conference organizer) and Ted Patrick (Adobe evangelist). I had been blogging for literally only 3 months. I was still very inexperienced (even if I tried to mask that) and I wasn’t at all confident in myself (I really tried to mask that).

Opportunity #1:
The night before the first keynote of 360|Flex I got an email from Mark Anders from Adobe (who I had never met) telling me he was going to be showing a demo app of mine during the keynote. In the email he said:

“Make sure you attend the keynote, because I’m going to feature your RSS reader in it as a demo!  I hope that’s ok! :-)  I’ll also be asking you to stand up and take a bow, so make sure you look great!  See you tomorrow!”

anders_rss_reader_360flexHope it’s OK? Fuck yeah, it’s awesome. Completely unexpected. I had butterflies in my stomach all night until the next morning, I was like a little schoolgirl. I got there early the next day, I wore a black sweater that I thought looked kind of professional in a dorky tech way, sort of Steve Jobs inspired. I realize now that I probably looked like a tool, but hey, it worked alright for me. Keynote came, the demo was shown, people seemed impressed and clapped and I did a little stand up wave kind of awkward thing. Super nervous, totally weird, but people saw my face. And after the keynote I met Mark and chatted with him and some fellow Adobe employees a bit (at the time I was seriously considering trying to get a job at Adobe, so I was sucking up pretty big time I think).

Opportunity #2:
On the second day of the conference Ted Patrick approached me and asked if I could put together a demo of FlexLib components to be shown at the keynote on the last day of the conference. I think this was in the afternoon and he was going to show it the next morning. I proceeded to go sit in the back of the last session of the day and crank out a little demo explorer app to showcase some of the components. To spice things up I threw in an extra water eyecandy effect that I was in the middle or working on. I threw the demo together fairly quickly and got it over to him for the next day. The demo was just a small part of his keynote, but during it he did get my name in there and point me out in the crowd, and people seemed to dig the water effect nonsense :)

So with a whole lot of luck I had managed to get my name mentioned in both the keynotes of 360|Flex. I’m grateful for these opportunities, they felt like they came out of nowhere and suddenly people were coming up to me to talk (as opposed to me sheepishly approaching people in the awkward tech-dork way).

Opportunity #3:
I remember sitting in the back of some session (can’t for the life of me remember anything about the session) and I happened to be sitting next to Tom Link and Brett Cortese from Universal Mind. I wasn’t at all tapped into the Flex consulting scene at that time, so I didn’t know the name Universal Mind. But I was sitting there in the back and Tom was reviewing some app that UM was working on. It was a sort of business dashboard app that had a slick UI that let you customize various charts. I can’t remember the exact conversation, but I remember looking over his shoulder and being impressed with the app and chatting about it a bit. He mentioned Darron Schall’s name at some point. I had been collaborating with Darron a bit on FlexLib and so I got online and started asking him about UM. Turns out he had only good things to say (and he apparently said good things about me to UM as well). After 360|Flex I left Tom with one of my ridiculous business cards. We ended up getting in touch shortly after and following up and the rest is history. I just passed my two year anniversary working for UM and I’ve had a hell of a good time.

Swing and miss
At the conference Ted Patrick introduced me to a guy from Yahoo. He was interested in hiring a Flex contractor for an internal project. If I recall correctly the basic plan was to have this Flex app that let Yahoo employees create different layouts and combinations of various Yahoo widgets, which would eventually turn into Yahoo web pages. It was sort of a prototyping tool for internal use. I had never done Flex consulting work before. I had never done any independent consulting whatsoever. I didn’t have the utmost confidence in my own abilities and I certainly didn’t feel comfortable asking for a lot of money. But I figured, fuck it, what’s the worst that can happen?

So with no prior consulting experience I quoted him $125 an hour. Pulled straight out of my ass. My logic was that I didn’t actually need the work yet (I was happily unemployed) but that if it came through and I was making over a hundred dollars an hour I’d would dive in head first (and be scared shitless in the process). I was far more scared of actually getting the gig than not, since then I’d have to actually be worth what I saw as such a ridiculous rate. I ended the email like this (parentheses are my thoughts, not in the actual email): “I’m happy to discuss the project further. I find the concept very interesting (I didn’t really), so even if I’m not involved in the development I’d be more than happy to give my opinions on the project (this was true, since I had no experience whatsoever I figured that any involvement would help me). Let me know how you want to proceed, and regardless let’s keep in touch about work that Yahoo is doing with Flex (having contacts inside companies like Yahoo can never hurt).” Send. And then I never even got a response. Ego bruised, but not badly.

The Jesse Warden validation
warden_thumbThis is funny because it’s a story I’ve never told Jesse (or anyone for that matter) and it showcases just how insecure I was at the time. I had been talking to Tom from UM about consulting and he had mentioned that Jesse Warden had been doing a bit of consulting for them as well, and that I should talk to him to see what UM was like. Jesse (who I had never met) was sitting against some wall working on a laptop and I walked over and said some awkward introduction like “Hey Jesse, it’s nice to meet you, I’m Doug McCune.” After being caught completely off guard with Jesse’s caffinated million-word-a-minute style of speech we got to talking a bit and then he said something like, “Where the hell did you come from? Like all the sudden here’s this Doug McCune guy coming out of nowhere making our computers blow up with this crazy fire and shit.” (I had recently released a fire effect demo). It’s funny how that interaction sticks so vividly in my mind. I probably laughed it off and said some kind of thanks, but inside I was beaming with pride. Jesse fucking Warden knew who I was and thought my shit was cool. That’s actually the exact moment that I knew that I was on some kind of path here, that this was actually going to work out.

This conference was huge for me
I had no expectations going into the conference, I figured I’d attend like I had attended various conferences before, sit in the back of sessions, take notes and write down links, and hopefully learn good material. I had no idea I’d get the kind of exposure in the keynotes that I ended up getting, or that I’d make the connections that I did. I ended up meeting Jesse, Deepa, Ben Stucki, Renaun Erickson, Mark Anders, Ely Greenfield, and countless other people. And these people that I met actually knew who I was. I couldn’t fucking believe it. My head was literally spinning at the end of it all.

del.icio.us:Reflections on the first 360|Flex digg:Reflections on the first 360|Flex spurl:Reflections on the first 360|Flex wists:Reflections on the first 360|Flex simpy:Reflections on the first 360|Flex newsvine:Reflections on the first 360|Flex blinklist:Reflections on the first 360|Flex furl:Reflections on the first 360|Flex reddit:Reflections on the first 360|Flex fark:Reflections on the first 360|Flex blogmarks:Reflections on the first 360|Flex Y!:Reflections on the first 360|Flex smarking:Reflections on the first 360|Flex magnolia:Reflections on the first 360|Flex segnalo:Reflections on the first 360|Flex gifttagging:Reflections on the first 360|Flex

On the RIAdventure conference I gave a presentation about the past, present, and future of data visualization as I see it (fun side note: RIAdventure is the only conference I can say I “went on”). Luckily, the organizers filmed the entire thing, and we now have the video of the whole presentation that you can watch. This presentation covered a brief history of the field of data visualization, with the focus on the invention (in the not too distant past) of many data visualization techniques we take for granted. The point of the historical exercise was to point out that new opportunities with new data that we have before us present new opportunities for invention. I talked about new trends I see emerging in the data itself (massive datasets, city data, you life data, stream data) and what those trends mean for us as data visualization software engineers (I also argue that everyone will be a “data viz” engineer to some degree in the future).

I hope you enjoy the presentation, it was a lot of fun to create and to present. I learned a ton from the research and it was exciting thinking about the future of the field. Below is the full video (low resolution streaming from vimeo, or you can find higher resolution streaming form screencast here, or you can even download the full video file). Also embedded below are the slides that go along with the presentation, and you can always download the slides as a PDF.

Also check out some of the other presentations from RIAdventure.

Video:

Slides:

del.icio.us:My Data Visualization Presentation from RIAdventure (full video and slides) digg:My Data Visualization Presentation from RIAdventure (full video and slides) spurl:My Data Visualization Presentation from RIAdventure (full video and slides) wists:My Data Visualization Presentation from RIAdventure (full video and slides) simpy:My Data Visualization Presentation from RIAdventure (full video and slides) newsvine:My Data Visualization Presentation from RIAdventure (full video and slides) blinklist:My Data Visualization Presentation from RIAdventure (full video and slides) furl:My Data Visualization Presentation from RIAdventure (full video and slides) reddit:My Data Visualization Presentation from RIAdventure (full video and slides) fark:My Data Visualization Presentation from RIAdventure (full video and slides) blogmarks:My Data Visualization Presentation from RIAdventure (full video and slides) Y!:My Data Visualization Presentation from RIAdventure (full video and slides) smarking:My Data Visualization Presentation from RIAdventure (full video and slides) magnolia:My Data Visualization Presentation from RIAdventure (full video and slides) segnalo:My Data Visualization Presentation from RIAdventure (full video and slides) gifttagging:My Data Visualization Presentation from RIAdventure (full video and slides)

One of the first time-series line charts ever drawn was a visualization of the great American credit crisis (but probably not the credit crisis that comes immediately to mind). If you were to look at this chart today you might even mistake it for the charts of the housing credit crisis of the past few years.

playfair_north_america_trade2
(see image credits below for image details)

This chart was created by William Playfair and published in 1786 in The Commercial and Political Atlas. In that work Playfair literally invented the line chart. This particular chart shows the imports and exports between Britain and America between 1700-1800. The red line is the line for exports from Britain to America, and the lighter yellow line is the imports from America. You can see the relationship of imports to exports stays relatively constant for the first 50 years (1700-1750) and then the exports start shooting up dramatically, at a rate much greater than the increase in imports.

Compare that with this chart of housing prices, created by the New York Times.
nyt_chart_cropped
(image from the New York Times)

I guess we know what a credit-driven catastrophe looks like. And it’s not only the image itself that looks similar, at times his words sound as if he’s writing today about our current financial mess.

Between 1750-1772 there was a rapid increase in exports from Britain to America. These exports were the result of many new merchants hoping to strike it big by shipping goods to the new settlers. But the reason things got out of control has to do with credit. Merchants started lending and borrowing on credit to finance their get-rich-quick schemes of selling stuff to America. Playfair writes (all emphasis added is mine),

Ever since the invention of paper credit, trade has had a latitude it did not before enjoy, and its progress being less natural, has become more intricate. That bound set and preserved by the nature of things was removed, when paper credit was first invented; previous to which, nothing represented wealth that was not wealth itself, or that was not physically worth the sum it represented; and in order to give credit in business, it was absolutely necessary either to possess, or to have borrowed capital.

And because of this new credit, people started making business decisions that were insane. They started shipping products to America before they knew they could sell them. Since the money was free they took irrational risks. And if your business venture failed miserably you could always just hide from your creditors in that new land of opportunity.

Of the eventual crash, Playfair writes,

For the first fifty years, we observe the simple and regular growth, from poverty to wealth, of a new country; during the succeeding twenty years, we are astonished at the extent and operation of a mad mercantile speculation carried on by our own country; and the period which succeeds, shews the catastrophe that so airy and so ill-founded a project was likely, sooner or later, to experience. There is not any branch of trade, which, from the nature of its progress, affords so much instruction as this. It merits equally the attention of the philosopher, the politician, and the merchant; for it throws light upon all the three different objects of their pursuits.

Isn’t that beautiful? Almost the same words could apply to the current financial crisis. And one final quote that I like, which also made me think of our current crisis:

Upon the manner in which business is conducted, depends something more than merely the gaining or losing a little money. The happiness of numbers of innocent individuals is frequently depending upon the success of projects, with the formation of which they had no concern. What numbers have been ruined, and how many more deprived of fortune, by our ill-conducted trade with America?

What numbers have been ruined indeed.

I’ve been reading the works of Playfair to understand the history of data visualization (in this same work he also invented the bar chart, and in a successive work he invented the pie chart). I wanted to make sure I understood the history of statistical charts, since as they say, those who cannot remember the past are condemned to repeat it. I didn’t realize that phrase would also apply so perfectly to the text accompanying the images.

* Image Credits
The first image above is from William Playfair’s Commercial and Political Atlas, 3rd edition, published in 1801. The scan is of a copy contained in the University of Pennsylvania’s Annenberg Rare Book and Manuscript Library. It was reproduced in a publication by Cambridge University Press entitled The Commercial and Political Atlas and Statistical Breviary, published in 2005, which was compiled by Howard Wainer and Ian Spence (and if you want to be even more technical the image above is a reproduction from a Google scan of the Cambridge University scan). As was decided in Bridgeman vs Corel Corp (full text), a reproduction of a work of art in the public domain is not protected by copyright. As was stated in that verdict: “While it may be assumed that this required both skill and effort, there was no spark of originality — indeed, the point of the exercise was to reproduce the underlying works with absolute fidelity. Copyright is not available in these circumstances.” I am reproducing the image here with that legal precedent in mind, and with the best of intentions. I would highly recommend that if you are interested in Playfair’s work you buy the reprint by Cambridge University Press. It contains full-color reproductions of the charts, and the introduction contains great biographic information about Playfair.
del.icio.us:The American Credit Crisis (of 1772) Visualized digg:The American Credit Crisis (of 1772) Visualized spurl:The American Credit Crisis (of 1772) Visualized wists:The American Credit Crisis (of 1772) Visualized simpy:The American Credit Crisis (of 1772) Visualized newsvine:The American Credit Crisis (of 1772) Visualized blinklist:The American Credit Crisis (of 1772) Visualized furl:The American Credit Crisis (of 1772) Visualized reddit:The American Credit Crisis (of 1772) Visualized fark:The American Credit Crisis (of 1772) Visualized blogmarks:The American Credit Crisis (of 1772) Visualized Y!:The American Credit Crisis (of 1772) Visualized smarking:The American Credit Crisis (of 1772) Visualized magnolia:The American Credit Crisis (of 1772) Visualized segnalo:The American Credit Crisis (of 1772) Visualized gifttagging:The American Credit Crisis (of 1772) Visualized

This is the time of year when people are making their resolutions, mapping out their big life changes for the coming year, and thinking ahead about what the future holds. I’m not one for resolutions, never have been. But I like the idea of taking stock of where you stand and thinking about changes you want to make.

And so to usher in 2010 I’ve decided to make a change: I’m removing “Adobe Flex Developer” from the title of my blog. The title used to read Doug McCune – Adobe Flex Developer. It will now read: Doug McCune – Data Visualization Engineer. Oooh, aaah.

This isn’t a change in anything other than my own mentality, but for me it is indeed significant. I’ll still be using largely the same tools, meaning I’ll be writing Flex applications and using ActionScript as my main language. I’m not changing jobs. My day-to-day work on Monday will be the same as my day-to-day work yesterday. This is a change in how I see myself, and most importantly how I see myself growing in the future.

Flex has been good to me, it provided a platform on which I could launch a career. Defining myself as a Flex Developer worked well and gave me enough room to grow. Within the small Flex community there were challenges, constantly new things to learn, and more than enough to keep me intellectually interested. The technology itself held my focus and through that focus I grew as a software developer.

But now that “Flex Developer” designation is a box that’s constricting rather than enabling. I gave a talk at 360|Flex in Indianapolis in which I talked about burning out. I explained that I had hit a point with the technology where the simply wasn’t enough left to keep me excited and passionate. As a side note, I love the fact that right on the first page of Google results for my name is an image of me standing in front of a slide that says “FUCK FLEX” :) How fitting.

fuck_flex

The work I’ve been doing on SpatialKey has developed my interest in the field of data visualization, which is a field with an amazingly rich history and a field I see rising in importance in the years ahead. Most importantly, it’s a field I don’t know much about. There is so much to learn, from new tools and languages (R, map-reduce, Processing) to historical visualization techniques (cartography alone has such an amazing history) to modern data never before available (the human genome is on Amazon AWS). The field as it is right now gets me pumped up, and it’s a field that will experience explosive growth in the coming decade.

So starting in 2010 I won’t be identifying myself as a Flex developer. Flex will still be the tool I choose to use for my work, but it isn’t part of my identity like it used to be. I’m a data visualization engineer. And I’m excited.

del.icio.us:My 2010 change: I'm not a Flex Developer digg:My 2010 change: I'm not a Flex Developer spurl:My 2010 change: I'm not a Flex Developer wists:My 2010 change: I'm not a Flex Developer simpy:My 2010 change: I'm not a Flex Developer newsvine:My 2010 change: I'm not a Flex Developer blinklist:My 2010 change: I'm not a Flex Developer furl:My 2010 change: I'm not a Flex Developer reddit:My 2010 change: I'm not a Flex Developer fark:My 2010 change: I'm not a Flex Developer blogmarks:My 2010 change: I'm not a Flex Developer Y!:My 2010 change: I'm not a Flex Developer smarking:My 2010 change: I'm not a Flex Developer magnolia:My 2010 change: I'm not a Flex Developer segnalo:My 2010 change: I'm not a Flex Developer gifttagging:My 2010 change: I'm not a Flex Developer


Images courtesy of the Image Science & Analysis Laboratory, NASA Johnson Space Center

As I was flying back home into San Francisco airport I was watching the city lights out the window and got struck by a bit of inspiration. I find cities beautiful, from the graffiti to the neon signs to the line of headlights on the highway. A city viewed from above at night is captivating. I wanted to try to recreate that same look, but by visualizing data (in one sense you can say that the real view of a city from above is already a visualization of population data).

I started searching for images of cities at night, and found these amazing images from NASA. All those images were taken from a space shuttle orbiting the earth. These images tell you a lot about the city, the layout, urban density, planning (or lack thereof). I wanted to take other meaningful data and create similar images.

All the visualizations below have been created with SpatialKey. However, this is some experimental work I’ve been playing with to generate the “night light” images, so it’s not released (and might not ever be). Basically this is a peak behind some of the R&D work I do for fun (yes, for a dataviz dork like me making fake “cities at night” images is my idea of fun).

Crime in San Francisco
This image is all crime in San Francisco for a 3-month period. You can see some of the same features that you can see in the NASA space image, such as Golgen Gate Park and the Presidio (the area on the north-west edge of the city). All in all it’s interesting how similar the crime image looks compared to the NASA image. Downtown is the brightest spot in both images, which means that it’s literally the brightest area of the city (the most streetlights), and also has the most crime.

SF_crime

And here are breakdowns for a few different crime types. Notice how different the distributions are. Narcotics crimes are heavily clustered and can be found downtown (in the Tenderloin), in the Mission (near the 16th St BART station), and along Haight Street near Golden Gate Park. Whereas vehicle theft is scattered fairly evenly throughout the city.

Narcotics
SF_narcotics
Theft
SF_theft
Vehicle Theft
SF_vehicle_theft
Burglary
SF_burglary

Graffiti Reports in San Francisco and New York
Both San Francisco and New York publish their 311 data, which is when citizens call for city services. One category of 311 calls is to report graffiti. Graffiti is interesting in that it often follows specific city streets. When we look at the graffiti data for both cities we see specific streets that have far more graffiti than others. I love these images (particularly the one of SF) because they really look like a view of street lights from a plane.

NYC_graffitiSF_311_graffiti

Trees planted in San Francisco
Another one of my favorites of this set is data for all the trees that the city of San Francisco has planted since 1990 (all this SF data is available at datasf.org). You can see the heavy planting along Market St (which cuts diagonally through downtown), as well as along streets like Sunset Blvd (the street running north/south on the western side of the city).

SF_trees

Street lights (or SF as a giant lite-brite)
One final image of San Francisco we have is the locations of every street light in the city. I liked this image because it reminded me of playing with a Lite-Brite when I was a kid. It almost makes city planning feel light a grown-up version of playing with little plastic lights.
SF_traffic_lights

del.icio.us:Data Visualized as City Lights at Night digg:Data Visualized as City Lights at Night spurl:Data Visualized as City Lights at Night wists:Data Visualized as City Lights at Night simpy:Data Visualized as City Lights at Night newsvine:Data Visualized as City Lights at Night blinklist:Data Visualized as City Lights at Night furl:Data Visualized as City Lights at Night reddit:Data Visualized as City Lights at Night fark:Data Visualized as City Lights at Night blogmarks:Data Visualized as City Lights at Night Y!:Data Visualized as City Lights at Night smarking:Data Visualized as City Lights at Night magnolia:Data Visualized as City Lights at Night segnalo:Data Visualized as City Lights at Night gifttagging:Data Visualized as City Lights at Night

Data can often tell you far more about people than you originally think. In my previous post I presented some of the data from the history of the FlexCoders mailing list. I showed some of the details of the individual usage patterns for different people. One of those people was the Flex product manager, Matt Chotin. Matt’s involvement with FlexCoders is pretty interesting if you start to dig into the data. In this post I’ll try to identify some changing trends in his usage patterns and we’ll see if we can do some detective work to figure out why his behavior changed.

A little background: Matt has been involved in Flex since basically forever. He was an engineer at Macromedia and is now the product manager for Flex. Matt has been quite prolific on flexcoders over the years (in the overall ranking he’s #3). So to start I was interested in his overall post volume on the list. Take a look at the timeline showing his posts per month and you’ll notice there’s a distinct drop-off:

flexcoders_timeline_chotin

Here’s a closeup of a period:

flexcoders_chotin_drop

See that big drop from April to May of 2006? Well in May Matt changed jobs to become the product manager of Flex. On his blog he noted:

So if you notice the number of flexcoders posts going down it’s because my brain will be slowly atrophying as I move away from the details of our vast offering.

And that’s exactly what happened.

Daily routines
Seeing the correlation between a change in professional life and a drop in activity is cool, but we can dig deeper. Not only is this data telling us when Matt changed his behavior throughout the year, but we can also figure out something about his daily routines and how that changed as well. I started looking at when (as in what time of day) Matt was posting to the list.

Here’s a chart that shows the distribution of posts by hour of day and day of week. It groups the posts by the combination of what day and what hour they occur on.

flexcoders_heatgrid_chotin

So you can see that Matt posted the most on weekday mornings (around 9-11am on Monday-Friday) and weekday evenings (around 8-10pm Monday-Thursday, note that he rarely posts on Friday nights).

This pattern is actually very similar to Alex Harui’s activity as well, although Alex’s activity is more weighted to during work hours than at night (except for Sunday night!).
flexcoders_harui_heatindex

I found the evening hotspots interesting (both in Matt and Alex’s cases). Clearly Matt was answering people’s questions a lot after work hours from home.

I dug a bit further into Matt’s trends. Here’s the graph of his activity by hour of day for 2005:

flexcoders_chotin_2005

We can see in 2005 he actually answered more questions in the evening than in the morning. Taking a look at 2006 this became even more pronounced, almost all his activity was at night (I wasn’t the only one who noticed this, see Ryan Stewart’s post about Matt posting at 9pm):

flexcoders_chotin_2006

And then there was a change in 2007. The graph for 2007 shows that he started answering more question during the workday. And that shift continued into 2008 and 2009, by which time almost all of Matt’s activity was during work hours.

flexcoders_chotin_2007flexcoders_chotin_2008flexcoders_chotin_2009

If you dig even deeper into the data you can find out that the transition from mainly evening activity to work-day activity happened mostly during the months of April 2007 – June 2007. After about July 2007 Matt almost primarily posts during the day. Taking a look at the release history of Flex, we see that the beta of Flex 3 came out in June 2007. So my guess is that Matt changed to a management role in May of 2006, but had far too much work to do to get Flex 3 ready and out the door between then and June 2007 (meaning his devotion to flexcoders had to be delegated to the evening hours). Finally once the Flex 3 beta was out the door he could devote some actual work hours to being involved in the community, instead of having to do it all from home.

Vacation Time
As if knowing the intimate details about Matt’s daily routine isn’t enough, we can learn something about his historical vacation time off as well. Matt’s impressive in that he’s never missed a month without posting. If you go even more granular there are actually very few weeks that he missed (as his overall activity declined in 2009 this became more common). So if we look at Matt’s activity around the holidays something interesting pops out (well, it’s only interesting if you’re a total stalker, but if you’ve read this far then you probably are). Here are a few timelines of different years, showing columns grouped by week. In 2005 we see Matt was posting pretty regularly through the holidays. There actually was a 5 day stretch with no posts, but that was it (due to the way the weeks are grouped that gap doesn’t show in this chart).
flexcoders_chotin_xmas2004

2005 is similar:
flexcoders_chotin_xmas2005

But then 2006 has a big gap:
flexcoders_chotin_xmas2006

And being the stalker that I am, I noticed that and then went to investigate further. Turns out Matt wrote about taking a vacation that year.

I’ll be on vacation until mid-January so emails to me will go unanswered as will responses to various forums and blog comments :-) Happy Holidays to all!

The data never lies.

Looks like a long vacation over the holidays didn’t turn into a regular thing though, since he was right back at it the following year:
flexcoders_chotin_xmas20071

I’m not a total nut job
I know it seems like I’m obsessed with Matt Chotin. And regardless of whether that’s true or not, I do want to assure people I’m not totally off my rocker. This little experiment in data mining and analysis isn’t really about Matt. It’s about the stories data tells about all of us. There are mountains of public information out there about us all, and the tiny little bits that we put out there, even if those are just little Facebook or Twitter status messages, can say a lot about us. Sure, a single Facebook status message doesn’t tell anyone much, but when you look at all of them over a multi-year period you can start learning a lot about a person. And often that information that the aggregate data tells about us isn’t something we’re aware of. From this data experiment I know when Matt eats dinner (pretty typical range of 6-8pm), when he goes to bed (around midnight), and when he gets to work (again pretty normal between 8-9). And this is all from only 4,000 data points. With social networking and microblogging sites we’re starting to create thousands of little data points like this all the time.

Thanks to Matt Chotin
I ran this post by Matt first, since I know it’s a bit creepy. He was cool with me posting it, so thanks Matt! And thanks for all the years of hard work answering questions on flexcoders, we’re a stronger community because of it.

The Data
Read more about the data here. This is 5 and a half years of mailing list activity, comprising about 148,826 individual email messages. Matt himself posted about 4,000 messages. You can download the full CSV dataset here.

del.icio.us:Stalking Someone with Data digg:Stalking Someone with Data spurl:Stalking Someone with Data wists:Stalking Someone with Data simpy:Stalking Someone with Data newsvine:Stalking Someone with Data blinklist:Stalking Someone with Data furl:Stalking Someone with Data reddit:Stalking Someone with Data fark:Stalking Someone with Data blogmarks:Stalking Someone with Data Y!:Stalking Someone with Data smarking:Stalking Someone with Data magnolia:Stalking Someone with Data segnalo:Stalking Someone with Data gifttagging:Stalking Someone with Data