Author Archives: Doug

Flex/Flash/Actionscript

Reflections on the first 360|Flex

The next 360|Flex conference is coming up in March in San Jose. I got to thinking about my history with 360|Flex and wanted to reflect on how important the conference has been for me. This post is about the very first 360|Flex conference that was in March 2007. It’s self-indulgent, but I hope it doesn’t come off as egotistical. My goal is to explain my personal experience and my path, not to brag about how cool I am.

I attended the first 360|Flex conference in San Jose simply as an attendee. I barely knew anyone (at least personally) in the Flex world. I had been attending the Flex user group meetings in San Francisco for just a bit before this conference and so I had met a few people very briefly like Tom Ortega (conference organizer) and Ted Patrick (Adobe evangelist). I had been blogging for literally only 3 months. I was still very inexperienced (even if I tried to mask that) and I wasn’t at all confident in myself (I really tried to mask that).

Opportunity #1:

The night before the first keynote of 360|Flex I got an email from Mark Anders from Adobe (who I had never met) telling me he was going to be showing a demo app of mine during the keynote. In the email he said:

“Make sure you attend the keynote, because I’m going to feature your RSS reader in it as a demo!  I hope that’s ok! 🙂  I’ll also be asking you to stand up and take a bow, so make sure you look great!  See you tomorrow!”

anders_rss_reader_360flexHope it’s OK? Fuck yeah, it’s awesome. Completely unexpected. I had butterflies in my stomach all night until the next morning, I was like a little schoolgirl. I got there early the next day, I wore a black sweater that I thought looked kind of professional in a dorky tech way, sort of Steve Jobs inspired. I realize now that I probably looked like a tool, but hey, it worked alright for me. Keynote came, the demo was shown, people seemed impressed and clapped and I did a little stand up wave kind of awkward thing. Super nervous, totally weird, but people saw my face. And after the keynote I met Mark and chatted with him and some fellow Adobe employees a bit (at the time I was seriously considering trying to get a job at Adobe, so I was sucking up pretty big time I think).

Opportunity #2:

On the second day of the conference Ted Patrick approached me and asked if I could put together a demo of FlexLib components to be shown at the keynote on the last day of the conference. I think this was in the afternoon and he was going to show it the next morning. I proceeded to go sit in the back of the last session of the day and crank out a little demo explorer app to showcase some of the components. To spice things up I threw in an extra water eyecandy effect that I was in the middle or working on. I threw the demo together fairly quickly and got it over to him for the next day. The demo was just a small part of his keynote, but during it he did get my name in there and point me out in the crowd, and people seemed to dig the water effect nonsense 🙂

So with a whole lot of luck I had managed to get my name mentioned in both the keynotes of 360|Flex. I’m grateful for these opportunities, they felt like they came out of nowhere and suddenly people were coming up to me to talk (as opposed to me sheepishly approaching people in the awkward tech-dork way).

Opportunity #3:

I remember sitting in the back of some session (can’t for the life of me remember anything about the session) and I happened to be sitting next to Tom Link and Brett Cortese from Universal Mind. I wasn’t at all tapped into the Flex consulting scene at that time, so I didn’t know the name Universal Mind. But I was sitting there in the back and Tom was reviewing some app that UM was working on. It was a sort of business dashboard app that had a slick UI that let you customize various charts. I can’t remember the exact conversation, but I remember looking over his shoulder and being impressed with the app and chatting about it a bit. He mentioned Darron Schall’s name at some point. I had been collaborating with Darron a bit on FlexLib and so I got online and started asking him about UM. Turns out he had only good things to say (and he apparently said good things about me to UM as well). After 360|Flex I left Tom with one of my ridiculous business cards. We ended up getting in touch shortly after and following up and the rest is history. I just passed my two year anniversary working for UM and I’ve had a hell of a good time.

Swing and miss

At the conference Ted Patrick introduced me to a guy from Yahoo. He was interested in hiring a Flex contractor for an internal project. If I recall correctly the basic plan was to have this Flex app that let Yahoo employees create different layouts and combinations of various Yahoo widgets, which would eventually turn into Yahoo web pages. It was sort of a prototyping tool for internal use. I had never done Flex consulting work before. I had never done any independent consulting whatsoever. I didn’t have the utmost confidence in my own abilities and I certainly didn’t feel comfortable asking for a lot of money. But I figured, fuck it, what’s the worst that can happen?

So with no prior consulting experience I quoted him $125 an hour. Pulled straight out of my ass. My logic was that I didn’t actually need the work yet (I was happily unemployed) but that if it came through and I was making over a hundred dollars an hour I’d would dive in head first (and be scared shitless in the process). I was far more scared of actually getting the gig than not, since then I’d have to actually be worth what I saw as such a ridiculous rate. I ended the email like this (parentheses are my thoughts, not in the actual email): “I’m happy to discuss the project further. I find the concept very interesting (I didn’t really), so even if I’m not involved in the development I’d be more than happy to give my opinions on the project (this was true, since I had no experience whatsoever I figured that any involvement would help me). Let me know how you want to proceed, and regardless let’s keep in touch about work that Yahoo is doing with Flex (having contacts inside companies like Yahoo can never hurt).” Send. And then I never even got a response. Ego bruised, but not badly.

The Jesse Warden validation

warden_thumbThis is funny because it’s a story I’ve never told Jesse (or anyone for that matter) and it showcases just how insecure I was at the time. I had been talking to Tom from UM about consulting and he had mentioned that Jesse Warden had been doing a bit of consulting for them as well, and that I should talk to him to see what UM was like. Jesse (who I had never met) was sitting against some wall working on a laptop and I walked over and said some awkward introduction like “Hey Jesse, it’s nice to meet you, I’m Doug McCune.” After being caught completely off guard with Jesse’s caffinated million-word-a-minute style of speech we got to talking a bit and then he said something like, “Where the hell did you come from? Like all the sudden here’s this Doug McCune guy coming out of nowhere making our computers blow up with this crazy fire and shit.” (I had recently released a fire effect demo). It’s funny how that interaction sticks so vividly in my mind. I probably laughed it off and said some kind of thanks, but inside I was beaming with pride. Jesse fucking Warden knew who I was and thought my shit was cool. That’s actually the exact moment that I knew that I was on some kind of path here, that this was actually going to work out.

This conference was huge for me

I had no expectations going into the conference, I figured I’d attend like I had attended various conferences before, sit in the back of sessions, take notes and write down links, and hopefully learn good material. I had no idea I’d get the kind of exposure in the keynotes that I ended up getting, or that I’d make the connections that I did. I ended up meeting Jesse, Deepa, Ben Stucki, Renaun Erickson, Mark Anders, Ely Greenfield, and countless other people. And these people that I met actually knew who I was. I couldn’t fucking believe it. My head was literally spinning at the end of it all.

Standard
Data Visualization

My Data Visualization Presentation from RIAdventure (full video and slides)

On the RIAdventure conference I gave a presentation about the past, present, and future of data visualization as I see it (fun side note: RIAdventure is the only conference I can say I “went on”). Luckily, the organizers filmed the entire thing, and we now have the video of the whole presentation that you can watch. This presentation covered a brief history of the field of data visualization, with the focus on the invention (in the not too distant past) of many data visualization techniques we take for granted. The point of the historical exercise was to point out that new opportunities with new data that we have before us present new opportunities for invention. I talked about new trends I see emerging in the data itself (massive datasets, city data, you life data, stream data) and what those trends mean for us as data visualization software engineers (I also argue that everyone will be a “data viz” engineer to some degree in the future).

I hope you enjoy the presentation, it was a lot of fun to create and to present. I learned a ton from the research and it was exciting thinking about the future of the field. Below is the full video (low resolution streaming from vimeo, or you can find higher resolution streaming form screencast here, or you can even download the full video file). Also embedded below are the slides that go along with the presentation, and you can always download the slides as a PDF.

Also check out some of the other presentations from RIAdventure.

Video:

Slides:

Standard
Data Visualization

The American Credit Crisis (of 1772) Visualized

One of the first time-series line charts ever drawn was a visualization of the great American credit crisis (but probably not the credit crisis that comes immediately to mind). If you were to look at this chart today you might even mistake it for the charts of the housing credit crisis of the past few years.

playfair_north_america_trade2
(see image credits below for image details)

This chart was created by William Playfair and published in 1786 in The Commercial and Political Atlas. In that work Playfair literally invented the line chart. This particular chart shows the imports and exports between Britain and America between 1700-1800. The red line is the line for exports from Britain to America, and the lighter yellow line is the imports from America. You can see the relationship of imports to exports stays relatively constant for the first 50 years (1700-1750) and then the exports start shooting up dramatically, at a rate much greater than the increase in imports.

Compare that with this chart of housing prices, created by the New York Times.
nyt_chart_cropped
(image from the New York Times)

I guess we know what a credit-driven catastrophe looks like. And it’s not only the image itself that looks similar, at times his words sound as if he’s writing today about our current financial mess.

Between 1750-1772 there was a rapid increase in exports from Britain to America. These exports were the result of many new merchants hoping to strike it big by shipping goods to the new settlers. But the reason things got out of control has to do with credit. Merchants started lending and borrowing on credit to finance their get-rich-quick schemes of selling stuff to America. Playfair writes (all emphasis added is mine),

Ever since the invention of paper credit, trade has had a latitude it did not before enjoy, and its progress being less natural, has become more intricate. That bound set and preserved by the nature of things was removed, when paper credit was first invented; previous to which, nothing represented wealth that was not wealth itself, or that was not physically worth the sum it represented; and in order to give credit in business, it was absolutely necessary either to possess, or to have borrowed capital.

And because of this new credit, people started making business decisions that were insane. They started shipping products to America before they knew they could sell them. Since the money was free they took irrational risks. And if your business venture failed miserably you could always just hide from your creditors in that new land of opportunity.

Of the eventual crash, Playfair writes,

For the first fifty years, we observe the simple and regular growth, from poverty to wealth, of a new country; during the succeeding twenty years, we are astonished at the extent and operation of a mad mercantile speculation carried on by our own country; and the period which succeeds, shews the catastrophe that so airy and so ill-founded a project was likely, sooner or later, to experience. There is not any branch of trade, which, from the nature of its progress, affords so much instruction as this. It merits equally the attention of the philosopher, the politician, and the merchant; for it throws light upon all the three different objects of their pursuits.

Isn’t that beautiful? Almost the same words could apply to the current financial crisis. And one final quote that I like, which also made me think of our current crisis:

Upon the manner in which business is conducted, depends something more than merely the gaining or losing a little money. The happiness of numbers of innocent individuals is frequently depending upon the success of projects, with the formation of which they had no concern. What numbers have been ruined, and how many more deprived of fortune, by our ill-conducted trade with America?

What numbers have been ruined indeed.

I’ve been reading the works of Playfair to understand the history of data visualization (in this same work he also invented the bar chart, and in a successive work he invented the pie chart). I wanted to make sure I understood the history of statistical charts, since as they say, those who cannot remember the past are condemned to repeat it. I didn’t realize that phrase would also apply so perfectly to the text accompanying the images.

* Image Credits
The first image above is from William Playfair’s Commercial and Political Atlas, 3rd edition, published in 1801. The scan is of a copy contained in the University of Pennsylvania’s Annenberg Rare Book and Manuscript Library. It was reproduced in a publication by Cambridge University Press entitled The Commercial and Political Atlas and Statistical Breviary, published in 2005, which was compiled by Howard Wainer and Ian Spence (and if you want to be even more technical the image above is a reproduction from a Google scan of the Cambridge University scan). As was decided in Bridgeman vs Corel Corp (full text), a reproduction of a work of art in the public domain is not protected by copyright. As was stated in that verdict: “While it may be assumed that this required both skill and effort, there was no spark of originality — indeed, the point of the exercise was to reproduce the underlying works with absolute fidelity. Copyright is not available in these circumstances.” I am reproducing the image here with that legal precedent in mind, and with the best of intentions. I would highly recommend that if you are interested in Playfair’s work you buy the reprint by Cambridge University Press. It contains full-color reproductions of the charts, and the introduction contains great biographic information about Playfair.
Standard
Uncategorized

My 2010 change: I’m not a Flex Developer

This is the time of year when people are making their resolutions, mapping out their big life changes for the coming year, and thinking ahead about what the future holds. I’m not one for resolutions, never have been. But I like the idea of taking stock of where you stand and thinking about changes you want to make.

And so to usher in 2010 I’ve decided to make a change: I’m removing “Adobe Flex Developer” from the title of my blog. The title used to read Doug McCune – Adobe Flex Developer. It will now read: Doug McCune – Data Visualization Engineer. Oooh, aaah.

This isn’t a change in anything other than my own mentality, but for me it is indeed significant. I’ll still be using largely the same tools, meaning I’ll be writing Flex applications and using ActionScript as my main language. I’m not changing jobs. My day-to-day work on Monday will be the same as my day-to-day work yesterday. This is a change in how I see myself, and most importantly how I see myself growing in the future.

Flex has been good to me, it provided a platform on which I could launch a career. Defining myself as a Flex Developer worked well and gave me enough room to grow. Within the small Flex community there were challenges, constantly new things to learn, and more than enough to keep me intellectually interested. The technology itself held my focus and through that focus I grew as a software developer.

But now that “Flex Developer” designation is a box that’s constricting rather than enabling. I gave a talk at 360|Flex in Indianapolis in which I talked about burning out. I explained that I had hit a point with the technology where the simply wasn’t enough left to keep me excited and passionate. As a side note, I love the fact that right on the first page of Google results for my name is an image of me standing in front of a slide that says “FUCK FLEX” 🙂 How fitting.

fuck_flex

The work I’ve been doing on SpatialKey has developed my interest in the field of data visualization, which is a field with an amazingly rich history and a field I see rising in importance in the years ahead. Most importantly, it’s a field I don’t know much about. There is so much to learn, from new tools and languages (R, map-reduce, Processing) to historical visualization techniques (cartography alone has such an amazing history) to modern data never before available (the human genome is on Amazon AWS). The field as it is right now gets me pumped up, and it’s a field that will experience explosive growth in the coming decade.

So starting in 2010 I won’t be identifying myself as a Flex developer. Flex will still be the tool I choose to use for my work, but it isn’t part of my identity like it used to be. I’m a data visualization engineer. And I’m excited.

Standard
Art, Data Visualization, Maps, SpatialKey

Data Visualized as City Lights at Night



Images courtesy of the Image Science & Analysis Laboratory, NASA Johnson Space Center

As I was flying back home into San Francisco airport I was watching the city lights out the window and got struck by a bit of inspiration. I find cities beautiful, from the graffiti to the neon signs to the line of headlights on the highway. A city viewed from above at night is captivating. I wanted to try to recreate that same look, but by visualizing data (in one sense you can say that the real view of a city from above is already a visualization of population data).

I started searching for images of cities at night, and found these amazing images from NASA. All those images were taken from a space shuttle orbiting the earth. These images tell you a lot about the city, the layout, urban density, planning (or lack thereof). I wanted to take other meaningful data and create similar images.

All the visualizations below have been created with SpatialKey. However, this is some experimental work I’ve been playing with to generate the “night light” images, so it’s not released (and might not ever be). Basically this is a peak behind some of the R&D work I do for fun (yes, for a dataviz dork like me making fake “cities at night” images is my idea of fun).

Crime in San Francisco

This image is all crime in San Francisco for a 3-month period. You can see some of the same features that you can see in the NASA space image, such as Golgen Gate Park and the Presidio (the area on the north-west edge of the city). All in all it’s interesting how similar the crime image looks compared to the NASA image. Downtown is the brightest spot in both images, which means that it’s literally the brightest area of the city (the most streetlights), and also has the most crime.

SF_crime

And here are breakdowns for a few different crime types. Notice how different the distributions are. Narcotics crimes are heavily clustered and can be found downtown (in the Tenderloin), in the Mission (near the 16th St BART station), and along Haight Street near Golden Gate Park. Whereas vehicle theft is scattered fairly evenly throughout the city.


Graffiti Reports in San Francisco and New York

Both San Francisco and New York publish their 311 data, which is when citizens call for city services. One category of 311 calls is to report graffiti. Graffiti is interesting in that it often follows specific city streets. When we look at the graffiti data for both cities we see specific streets that have far more graffiti than others. I love these images (particularly the one of SF) because they really look like a view of street lights from a plane.

NYC_graffitiSF_311_graffiti

Trees planted in San Francisco

Another one of my favorites of this set is data for all the trees that the city of San Francisco has planted since 1990 (all this SF data is available at datasf.org). You can see the heavy planting along Market St (which cuts diagonally through downtown), as well as along streets like Sunset Blvd (the street running north/south on the western side of the city).

SF_trees

Street lights (or SF as a giant lite-brite)

One final image of San Francisco we have is the locations of every street light in the city. I liked this image because it reminded me of playing with a Lite-Brite when I was a kid. It almost makes city planning feel light a grown-up version of playing with little plastic lights.

SF_traffic_lights

Standard
Data Visualization, Flex/Flash/Actionscript

Stalking Someone with Data

Data can often tell you far more about people than you originally think. In my previous post I presented some of the data from the history of the FlexCoders mailing list. I showed some of the details of the individual usage patterns for different people. One of those people was the Flex product manager, Matt Chotin. Matt’s involvement with FlexCoders is pretty interesting if you start to dig into the data. In this post I’ll try to identify some changing trends in his usage patterns and we’ll see if we can do some detective work to figure out why his behavior changed.

A little background: Matt has been involved in Flex since basically forever. He was an engineer at Macromedia and is now the product manager for Flex. Matt has been quite prolific on flexcoders over the years (in the overall ranking he’s #3). So to start I was interested in his overall post volume on the list. Take a look at the timeline showing his posts per month and you’ll notice there’s a distinct drop-off:

flexcoders_timeline_chotin

Here’s a closeup of a period:

flexcoders_chotin_drop

See that big drop from April to May of 2006? Well in May Matt changed jobs to become the product manager of Flex. On his blog he noted:

So if you notice the number of flexcoders posts going down it’s because my brain will be slowly atrophying as I move away from the details of our vast offering.

And that’s exactly what happened.

Daily routines

Seeing the correlation between a change in professional life and a drop in activity is cool, but we can dig deeper. Not only is this data telling us when Matt changed his behavior throughout the year, but we can also figure out something about his daily routines and how that changed as well. I started looking at when (as in what time of day) Matt was posting to the list.

Here’s a chart that shows the distribution of posts by hour of day and day of week. It groups the posts by the combination of what day and what hour they occur on.

flexcoders_heatgrid_chotin

So you can see that Matt posted the most on weekday mornings (around 9-11am on Monday-Friday) and weekday evenings (around 8-10pm Monday-Thursday, note that he rarely posts on Friday nights).

This pattern is actually very similar to Alex Harui’s activity as well, although Alex’s activity is more weighted to during work hours than at night (except for Sunday night!).
flexcoders_harui_heatindex

I found the evening hotspots interesting (both in Matt and Alex’s cases). Clearly Matt was answering people’s questions a lot after work hours from home.

I dug a bit further into Matt’s trends. Here’s the graph of his activity by hour of day for 2005:

flexcoders_chotin_2005

We can see in 2005 he actually answered more questions in the evening than in the morning. Taking a look at 2006 this became even more pronounced, almost all his activity was at night (I wasn’t the only one who noticed this, see Ryan Stewart’s post about Matt posting at 9pm):

flexcoders_chotin_2006

And then there was a change in 2007. The graph for 2007 shows that he started answering more question during the workday. And that shift continued into 2008 and 2009, by which time almost all of Matt’s activity was during work hours.

flexcoders_chotin_2007flexcoders_chotin_2008flexcoders_chotin_2009

If you dig even deeper into the data you can find out that the transition from mainly evening activity to work-day activity happened mostly during the months of April 2007 – June 2007. After about July 2007 Matt almost primarily posts during the day. Taking a look at the release history of Flex, we see that the beta of Flex 3 came out in June 2007. So my guess is that Matt changed to a management role in May of 2006, but had far too much work to do to get Flex 3 ready and out the door between then and June 2007 (meaning his devotion to flexcoders had to be delegated to the evening hours). Finally once the Flex 3 beta was out the door he could devote some actual work hours to being involved in the community, instead of having to do it all from home.

Vacation Time

As if knowing the intimate details about Matt’s daily routine isn’t enough, we can learn something about his historical vacation time off as well. Matt’s impressive in that he’s never missed a month without posting. If you go even more granular there are actually very few weeks that he missed (as his overall activity declined in 2009 this became more common). So if we look at Matt’s activity around the holidays something interesting pops out (well, it’s only interesting if you’re a total stalker, but if you’ve read this far then you probably are). Here are a few timelines of different years, showing columns grouped by week. In 2005 we see Matt was posting pretty regularly through the holidays. There actually was a 5 day stretch with no posts, but that was it (due to the way the weeks are grouped that gap doesn’t show in this chart).
flexcoders_chotin_xmas2004

2005 is similar:
flexcoders_chotin_xmas2005

But then 2006 has a big gap:
flexcoders_chotin_xmas2006

And being the stalker that I am, I noticed that and then went to investigate further. Turns out Matt wrote about taking a vacation that year.

I’ll be on vacation until mid-January so emails to me will go unanswered as will responses to various forums and blog comments 🙂 Happy Holidays to all!

The data never lies.

Looks like a long vacation over the holidays didn’t turn into a regular thing though, since he was right back at it the following year:
flexcoders_chotin_xmas20071

I’m not a total nut job

I know it seems like I’m obsessed with Matt Chotin. And regardless of whether that’s true or not, I do want to assure people I’m not totally off my rocker. This little experiment in data mining and analysis isn’t really about Matt. It’s about the stories data tells about all of us. There are mountains of public information out there about us all, and the tiny little bits that we put out there, even if those are just little Facebook or Twitter status messages, can say a lot about us. Sure, a single Facebook status message doesn’t tell anyone much, but when you look at all of them over a multi-year period you can start learning a lot about a person. And often that information that the aggregate data tells about us isn’t something we’re aware of. From this data experiment I know when Matt eats dinner (pretty typical range of 6-8pm), when he goes to bed (around midnight), and when he gets to work (again pretty normal between 8-9). And this is all from only 4,000 data points. With social networking and microblogging sites we’re starting to create thousands of little data points like this all the time.

Thanks to Matt Chotin

I ran this post by Matt first, since I know it’s a bit creepy. He was cool with me posting it, so thanks Matt! And thanks for all the years of hard work answering questions on flexcoders, we’re a stronger community because of it.

The Data

Read more about the data here. This is 5 and a half years of mailing list activity, comprising about 148,826 individual email messages. Matt himself posted about 4,000 messages. You can download the full CSV dataset here.

Standard
Flex/Flash/Actionscript

FlexCoders Mailing List Stats, Pretty Graphs, Full Dataset

In this post I’m going to dive into details about the stats of the FlexCoders mailing list usage over the past 5 and a half years. It’s full of graphs of various fun statistics, like who’s most active on the list, when people post, and the overall traffic over time. It’s a bit of a trip down memory lane, and I apologize if I ramble, I like data and pretty pictures, and I have a soft spot in my heart for FlexCoders, so bear with me and hopefully for those of you on the mailing list it will be a fun trip.

Background

I’ve been on the FlexCoders mailing for a few years now (my first post was back in September 2006). As the Flex community grew, the list grew, some would say it grew to unmanageable levels. It’s certainly a lot of mail, I currently have 22,100 unread flexcoders emails in GMail. At one point we even debated furiously whether the list should be split up into multiple more focused lists, or if the whole thing was going to die. Regardless of that outcome the flexcoders list remained as it has been for years. One thing did change though: Adobe replaced their official forum (which was literally God’s worst forum software) with a new one. And the Adobe employees definitely seemed to be pushing people there, which isn’t to say they stopped answering flexcoders questions, but the community was certainly now split between two lists.

I subscribe to both flexcoders and the Adobe Flex forums (which you can setup to receive emails from). I started noticing a trend. Take a look at this picture of my inbox (only flexcoders and Adobe forums emails) as of right now:

flexcoders_flexforum

The orange label is used to tag the posts from flexcoders and the green label is posts from the Adobe forums. I started noticing that the number of posts from the forums were more than on flexcoders. That obviously made me wonder if the overall traffic on flexcoders was in decline. I’ve been inactive on the list for quite a while (been quiet for most of 2009). So I didn’t exactly have my finger on the pulse of flexcoders.

The Data

So I wanted to download the entire Yahoo group dataset to start playing with it. Turns out Yahoo doesn’t make this easy, but I found a sweet program called PG Offline that I used to pull down the entire list. It took me a few days to get all 148,826 messages (as of about 7pm tonight). But PG Offline worked incredibly well and I then had an Access database file with all the emails (it was about 1.5 gigs). I then used another program called MDB Converter to convert that to a text CSV file.

If you want to play with the data yourself you can download the CSV file (11 megs). It includes columns for the sender, date, and subject. I did not include the full-text of the emails, since that would make it a gig and a half.

Analyzing the Trends

I pulled the data into SpatialKey (which is what I work on for my day job) and started digging into the data. Here’s the report setup I created in SpatialKey to play around and filter down the data (click for a larger view):

flexcoders_report_sk

So we can start seeing the overall trend in the main timeline, which shows the rise and fall in traffic.
flexcoders_timeline

So there certainly has been a decline in traffic to the list. The most active month ever in the list’s history was March 2008 with 3,834 posts. And then it’s been a fairly steady decline since that peak.

Some other interesting high level stats are the hours of the most activity. This chart shows the number of posts by hour of day. Hour of day is Pacific time.

flexcoders_hourofday

You can clearly see the work-day hours there. 8, 9, and 10am are the most active, and then it slows down as the work-day finishes up (earlier for east coast people), and then there’s another small bump around 9pm in the evening.

Who’s Most Active?

Anyone who reads FlexCoders knows that Alex Harui (from Adobe) is the king. Here are two charts showing the top 10 posters of all time and the top 10 from just 2009.

flexcoders_top10_alltimeflexcoders_top10_2009

Alex certainly still holds the number one spot overall, but Tracy has him beat for this past year.

Diving into Individual Activity

It’s also pretty interesting to look at how different individuals use flexcoders, and how their usage has changed. Here are just a few selected people that I was curious about:

Alex wasn’t always the king. He had a few messages back in 2005, but his heavy involvement on the list actually started relatively late, in March of 2007 (which is also when he started blogging coincidentally).
flexcoders_timeline_harui

Tracy Spratt, on the other hand, has been on the list since its very beginning:
flexcoders_timeline_spratt

Matt Chotin (Flex product manager) has also been active since the list started:
flexcoders_timeline_chotin

Actually, Matt Chotin and Tracy Spratt are the only two people who have posted to the list at least once every single month since the very beginning (from April 2004 to now). They get the FlexCoders Lifetime Achievement Award!

Some people were around in the early days but then dropped out. Here’s Jesse Warden‘s activity:
flexcoders_timeline_warden

Some people get sucked into the list fast and then fizzle out. Josh McDonald was the third most prolific poster of 2008, but then stopped posting as quickly as he started:
flexcoders_timeline_mcdonald

And some people stop posting when it’s no longer part of their job, like Roger Gonzalez who worked for Adobe and left in March 2007 (which was also the last time he posted to the list):
flexcoders_timeline_gonzalez

Ely Greenfield (Principal Architect at Adobe working on Flex 4) used to be fairly active back in 2006/2007, but hasn’t said a word in the past two years:
flexcoders_timeline_greenfield

And what about me? I was fairly active on the list from about 2007 through the beginning of 2009, then pretty much radio silence:
flexcoders_timeline_mccune

And some people don’t live in the USA and post at completely different times. Here’s Tom Chiverton‘s (4th most prolific poster of all time) usage pattern by hour of day and day of week. It groups the posts by the combination of what day and what hour they occur on.
flexcoders_heatgrid_chiverton

At first glance it looks like Tom emails the list in the middle of the night, until you realize that he lives in England 🙂

I’ve had a lot of fun drilling into the history of this list. It’s really cool what kinds of trends you can find (probably another post in more detail on that later).

Want to play with the data?

You can download the complete CSV file and use it if you want. I’d love to see people turn it into much more interesting visualizations. This dataset goes up until November 2 2009. Since it’s a bit of a pain to keep it updated I probably won’t update it very often, but if there is interest I might do so once a month or so.

Notes on privacy

All this data is public, you can see it all by going to the Yahoo group and searching. There are no email addresses in this data (unless perhaps if someone used their email address as their name as well). Any names in this data are there because the person knowingly emailed the public flexcoders email list. This CSV download is obviously a much easier format to work with all the data, and it can certainly be mined for interesting trends. I just ask that people play nice with the data. We’re a community, and this is data that represents our lives (or at least one small sliver of our lives) for the past 5 years.

Standard
Data Visualization, Maps

Dorky Data Visualization Pumpkin: Minard’s Graph of Napoleon’s March

Happy Halloween! This is the dorkiest pumpkin I’ve ever carved. For those of you into data visualization or mapping, maybe you can recognize it:

napoleon_pumpkin_1

This is a pumpkin representation of Charle’s Minard‘s visualization of Napoleon’s march into Russia in 1812. This graphic is considered by some (ie Edward Tufte) to be the “best statistical graphic ever drawn.” The graph shows the size of Napoleon’s army as they marched to and from Moscow. You can see how the army shrank as they approached Moscow. Once they reached Moscow they found the city had been abandoned and burned. Then they marched back home, except it was through a brutal Russian winter and nearly killed the remaining army. By the time they return home you can see the size of the army is just a small trickle.

minard_small

Beyond just the two charts of the march to and from Moscow, the graphic also serves as a map, with the paths indicating where the troops were geographically. And below the map is a temperature chart that visualizes how severe the winter weather was, which correlates with some of the major drops in troops on the way home.

The carved pumpkin ended up being very hard to take a photo of because the graph wraps around over half the pumpkin’s circumference. So I tried to take a few pictures to get the different sides. I carved the march to and from Moscow, as well as the temperature chart along the bottom.

napoleon_pumpkin_2

napoleon_pumpkin_3

Hope everyone has a great Halloween tonight!

Standard
Maps, SpatialKey

New SpatialKey Crime Example for San Francisco

We just posted a new example of using SpatialKey to visualize crime in San Francisco. We load in 90 days of crime data from the city, then filter down to only include sales of heroin, crack cocaine, and methamphetamine within 1,000 feet of a school. Why those particular crimes around schools? The SFPD just launched a new initiative called “Operation Safe Schools” that specifically targets these drug crimes. If you’re caught dealing crack, heroin, or meth around a school while the school is in session you can get extra prison time.

Check out the video below and read the full article on the SpatialKey blog.

Read the whole article on the SpatialKey blog to see how we put this together and learn more about the SFPD’s “Operation Safe Schools.” You can also watch the full resolution video on YouTube

Standard
Uncategorized

UX is making me dumb

User Experience Design (or UX for short) has exploded on the software scene carrying a blazing torch of freedom and promising to guide us to the holy land. We are now dedicating time and resources specifically to user experience design. This emphasis on actually designing user interactions and the experience of our applications is a fantastic effort and I applaud all those that wave the UX flag and sing its praises. But I have one problem: UX is making me a worse software developer.

An unfortunate byproduct of the UX revolution is the misinformation that developers simply can’t design good experiences. We see article after article explaining what happens when you leave developers in charge of UX design. People laugh at how stupid developers can be, that they just don’t “get it” when it comes to designing things well. These articles should highlight the importance of explicit thought when designing interactions in applications, they should not strip developers of their confidence and creativity. And yet that’s exactly what I’ve noticed happening to me.

Let me relate a brief hypothetical example. On our software project I am the lead clientside developer. We have a UX Specialist (aka UX Guy) that is responsible for the design of wireframes and mockups that explain how the user navigates through the application. During the UX pass, our UX Guy listens to us (developers and business guys) explain what the problem is and what we think we want the software to do. Then UX Guy goes off and comes back with a set of wireframes for how the application will work. We discuss, bring up problems, send back the wireframes for further iteration. Repeat that process until everyone is satisfied with the wireframes, then move on to actual development of features.

But then during development I have found myself getting the following question from my manager: “Hey Doug, how come you can’t do Feature X in the app?” and I have found myself actually saying this (shudder): “Oh, I guess that wasn’t included in UX Guy’s wireframes, you should ask him.” That’s a bullshit response. I’m not a neutered incompetent mindless developer. If something doesn’t work or was forgotten, I can figure out how to fix it. That doesn’t mean throw the code in without thinking about design. That means pausing my code writing, thinking through the problem carefully, designing a solution that will be sleek and elegant, and then continuing on with development.

And yet in my mind I have the unconscious prejudice that I, as a developer, cannot be allowed to make “UX” decisions. Fuck that. We are all responsible for designing the experience of our software. We are all responsible for carefully thinking through every interaction. We are all responsible for making things beautiful, making things simple, making things elegant. And more than that, we are all capable.

Standard