Inspired by this fantastic piece of work by Eric Testroete, my friend and I created my very own paper craft giant head (my buddy Todd did all the hard work of the 3D modeling and texturing). Of course, once you have such an amazing giant head, you need to figure out what the hell to do with it. And so boredom on a Friday night plus a few beers plus a giant head plus a roomba equals a magical vacuuming head!

A few more pictures of the head in action:

(*not my baby)

As I figure out more shenanigans to get into with my giant head I’m sure I’ll post more ridiculous photos and videos.

And so to usher in 2010 I’ve decided to make a change: I’m removing “Adobe Flex Developer” from the title of my blog. The title used to read Doug McCune – Adobe Flex Developer. It will now read: Doug McCune – Data Visualization Engineer. Oooh, aaah.

This isn’t a change in anything other than my own mentality, but for me it is indeed significant. I’ll still be using largely the same tools, meaning I’ll be writing Flex applications and using ActionScript as my main language. I’m not changing jobs. My day-to-day work on Monday will be the same as my day-to-day work yesterday. This is a change in how I see myself, and most importantly how I see myself growing in the future.

Flex has been good to me, it provided a platform on which I could launch a career. Defining myself as a Flex Developer worked well and gave me enough room to grow. Within the small Flex community there were challenges, constantly new things to learn, and more than enough to keep me intellectually interested. The technology itself held my focus and through that focus I grew as a software developer.

But now that “Flex Developer” designation is a box that’s constricting rather than enabling. I gave a talk at 360|Flex in Indianapolis in which I talked about burning out. I explained that I had hit a point with the technology where the simply wasn’t enough left to keep me excited and passionate. As a side note, I love the fact that right on the first page of Google results for my name is an image of me standing in front of a slide that says “FUCK FLEX” How fitting.

The work I’ve been doing on SpatialKey has developed my interest in the field of data visualization, which is a field with an amazingly rich history and a field I see rising in importance in the years ahead. Most importantly, it’s a field I don’t know much about. There is so much to learn, from new tools and languages (R, map-reduce, Processing) to historical visualization techniques (cartography alone has such an amazing history) to modern data never before available (the human genome is on Amazon AWS). The field as it is right now gets me pumped up, and it’s a field that will experience explosive growth in the coming decade.

So starting in 2010 I won’t be identifying myself as a Flex developer. Flex will still be the tool I choose to use for my work, but it isn’t part of my identity like it used to be. I’m a data visualization engineer. And I’m excited.

This is a pumpkin representation of Charle’s Minard‘s visualization of Napoleon’s march into Russia in 1812. This graphic is considered by some (ie Edward Tufte) to be the “best statistical graphic ever drawn.” The graph shows the size of Napoleon’s army as they marched to and from Moscow. You can see how the army shrank as they approached Moscow. Once they reached Moscow they found the city had been abandoned and burned. Then they marched back home, except it was through a brutal Russian winter and nearly killed the remaining army. By the time they return home you can see the size of the army is just a small trickle.

Beyond just the two charts of the march to and from Moscow, the graphic also serves as a map, with the paths indicating where the troops were geographically. And below the map is a temperature chart that visualizes how severe the winter weather was, which correlates with some of the major drops in troops on the way home.

The carved pumpkin ended up being very hard to take a photo of because the graph wraps around over half the pumpkin’s circumference. So I tried to take a few pictures to get the different sides. I carved the march to and from Moscow, as well as the temperature chart along the bottom.

Hope everyone has a great Halloween tonight!

You can also download the PDF of the slides if you want.

Adobe Open Source Media Framework + FlexLingo
This demo uses Adobe’s OSMF project to synchronize subtitles with video. This was the demo that I used to create the Hitler Goes to CFUnited video I posted before. You can seek around the video and OSMF makes sure to show the right subtitles. Then you can click on the different flag icons in the demo to use the FlexLingo library to translate the subtitle text to other languages (using Google translation API).

View the source
This demo is made with Flex 4, so to use the source code you’ll need to be able to build a Flex 4 project.

Axiis + Tweetr
The second demo I showed used the Axiis framework to create a chart of trending Twitter topics that have associated pictures on TwitPic. Unfortunately there’s no crossdomain file in place for TwitPic images, so I can’t run a live version of this app (but running locally works fine).

View the source
This demo is also made with Flex 4, so to use the source code you’ll need to be able to build a Flex 4 project.

Safe Sexting
The third demo I showed was the “Safe Sexting” app that uses the Marilena library for facial detection and uses Pixel Bender to automatically blur out your face. This demo was also shown in 360|Flex Indianapolis. You can read a bit more about this demo in a previous post about the 360|Flex presentation.

Run the demo | View the source

I’ve put together a spreadsheet that includes each class in the Flex framework, including charting and the advanced data visualization package. This shows the number of lines of code in each file, breaking down the total number of blank lines, comment lines, and actual code lines. I’ve colored the rows to indicate if the class belongs to the advanced data visualization package of Flex 3 (yellow), the charting framework (blue), the AIR SDK (red), or the open source Flex SDK (white).

Here are the top 20 classes, ordered by actual lines of code (not counting comments and whitespace):

To get these numbers, I used a free program called cloc to analyze the source code. Cloc is a perl script that analyzes a bunch of different languages, including ActionScript.

For the full list, check out the spreadsheet. This is for version 3.0.0 of the Flex SDK, it does not include anything released in SDK versions 3.0.2 or 3.2 or any of the new Flex 4 classes.

So just look at that list of top classes. Out of the top five, three of them are the new classes for the AdvancedDataGrid. It’s also worth nothing that AdvancedListBase and AdvancedDataGridBaseEx are not used for anything other than base classes for AdvancedDataGrid. If you look at the inheritance chain for AdvancedDataGrid, it looks like this:

UIComponent
ScrollControlBase
AdvancedListBase (4,553 lines of code, 8,573 total)
AdvancedDataGridBase (1,184 lines of code, 2,084 total)
AdvancedDataGridBaseEx (4,503 lines of code, 7,428 total)
AdvancedDataGrid (5,385 lines of code, 8,432 total)

That means that the entire codebase for the AdvancedDataGrid is 15,605 lines of code (26,517 if you include comments and whitespace). That’s 7% of the entire Flex codebase in those 4 classes. Note that this is not including ScrollControlBase or UIComponent, only the new classes in Flex 3. Also note that AdvancedDataGrid doesn’t inherit from ListBase or DataGrid at all, and ListBase is the second largest class in the entire framework.

So this means that when I run into a bug in the AdvancedDataGrid (which has happened a lot), I’ve got about 30,000 lines of code to wade through to try to debug my problem. No thank you.

The other two out of the top five are ListBase (which is why I said the entirety of the list classes is too big) and UIComponent (which I think is understandable, and I haven’t run into too many problems that have root causes in UIComponent, unlike ListBase which continues to be a big headache for me). Then number 6 on the list is DataGrid (oh, but don’t forget about #12 on the list, DataGridBase). If you group all list-based classes together (meaning the advanced classes and the original list/datagrid/tree classes), you’ve got 9 out of the top 20.

Code quality
This post is only about raw numbers. There’s an entirely different topic of code quality that I can write post after post about when it comes to the quality of the AdvancedDataGrid (and related classes). They use noticeably different coding conventions, they deviate dramatically from the best practices employed by the other components, and they are plagued by performance problems. But that discussion is for another post. For now I just wanted to point out the shear massive volume of code that you’re using when you use these classes.

Am I being too harsh?
Some would argue that the classes have to be big, given that they do so much. After all, it is called the AdvancedDataGrid, it’s doing crazy advanced stuff. I call bullshit on that argument. In my opinion, I just don’t think there’s an excuse for such massive classes. They make developers lives miserable when trying to debug, but I also think it is indicative of a sloppy coding style. The mentality sort of seems to be “throw more code at the problem until you get it to work.” Check out the expandItem method of AdvancedDataGrid. It’s 445 lines long – this single method is longer than 80% of all the other Flex classes in the framework (yes, I checked how many classes are under 445 total lines). That method even includes a spectacular commented out block of code with this comment:

// Dont know why this is done
// So, commenting it and using the rowHeight instead

Awesome. I guess as long as it passes the unit tests go ahead and ship it

Quick demo and the code
Since I know people don’t like reading, I’ll post the code at the beginning of this post and show some screenshots and a demo AIR app that uses the library. If you want to tune out after this, it’s cool, I understand.
Download as3svnlib source code, or just the compiled swc file.

Demo
Since this uses socket communication on port 80, which is off limits under normal circumstances due to Flash player security restrictions, you have to run the demo as an AIR app. Lame, I know. You can install this tiny AIR app to try out the library if you want. The demo app will let you enter the URL of a repository and load the last few revisions. The repository must be publicly accessible for the demo to work (note that the library itself allows you to use basic authentication, but I didn’t build support for that into the simple demo).

The screenshot below shows the bare-bones example AIR app. You can enter a URL for a SVN repo and then you can load the latest revision number or you can load the last 50 revisions. That’s all you can do in the app, but with the library you can get more detailed information about all the revisions and you can decide exactly which revisions you want to load (or load all of them).

You can download the example AIR app if you want to see if it works with your SVN repo (again, note that this demo app won’t work with URLs behind authentication, but read below about how to do that). View-source is enabled on the AIR app itself once you install it, or you can view the source of the example here.

The scope of this library
My immediate goal was to load revision history, complete with commiter name, date, comment, and ideally more detailed information about which files were affected or scope of the changes. I was not looking to build a full SVN client capable of committing changes or doing diffs or anything fancy like that.

How I went about it
I first started thinking I would implement a legit SVN client that would connect to a server using a socket connection and use the SVN protocol to communicate. I read up on the protocol and it didn’t seem all that complex, especially if I was going to limit myself to read-only operations. However, when I first started down this road I realized I didn’t know shit about doing socket communication in ActionScript. So I downloaded as3httpclientlib (current source is at github here), which is an AS3 library for doing HTTP communication written by Gabriel Handford. My goal was to use that project to learn how to do my own socket stuff and then build my SVN library.

But once I got my head around the http client stuff I learned something even more important. Turns out that most SVN installations include an Apache server that lets you browse the repo with a web interface. What’s even more interesting is that Subversion uses the WebDAV protocol as well, which is a protocol based on HTTP for source control. Suddenly I had a new option that didn’t require writing my own full-on SVN client. Fuck yeah! Less work for me!

So I extended as3httpclientlib and added in some basic support for WebDav (namely the PROPFIND request and a basic REPORT request). Then I added in support for some custom reports that Subversion uses, namely log-report. All the reports that Subversion supports via WebDav are documented here. When you request a report you send a custom HTTP request to the server and fill the request body with some XML that defines what kind of report you want. Then the server sends back its response in XML.

Then I was able to use simple XML parsing to read my list of revisions, which included revision number, creator name, date, comment, and details about all the files that were added, removed, modified, etc. Easy breezy.

Most basic use of the library
To use the library you interact with the SVNClient class. This class is used to fire off all your requests (either to get the latest revision number or to get specific sets of revisions). All requests are asynchronous, so you need to add an event listener to the SVNClient class, then send off your request, and then handle the result.

<?xml version="1.0" encoding="utf-8"?>
<mx:WindowedApplication xmlns:mx="http://www.adobe.com/2006/mxml"
	creationComplete="getLatest()">
	<mx:Script>
		<![CDATA[
			import com.dougmccune.subversion.SVNRevision;
			import com.dougmccune.subversion.events.SVNRevisionListEvent;
			import com.dougmccune.subversion.SVNClient;

			private var client:SVNClient;

			private function getLatest():void {
				client = new SVNClient();
				client.addEventListener(SVNRevisionListEvent.REVISIONS_LOADED, revisionsLoadedHandler);
				client.getLatestRevisions("http://opensource.adobe.com/svn/opensource/flex", 25);
			}

			private function revisionsLoadedHandler(event:SVNRevisionListEvent):void {
				for each(var revision:SVNRevision in event.revisions) {
					trace("Revision #" + revision.revisionNumber + " committed by " + revision.creatorName);
				}
			}
		]]>

	</mx:Script>
</mx:WindowedApplication>

Or here’s another very simple example that uses all MXML:

<?xml version="1.0" encoding="utf-8"?>
<mx:WindowedApplication xmlns:mx="http://www.adobe.com/2006/mxml"
	xmlns:subversion="com.dougmccune.subversion.*"
	layout="vertical">

	<subversion:SVNClient id="client" revisionsLoaded="grid.dataProvider = event.revisions" />

	<mx:HBox width="100%">
		<mx:TextInput width="100%" id="repoInput" text="http://opensource.adobe.com/svn/opensource/flex" />
		<mx:Button label="Load" click="client.getLatestRevisions(repoInput.text, 25)" />
	</mx:HBox>

	<mx:DataGrid id="grid" width="100%" height="100%">
		<mx:columns>
			<mx:DataGridColumn dataField="revisionNumber" headerText="#" width="40" />
			<mx:DataGridColumn dataField="creatorName" />
			<mx:DataGridColumn dataField="date" />
			<mx:DataGridColumn dataField="comment" showDataTips="true" />

		</mx:columns>
	</mx:DataGrid>

</mx:WindowedApplication>

Accessing repositories with authentication
The library supports using basic HTTP authentication to access repositories as well. Essentially all you have to do is pass the username and password into the SVNClient constructor, so instead of just calling var client:SVNClient = new SVNClient(), you would call something like var client:SVNClient = new SVNClient("username", "password");. Note that just like with all browser HTTP basic authentication, the password is sent using a Base 64 hash that is easily decoded.

But for a more secure connection, thanks to the SSL stuff that’s included in the as3crypto library and used in the as3httpclientlib library, you can also access SVN repos using HTTPS. If there are repositories that use Digest (as opposed to Basic) HTTP authentication, I do not currently support that. Frankly, I don’t have any repos I use that use digest authentication, so I don’t have any way to test it (or desire to write it).

Going further
If I was going to build a full featured SVN client I would probably still try to do a custom socket approach that spoke the SVN protocol. The WebDav interface seems a bit limited compared to “real” Subversion commands, and I would be worried that it wouldn’t be good enough. However, WebDav does support stuff like updating files and committing changesets, so I think you could add in some basic committing functionality and still use the WebDav approach. I don’t have any immediate plans to do any of this, for now I’m happy with the capability of reading from a repository (I’ll write more later about an application I’m working on that uses this library). But if you’re interested in taking this code and adding in more support for things like checking out full revisions, committing changes, etc., please be my guest.

License
Since this uses the as3crypto library, which is licensed under the New BSD license, I’m not entirely sure what that means for my code. I’d like to say mine’s MIT, but if I have to say it’s New BSD (which is almost just as flexible), then I guess I’ll say it’s New BSD. I think that really just means that I need to include the license file that contains copyright information in it when I redistribute the as3crypto library (which I have done). So you can pretty much do whatever you want.

You can be one of the cool kids too and follow me on Twitter. Twitter’s pretty much the best service in the entire world, and I simply can’t live without it.

Download the code here.

You’ll see that each chapter contains two links, one to view the source code online and one to download a zip file. If you view the code online you can browse through each of the code listings for the chapter right in your web browser (and cut and paste to try running the sample yourself in Flex Builder). If you download the zip file you can get all the source files on your computer and load them right into Flex Builder yourself. Each chapter was created as a separate Flex project, so you can just create a new project for whichever chapter you like and copy over the files that are in the zip.

We’re still pulling together the last of the samples, so the code for chapters 2, 8, and 12 is on its way. We hope to have those chapters posted by the end of the week. My sincere apologies for not having the code samples online immediately when the book shipped. Wrapping up the creation and production of a book like this was a whirlwind of activity, and we certainly slipped a little in getting the code online.

First, the standard stuff:

• Intense work
  I took a full-time gig with Universal Mind and we’re about to launch the product that I’ve been working on for a very long time. It’s one thing to work on a consulting gig where you walk in, do your shit, and get out. This is different. This product has so much of my time and energy invested in it. It’s my baby, and because I’m so invested, it consumes more of my mental time than any consulting project ever did. It’s not like I work 100 hour weeks for UM, I work normal hours. But my head doesn’t shut off when I commit my night’s changes and walk away from the computer. So that means I’m thinking about Flex for a huge portion of my non-work hours.
• Writing Flex for Dummies
  Flex For Dummies basically took over my weekends for 7 months of my life. I wrote from Friday night to Sunday night (our deadlines were Monday and I usually turned in a chapter around 1 am). I didn’t go out, I didn’t go on weekend trips. And the last thing I wanted to do was blog. But that process is over and the book’s about to come out! I now have my weekends back, but I also have a new appreciation for my free time. I used to do a lot of blog posts over the weekends, now sitting in front of the computer is the last thing I want to do.
• Moving in with my girlfriend
  It’s been about a month since I moved from San Francisco to Berkeley and moved in with my girlfriend. So I was dealing with the whole packing and moving thing, and also adjusting to the new living situation. Things have been fantastic since moving in, and this is another reason I’m not blogging as much. After a day of work I’d much rather go to dinner with my girlfriend and spend time hanging out with her instead of sitting in front of my monitor writing code.

So that’s all the stuff that I’d tell you if you asked me why I haven’t been blogging. “Man, I’ve just been so busy, I can’t find the time.” There’s a lot of truth to that, I certainly don’t want it to sound like I haven’t been busy. But largely the “I’ve been busy” excuse is bullshit. So why haven’t I been blogging?

I’m not hungry anymore.

Now we’re getting to the meat of this. I’ve been thinking about what drives me to blog and reflecting on the past year and a half. I’ve had a crazy ride. I went from being an unknown kid learning ActionScript and Flex to being a semi-famous-within-a-very-specific-geek-circle, book-writing, highly-paid, influential expert (and yeah, clearly you can tell that it’s all gone to my head). And it all happened in a year. My salary multiplied, I was speaking at conferences, I had a book deal. That’s some seriously crazy shit. That wasn’t entirely my plan from day one, although as things started picking up steam I started aiming higher and higher. I wanted to be famous. I wanted people to think I was good. I wanted to be in demand, to be sought after.

And so now what? What’s the next step? I’ve got a stellar job, I make bank, my book’s about to come out, I’ve got a reputation I’m proud of, and I’m being flown halfway around the world to speak at a conference. The people that I thought were rockstars, that I learned so much from while I was getting started (and still do every day); they all know who I am now (some of them are even good friends).

I used to be so hungry to prove myself. If someone had a coding problem they couldn’t solve I wanted to be the guy who showed everyone how to do it. I wanted people to stop and say “woah, where the fuck did this guy come from?” And now that I got that I guess I feel a bit drained. That fire to be the best, to be known, is dampened. I’m not saying I’m the best, I’m just saying I no longer care like I used to.

The Tech
There’s also a technology aspect to this. I’m starting to feel like I’ve taken Flex as far as it will go. I started proving small things; how to make little components, or how to hack some piece of the Flex framework. And I rode that until I knew Flex inside and out. I hit every brick wall (I climbed over or busted through some). And it was all so challenging and rewarding. I was unlocking secrets, uncovering things people didn’t know. But in terms of intellectual curiosity I’ve exhausted Flex 2 and 3. Of course there’s always new stuff in Flash Player 10 and Flex 4 that is going to be really interesting. But my interest has been shifting. I don’t want to solve small problems with a component or with a framework. I want big problems. I want new problems. I want real problems. I want to solve something that’s not just a technical detail.

I’m not ditching Flex or ActionScript. On the contrary, I feel like I’ve gotten to the point with the technology that I’m supposed to be: Flex and ActionScript are just tools to solve problems, not problems in themselves. When I started blogging Flex was a problem I was chipping away at, hacking together examples and tweaking the framework to get it to serve my needs. Now I want to move beyond that, and take on problems that are information problems on their own, and Flex just happens to be the best tool I have at my disposal.

The Next Step
I’ve been reading a good bit about information visualization as a field of research and soaking in as much as I can. The product I’ve been working on for Universal Mind takes geographic information visualization to a level that nobody’s done yet on the web (I put the “on the web” qualifier there, but I hope we’re working on stuff that pushes limits for any software, online or offline). And that’s just scratching the surface. There’s a ton room for innovation in the geo space. It’s a field that has so much potential but is stifled by a few big players and their ideas about how things are supposed to work. So I’m hoping I can help shake that up a bit. But geospatial information is just one small piece of the whole pie. I want to visualize everything. I want to create new ways of seeing data. And this is a field without limits. There is no end to the possible innovation.

So in the months ahead I hope to use this blog as an experimental proving ground for my forays into data visualization. I’m doing a lot of this for work, which is awesome because that means I get paid to do interesting work, but it has the downside that I often can’t share code. I imagine there will be more posts that are examples of what we’ve done (videos or live apps) that don’t show exactly how we did it. That sucks, but I think there’s still a lot to be said for showing off inspirational stuff. I’m also going to be trying to refocus on blogging small experiments in data visualization.

So if you’ve made it this far I’m impressed you read through my psycho-babble ramblings (or maybe you just skipped to the end looking for something worthwhile). From here on out I’ll probably keep the “dive into Doug’s head” type of posts like this to a minimum, and get back to blogging cool tech.

Thanks Dad.