I’m in academic publishing. My grandparents founded a publishing company. My father ran it for a decade. I sit on the board of directors. You could say academic publishing is in my blood.

Today I am nauseous. Aaron Swartz is dead. I don’t know whether or not he would be alive today if he wasn’t prosecuted so aggressively for “stealing” academic journal articles. But what I do know is that this is a dark day in our history. It is a stain on the entire academic publishing industry.

I fiercely believe that as academic publishers we make the world a better place. We do good. I also believe there is a place for publishers in the Internet age. We’re working hard to figure out how to navigate these times. But everyone involved in this industry should be ashamed today.

We lost a genius. We lost a rebel.

I’m proud to be in publishing. But today I am nauseous. Today I am deeply sad. Today I am ashamed.

I’m happy to announce a new crime mapping application I’ve been working on that just went live on the Guardian DataBlog. The app lets you compare different cities in England to see where crimes of different types are distributed. You can either compare two cities side by side, or two different crime types in the same city. So if you’ve ever wondered which areas of London have high amounts of violent crime but low amounts of burglary, now you can find out.

This custom app was built upon SpatialKey, which made cranking it out only take a matter of days (the whole thing start to finish took about 4 days).

You make it vacuum your floor of course.

Inspired by this fantastic piece of work by Eric Testroete, my friend and I created my very own paper craft giant head (my buddy Todd did all the hard work of the 3D modeling and texturing). Of course, once you have such an amazing giant head, you need to figure out what the hell to do with it. And so boredom on a Friday night plus a few beers plus a giant head plus a roomba equals a magical vacuuming head!

A few more pictures of the head in action:

(*not my baby)

As I figure out more shenanigans to get into with my giant head I’m sure I’ll post more ridiculous photos and videos.

This is the time of year when people are making their resolutions, mapping out their big life changes for the coming year, and thinking ahead about what the future holds. I’m not one for resolutions, never have been. But I like the idea of taking stock of where you stand and thinking about changes you want to make.

And so to usher in 2010 I’ve decided to make a change: I’m removing “Adobe Flex Developer” from the title of my blog. The title used to read Doug McCune – Adobe Flex Developer. It will now read: Doug McCune – Data Visualization Engineer. Oooh, aaah.

This isn’t a change in anything other than my own mentality, but for me it is indeed significant. I’ll still be using largely the same tools, meaning I’ll be writing Flex applications and using ActionScript as my main language. I’m not changing jobs. My day-to-day work on Monday will be the same as my day-to-day work yesterday. This is a change in how I see myself, and most importantly how I see myself growing in the future.

Flex has been good to me, it provided a platform on which I could launch a career. Defining myself as a Flex Developer worked well and gave me enough room to grow. Within the small Flex community there were challenges, constantly new things to learn, and more than enough to keep me intellectually interested. The technology itself held my focus and through that focus I grew as a software developer.

But now that “Flex Developer” designation is a box that’s constricting rather than enabling. I gave a talk at 360|Flex in Indianapolis in which I talked about burning out. I explained that I had hit a point with the technology where the simply wasn’t enough left to keep me excited and passionate. As a side note, I love the fact that right on the first page of Google results for my name is an image of me standing in front of a slide that says “FUCK FLEX” How fitting.

The work I’ve been doing on SpatialKey has developed my interest in the field of data visualization, which is a field with an amazingly rich history and a field I see rising in importance in the years ahead. Most importantly, it’s a field I don’t know much about. There is so much to learn, from new tools and languages (R, map-reduce, Processing) to historical visualization techniques (cartography alone has such an amazing history) to modern data never before available (the human genome is on Amazon AWS). The field as it is right now gets me pumped up, and it’s a field that will experience explosive growth in the coming decade.

So starting in 2010 I won’t be identifying myself as a Flex developer. Flex will still be the tool I choose to use for my work, but it isn’t part of my identity like it used to be. I’m a data visualization engineer. And I’m excited.

Happy Halloween! This is the dorkiest pumpkin I’ve ever carved. For those of you into data visualization or mapping, maybe you can recognize it:

This is a pumpkin representation of Charle’s Minard‘s visualization of Napoleon’s march into Russia in 1812. This graphic is considered by some (ie Edward Tufte) to be the “best statistical graphic ever drawn.” The graph shows the size of Napoleon’s army as they marched to and from Moscow. You can see how the army shrank as they approached Moscow. Once they reached Moscow they found the city had been abandoned and burned. Then they marched back home, except it was through a brutal Russian winter and nearly killed the remaining army. By the time they return home you can see the size of the army is just a small trickle.

Beyond just the two charts of the march to and from Moscow, the graphic also serves as a map, with the paths indicating where the troops were geographically. And below the map is a temperature chart that visualizes how severe the winter weather was, which correlates with some of the major drops in troops on the way home.

The carved pumpkin ended up being very hard to take a photo of because the graph wraps around over half the pumpkin’s circumference. So I tried to take a few pictures to get the different sides. I carved the march to and from Moscow, as well as the temperature chart along the bottom.

Hope everyone has a great Halloween tonight!

Here are the slides and the code samples I presented at CFUnited.

You can also download the PDF of the slides if you want.

Adobe Open Source Media Framework + FlexLingo
This demo uses Adobe’s OSMF project to synchronize subtitles with video. This was the demo that I used to create the Hitler Goes to CFUnited video I posted before. You can seek around the video and OSMF makes sure to show the right subtitles. Then you can click on the different flag icons in the demo to use the FlexLingo library to translate the subtitle text to other languages (using Google translation API).

View the source
This demo is made with Flex 4, so to use the source code you’ll need to be able to build a Flex 4 project.

Axiis + Tweetr
The second demo I showed used the Axiis framework to create a chart of trending Twitter topics that have associated pictures on TwitPic. Unfortunately there’s no crossdomain file in place for TwitPic images, so I can’t run a live version of this app (but running locally works fine).

View the source
This demo is also made with Flex 4, so to use the source code you’ll need to be able to build a Flex 4 project.

Safe Sexting
The third demo I showed was the “Safe Sexting” app that uses the Marilena library for facial detection and uses Pixel Bender to automatically blur out your face. This demo was also shown in 360|Flex Indianapolis. You can read a bit more about this demo in a previous post about the 360|Flex presentation.

Run the demo | View the source

In my recent Flasher Magazine interview I talked about how much I dislike the “advanced” classes in the Flex SDK that were shipped with Flex 3, namely the AdvancedDataGrid, the OLAPDataGrid, and the grouping and hierarchical collection classes. I also said that I thought that all the list-based controls (DataGrid, Tree, List, etc) were too bloated and slow and that they should all be rewritten from scratch. I realized that that sounds a bit harsh, so I wanted to try to back up those statements with some code metrics.

I’ve put together a spreadsheet that includes each class in the Flex framework, including charting and the advanced data visualization package. This shows the number of lines of code in each file, breaking down the total number of blank lines, comment lines, and actual code lines. I’ve colored the rows to indicate if the class belongs to the advanced data visualization package of Flex 3 (yellow), the charting framework (blue), the AIR SDK (red), or the open source Flex SDK (white).

Here are the top 20 classes, ordered by actual lines of code (not counting comments and whitespace):

To get these numbers, I used a free program called cloc to analyze the source code. Cloc is a perl script that analyzes a bunch of different languages, including ActionScript.

For the full list, check out the spreadsheet. This is for version 3.0.0 of the Flex SDK, it does not include anything released in SDK versions 3.0.2 or 3.2 or any of the new Flex 4 classes.

So just look at that list of top classes. Out of the top five, three of them are the new classes for the AdvancedDataGrid. It’s also worth nothing that AdvancedListBase and AdvancedDataGridBaseEx are not used for anything other than base classes for AdvancedDataGrid. If you look at the inheritance chain for AdvancedDataGrid, it looks like this:

UIComponent
ScrollControlBase
AdvancedListBase (4,553 lines of code, 8,573 total)
AdvancedDataGridBase (1,184 lines of code, 2,084 total)
AdvancedDataGridBaseEx (4,503 lines of code, 7,428 total)
AdvancedDataGrid (5,385 lines of code, 8,432 total)

That means that the entire codebase for the AdvancedDataGrid is 15,605 lines of code (26,517 if you include comments and whitespace). That’s 7% of the entire Flex codebase in those 4 classes. Note that this is not including ScrollControlBase or UIComponent, only the new classes in Flex 3. Also note that AdvancedDataGrid doesn’t inherit from ListBase or DataGrid at all, and ListBase is the second largest class in the entire framework.

So this means that when I run into a bug in the AdvancedDataGrid (which has happened a lot), I’ve got about 30,000 lines of code to wade through to try to debug my problem. No thank you.

The other two out of the top five are ListBase (which is why I said the entirety of the list classes is too big) and UIComponent (which I think is understandable, and I haven’t run into too many problems that have root causes in UIComponent, unlike ListBase which continues to be a big headache for me). Then number 6 on the list is DataGrid (oh, but don’t forget about #12 on the list, DataGridBase). If you group all list-based classes together (meaning the advanced classes and the original list/datagrid/tree classes), you’ve got 9 out of the top 20.

Code quality
This post is only about raw numbers. There’s an entirely different topic of code quality that I can write post after post about when it comes to the quality of the AdvancedDataGrid (and related classes). They use noticeably different coding conventions, they deviate dramatically from the best practices employed by the other components, and they are plagued by performance problems. But that discussion is for another post. For now I just wanted to point out the shear massive volume of code that you’re using when you use these classes.

Am I being too harsh?
Some would argue that the classes have to be big, given that they do so much. After all, it is called the AdvancedDataGrid, it’s doing crazy advanced stuff. I call bullshit on that argument. In my opinion, I just don’t think there’s an excuse for such massive classes. They make developers lives miserable when trying to debug, but I also think it is indicative of a sloppy coding style. The mentality sort of seems to be “throw more code at the problem until you get it to work.” Check out the expandItem method of AdvancedDataGrid. It’s 445 lines long – this single method is longer than 80% of all the other Flex classes in the framework (yes, I checked how many classes are under 445 total lines). That method even includes a spectacular commented out block of code with this comment:

// Dont know why this is done
// So, commenting it and using the rowHeight instead

Awesome. I guess as long as it passes the unit tests go ahead and ship it

I’ve been working on an ActionScript library that can load revision history from Subversion (SVN) repositories. This is something that I’ve had in the back of my mind for about a year now and finally sat down and figured out a way to do it. I remember seeing a post way back in September of 2006 on Ted Patrick’s blog talking about someone who had written a SVN library using the Socket class. As far as I could tell, this was the only mention of the project, and it was never released publicly. So I decided to figure out how I could read SVN revision history with AS3. I’m releasing the library in its current state (more on that in a second) as an open source project.

Quick demo and the code
Since I know people don’t like reading, I’ll post the code at the beginning of this post and show some screenshots and a demo AIR app that uses the library. If you want to tune out after this, it’s cool, I understand.
Download as3svnlib source code, or just the compiled swc file.

Demo
Since this uses socket communication on port 80, which is off limits under normal circumstances due to Flash player security restrictions, you have to run the demo as an AIR app. Lame, I know. You can install this tiny AIR app to try out the library if you want. The demo app will let you enter the URL of a repository and load the last few revisions. The repository must be publicly accessible for the demo to work (note that the library itself allows you to use basic authentication, but I didn’t build support for that into the simple demo).

The screenshot below shows the bare-bones example AIR app. You can enter a URL for a SVN repo and then you can load the latest revision number or you can load the last 50 revisions. That’s all you can do in the app, but with the library you can get more detailed information about all the revisions and you can decide exactly which revisions you want to load (or load all of them).

You can download the example AIR app if you want to see if it works with your SVN repo (again, note that this demo app won’t work with URLs behind authentication, but read below about how to do that). View-source is enabled on the AIR app itself once you install it, or you can view the source of the example here.

The scope of this library
My immediate goal was to load revision history, complete with commiter name, date, comment, and ideally more detailed information about which files were affected or scope of the changes. I was not looking to build a full SVN client capable of committing changes or doing diffs or anything fancy like that.

How I went about it
I first started thinking I would implement a legit SVN client that would connect to a server using a socket connection and use the SVN protocol to communicate. I read up on the protocol and it didn’t seem all that complex, especially if I was going to limit myself to read-only operations. However, when I first started down this road I realized I didn’t know shit about doing socket communication in ActionScript. So I downloaded as3httpclientlib (current source is at github here), which is an AS3 library for doing HTTP communication written by Gabriel Handford. My goal was to use that project to learn how to do my own socket stuff and then build my SVN library.

But once I got my head around the http client stuff I learned something even more important. Turns out that most SVN installations include an Apache server that lets you browse the repo with a web interface. What’s even more interesting is that Subversion uses the WebDAV protocol as well, which is a protocol based on HTTP for source control. Suddenly I had a new option that didn’t require writing my own full-on SVN client. Fuck yeah! Less work for me!

So I extended as3httpclientlib and added in some basic support for WebDav (namely the PROPFIND request and a basic REPORT request). Then I added in support for some custom reports that Subversion uses, namely log-report. All the reports that Subversion supports via WebDav are documented here. When you request a report you send a custom HTTP request to the server and fill the request body with some XML that defines what kind of report you want. Then the server sends back its response in XML.

Then I was able to use simple XML parsing to read my list of revisions, which included revision number, creator name, date, comment, and details about all the files that were added, removed, modified, etc. Easy breezy.

Most basic use of the library
To use the library you interact with the SVNClient class. This class is used to fire off all your requests (either to get the latest revision number or to get specific sets of revisions). All requests are asynchronous, so you need to add an event listener to the SVNClient class, then send off your request, and then handle the result.

<?xml version="1.0" encoding="utf-8"?>
<mx:WindowedApplication xmlns:mx="http://www.adobe.com/2006/mxml"
	creationComplete="getLatest()">
	<mx:Script>
		<![CDATA[
			import com.dougmccune.subversion.SVNRevision;
			import com.dougmccune.subversion.events.SVNRevisionListEvent;
			import com.dougmccune.subversion.SVNClient;

			private var client:SVNClient;

			private function getLatest():void {
				client = new SVNClient();
				client.addEventListener(SVNRevisionListEvent.REVISIONS_LOADED, revisionsLoadedHandler);
				client.getLatestRevisions("http://opensource.adobe.com/svn/opensource/flex", 25);
			}

			private function revisionsLoadedHandler(event:SVNRevisionListEvent):void {
				for each(var revision:SVNRevision in event.revisions) {
					trace("Revision #" + revision.revisionNumber + " committed by " + revision.creatorName);
				}
			}
		]]>

	</mx:Script>
</mx:WindowedApplication>

Or here’s another very simple example that uses all MXML:

<?xml version="1.0" encoding="utf-8"?>
<mx:WindowedApplication xmlns:mx="http://www.adobe.com/2006/mxml"
	xmlns:subversion="com.dougmccune.subversion.*"
	layout="vertical">

	<subversion:SVNClient id="client" revisionsLoaded="grid.dataProvider = event.revisions" />

	<mx:HBox width="100%">
		<mx:TextInput width="100%" id="repoInput" text="http://opensource.adobe.com/svn/opensource/flex" />
		<mx:Button label="Load" click="client.getLatestRevisions(repoInput.text, 25)" />
	</mx:HBox>

	<mx:DataGrid id="grid" width="100%" height="100%">
		<mx:columns>
			<mx:DataGridColumn dataField="revisionNumber" headerText="#" width="40" />
			<mx:DataGridColumn dataField="creatorName" />
			<mx:DataGridColumn dataField="date" />
			<mx:DataGridColumn dataField="comment" showDataTips="true" />

		</mx:columns>
	</mx:DataGrid>

</mx:WindowedApplication>

Accessing repositories with authentication
The library supports using basic HTTP authentication to access repositories as well. Essentially all you have to do is pass the username and password into the SVNClient constructor, so instead of just calling var client:SVNClient = new SVNClient(), you would call something like var client:SVNClient = new SVNClient("username", "password");. Note that just like with all browser HTTP basic authentication, the password is sent using a Base 64 hash that is easily decoded.

But for a more secure connection, thanks to the SSL stuff that’s included in the as3crypto library and used in the as3httpclientlib library, you can also access SVN repos using HTTPS. If there are repositories that use Digest (as opposed to Basic) HTTP authentication, I do not currently support that. Frankly, I don’t have any repos I use that use digest authentication, so I don’t have any way to test it (or desire to write it).

Going further
If I was going to build a full featured SVN client I would probably still try to do a custom socket approach that spoke the SVN protocol. The WebDav interface seems a bit limited compared to “real” Subversion commands, and I would be worried that it wouldn’t be good enough. However, WebDav does support stuff like updating files and committing changesets, so I think you could add in some basic committing functionality and still use the WebDav approach. I don’t have any immediate plans to do any of this, for now I’m happy with the capability of reading from a repository (I’ll write more later about an application I’m working on that uses this library). But if you’re interested in taking this code and adding in more support for things like checking out full revisions, committing changes, etc., please be my guest.

License
Since this uses the as3crypto library, which is licensed under the New BSD license, I’m not entirely sure what that means for my code. I’d like to say mine’s MIT, but if I have to say it’s New BSD (which is almost just as flexible), then I guess I’ll say it’s New BSD. I think that really just means that I need to include the license file that contains copyright information in it when I redistribute the as3crypto library (which I have done). So you can pretty much do whatever you want.

Which I consider pretty amazing since I’ve never sent a single tweet. That pretty much puts my follower to tweet ratio at Infinity.

You can be one of the cool kids too and follow me on Twitter. Twitter’s pretty much the best service in the entire world, and I simply can’t live without it.

The compiled list of code samples from our Flex for Dummies book have been posted on the dedicated Flex for Dummies page of my blog. Each chapter (other than chapters 1, 4, 19, and 20) contains source code samples, and you can browse through them all online or download the code to run in Flex Builder yourself. We’ve double and triple checked everything to make sure it’s all error-free.

Download the code here.

You’ll see that each chapter contains two links, one to view the source code online and one to download a zip file. If you view the code online you can browse through each of the code listings for the chapter right in your web browser (and cut and paste to try running the sample yourself in Flex Builder). If you download the zip file you can get all the source files on your computer and load them right into Flex Builder yourself. Each chapter was created as a separate Flex project, so you can just create a new project for whichever chapter you like and copy over the files that are in the zip.

We’re still pulling together the last of the samples, so the code for chapters 2, 8, and 12 is on its way. We hope to have those chapters posted by the end of the week. My sincere apologies for not having the code samples online immediately when the book shipped. Wrapping up the creation and production of a book like this was a whirlwind of activity, and we certainly slipped a little in getting the code online.