Instead of reposting the whole thing, I’ll just point to my two messages in context:

why do I need to spend my time on this crap?
why is the new file header what it is?

Right now, if I have X hours to work on a project, I seem to be averaging what feels like X/2 hours in paperwork and other “important steps” of the development process. None of which have helped my project to ship higher quality, or more feature complete code. Which means either my or my employer’s time is being wasted. I don’t have time to waste when I have 108 bugs open in Gerrit Code Review, and 64 bugs open in EGit and JGit.

Based on a private email chain I’m having with the Eclipse IP review team, it looks like the initial EGit code contribution was bungled not just by myself, but also by the foundation’s IP review process. Which means I probably have to run EGit back through IP review, almost from scratch. But only after I write a script to datamine contributors out of the old EGit history and inject a complete, per-file git short-log into each file header. Its a good thing I have an awesome version control system like Git to keep these records for me. Too bad nobody else on the planet can use it to obtain information they might want to know about our source code. I guess running software to read information about a file is too scary for some individuals. So I have to do it for them. Now, and for every change we make in the future. Yay.

The astute reader may notice in that above paragraph, “private email chain” doesn’t jive with other publications from the Eclipse Foundation demanding that projects be run in an open and transparent manner (see how do I start a project on Eclipse Newcomers). I really do feel like JGit is a less open project now that it has moved to Eclipse.org. Conversations with the Eclipse IP team about the legal status of any contribution is always discussed by private email. These things never make it to the project mailing list. The IPzilla database is closed to everyone but committers. There are backroom deals going on about what our file headers should look like in order to sufficiently convey that the source code is under the new-style BSD. The discussion that led to the approval of the EGit IP log for 0.7.0, approved despite what appears to be an error in the initial review, also happened by private email.

It took a significant amount of effort on my part to even get JGit hosted at Eclipse.org. Originally, the new-style BSD license wasn’t permissible for a hosted project, and I had to seek a special exemption from the Eclipse Board of Directors. A process that required significant backroom conversations, over at least 6 months. Again, not exactly open. The only reason I think I haven’t pulled the project back is because of the huge initial investment I’ve already made in this.

Maybe JGit and EGit are just unique projects. But in my experience, I am not a unique snowflake, and neither is my work. I’m not as special as I might seem at first glance.

I wouldn’t be surprised if I’ve lost at least 2 days every month to paperwork. That’s about 30 days, or 1.5 person-months since the project really started this move in January 2009. 1/12 of my time over the past year has just gone to catering to the Eclipse development process. Food for thought. Join Eclipse… make sure you pick up at least 1/12 of another full-time developer just to deal with the red-tape.

The part that really troubles me with the red-tape isn’t so much that it is there, but that committers bear the brunt of the effort, while large corporations that are strategic members reap the benefits of having a concise change history listed inside of each source code file, or knowing that every contributor who ever touched this source code has been grilled in detail on a bug tracker.

So back to my post title. The real tragedy is, these corporations who sell commercial products based on top of Eclipse.org distributions are pushing not just the open source development work, but also a whole ton of onerous legal and reporting constraints back onto their project committers. Its enough to make this committer start to reconsider things. I wish I had been using a time clock this past year, to accurately record how many days the Eclipse development process has robbed me of since the start of all of this. It feels significant enough that if I went to my manager with it, I think he’d go ballistic.

Commit messages, when combined with a powerful blame engine in your version control, can give you really powerful insight into what you were thinking at the time. This can be incredibly handy when someone asks a question later.

Yesterday, Junio Hamano, git maintainer extraordinaire, asked me why git-gui implements its own clone function. When I wrote this code, it must have been really obvious to me why it needed to reimplement the same logic as git clone. But I wrote it back in 2007. I’ve done a ton of things since then. There’s no way I can remember what I was doing, or why I was doing it. I do however remember thinking, “this code is done, it works, I’ll never have to look at or think about it again”. Famous last words.

When Junio asked this question… I honestly couldn’t remember what I was doing. I’m usually somewhat against reinventing the wheel, and I try to avoid rewriting something unless I seem to have a good reason for it. So I really was looking at his question saying, “yea, why did I do that there…”.

Fortunately, I write fairly detailed commit messages, and git blame is an incredible tool:

  $ git blame lib/choose_repository.tcl
  ...
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  633)
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  634)           $o_cons start \
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  635)                   [mc "Counting objects"] \
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  636)                   [mc "buckets"]
  ...
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  673)           update
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  674)
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  675)           file mkdir [file join .git objects pack]
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  676)           foreach i [glob -tails -nocomplain \
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  677)                   -directory [file join $objdir pack] *] {
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  678)                   lappend tolink [file join pack $i]
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  679)           }
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  680)           $o_cons update [incr bcur] $bcnt
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  681)           update
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  682)
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  683)           foreach i $buckets {
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  684)                   file mkdir [file join .git objects $i]
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  685)                   foreach j [glob -tails -nocomplain \
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  686)                           -directory [file join $objdir $i] *] {
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  687)                           lappend tolink [file join $i $j]
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  688)                   }
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  689)                   $o_cons update [incr bcur] $bcnt
  81d4d3dd (Shawn O. Pearce     2007-09-24 08:40:44 -0400  690)                   update
  ab08b363 (Shawn O. Pearce     2007-09-22 03:47:43 -0400  691)           }

It would seem that 81d4d3dd, and ab08b363 are commits adding code to do a clone.

  $ git show 81d4d3dd
  commit 81d4d3dddc5e96aea45a2623c9b1840491348b92
  Author: Shawn O. Pearce  spearce.org>
  Date:   Mon Sep 24 08:40:44 2007 -0400

    git-gui: Keep the UI responsive while counting objects in clone

    If we are doing a "standard" clone by way of hardlinking the
    objects (or copying them if hardlinks are not available) the
    UI can freeze up for a good few seconds while Tcl scans all
    of the object directories.  This is espeically noticed on a
    Windows system when you are working off network shares and
    need to wait for both the NT overheads and the network.

    We now show a progress bar as we count the objects and build
    our list of things to copy.  This keeps the user amused and
    also makes sure we run the Tk event loop often enough that
    the window can still be dragged around the desktop.

    Signed-off-by: Shawn O. Pearce  spearce.org>

  $ git show ab08b363
  commit ab08b3630414dfb867825c4a5828438e1c69199d
  Author: Shawn O. Pearce  spearce.org>
  Date:   Sat Sep 22 03:47:43 2007 -0400

    git-gui: Allow users to choose/create/clone a repository
  …
    Rather than relying on the git-clone Porcelain that ships with
    git we build the new repository ourselves and then obtain content
    by git-fetch.  This technique simplifies the entire clone process
    to roughly: `git init && git fetch && git pull`.  Today we use
    three passes with git-fetch; the first pass gets us the bulk of
    the objects and the branches, the second pass gets us the tags,
    and the final pass gets us the current value of HEAD to initialize
    the default branch.

    If the source repository is on the local disk we try to use a
    hardlink to connect the objects into the new clone as this can
    be many times faster than copying the objects or packing them and
    passing the data through a pipe to index-pack.  Unlike git-clone
    we stick to pure Tcl [file link -hard] operation thus avoiding the
    need to fork a cpio process to setup the hardlinks.  If hardlinks
    do not appear to be supported (e.g. filesystem doesn't allow them or
    we are crossing filesystem boundaries) we use file copying instead.

    Signed-off-by: Shawn O. Pearce  spearce.org>

So 30 seconds after being asked, I’ve managed to remember this was mostly about git-gui on Windows, where Cygwin can be pretty slow for file operations, and hardlinks are available on NTFS if your application knows how to make them. By doing the clone logic within Tcl, which is a native Win32 application, we can bypass Cygwin overheads, including the need to fork and execute a bunch of commands from the git-clone.sh shell script. Because, back in 2007, git-clone was still just a shell script.

In hindsight, that paragraph above should also be in the commit messages. And I probably should have ported git clone to C instead. Its C now, but not because of my efforts. And now git-gui maybe should just call it. It would have made git-gui a whole lot smaller.

You can follow the rest of the thread.

assertTrue("isa TransportHttp", t instanceof TransportHttp);
assertTrue("isa HttpTransport", t instanceof HttpTransport);

What is wrong with me…

Today I just tried to install the Eclipse Test & Performance Tools, so I could try to find out why Apache MINA SSHD has such poor throughput during uploads into the server.  Unfortunately I can’t run version 4.5 because it depends on a decade old version of libstdc++.  But I can’t install version 4.6, which supposedly has a newer linkage, because the 4.6 requires SWT 3.4.0.  But I’m running Eclipse 3.4.2, which apparently does not have a new enough SWT.

Folks.  Seriously?

The only major consumer of SWT that really matters is the Eclipse SDK.  And the SDK platform version numbers don’t match SWT version numbers.  And the test and performance tools require a decade old shared library which isn’t even distributed with Ubuntu Hardy.

I guess since its Java, its OK to repeat decade old mistakes, because its in a different programming language.

So I’m currently reading Peter Seibel’s Coders at Work.  Tonight, while reading through Brendan Eich’s interview and his history at Netscape, it reminded me of my own development history around the same time period.  I really don’t talk about my past very much, so before I get to be too old to remember it, I might as well write some of it down.  :-)

During the mid-to-late 90’s I was still in high school, but was working part-time in the afternoons after school as a software developer (err, script monkey) for a now defunct website and ISP called INJersey.  INJersey was the aspiring, and perhaps too early for its time, online arm of the local daily print paper, the Asbury Park Press. Somehow its owners realized this Internet thing was worth investing in, before a lot of users figured it out and actually got online.

Back then, we didn’t have Apache and all of its module glory.  We had a pile of patch files you had to apply to NSCA httpd to get A PAtCHy server.  We didn’t have FastCGI, we had plain old CGI scripts, often written in Perl 4, where &article'render() was an actual function call and not some syntax error brought on by a lack of caffeine.  We didn’t even have JavaScript or animated images. <blink> was as good as it got.

My job back then was really simple.  Take Microsoft Office documents from writers who couldn’t be bothered to learn HTML, and put them online in HTML.  I wrote a lot of Perl and AppleScript to rip the stuff apart and put it back together again as plain HTML files that we could serve to web users.  This was out of shear laziness, the first great virtue of a programmer.  INJersey hired me not as a software developer, but as a data entry monkey to copy and paste the text from Office into an HTML document, and stick in the <td> or <b> tag when necessary.  I quickly grew bored with that task and found it much more fun to write scripts to do my job for me, while I explored the wonders of a T1 internet connection.

My managers quickly realized I was able to do more than just copy and paste text with a computer, so they started giving me simple programming assignments.  When Netscape 2.0 launched, one of our real programmers figured out we could do server-push based animated images.  This is one of those monstrously stupid ideas that I’m glad has died on the web.  I’m quite happy that I actually can’t come up with a great reference link for it anymore.  Anyway, INJersey’s website was just awesome one day, because we had animated images, and others didn’t.

About this time one of my managers saw this website called JangaChat.  It was a free online web based chat room system.  No IRC client needed. No Java applets. No plugins.  Just Netscape 2.0.  Even better, they allowed HTML to be entered without escaping, so you could write fancy messages like “Bob, that was the best fish <b>ever</b>!” and actually have it come out in bold.  Their site was more awesome than our animated images.  We somehow had to out awesome them again.

Remember, this is like 1995.  We just got Netscape 2.0, and Brendan Eich had just unleased this JavaScript thing on the world.  We didn’t know what we could do with it… or the damage it could cause.  Cross site scripting hadn’t even been invented yet.

My managers gave me a simple task, create an INJersey version of JangaChat that we could run on our own servers, so our users could chat online in real time about whatever they felt like, without needing to first install an IRC client.

At the time, I only really knew Perl, and was only self-taught at that.  So I wrote the first version as a Perl CGI.  I remember abusing the server-based multipart push feature we used for image animation to allow the Perl CGI to stream new messages to the browser as they arrived in the chat room.  JangaChat, and this system I worked on at INJersey, may have been some of the first uses of hanging GETs.

Unfortunately, not only did I not know C, I also didn’t know how to do proper interprocess communication on UNIX.  So I implemented with what I did know:  each chat room was assigned a local file.  New messages were appended onto the end of the file using a POST CGI, and messages were read out to active users by their hanging GET CGIs, which were continuously trying to read the tail of the room’s file.  Over a decade later, I can’t believe I was once foolish enough to believe this was a good idea.

We launched under the name ChatterBox.  We quickly stole most of the users from JangaChat, plus picked up our own users, and our little Pentium 133 server could barely keep up with all of those Perl based hanging GET CGI processes demanding resources.  Messages were also often truncated or otherwise badly mangled, as I had no file locking, and no way to ensure a full message was written before being read and sent to a browser.

Management really wanted this product to work, because the demo was flashy, and their local advertising customers were wowed by it.  I rewrote the Perl code into C, but kept the basic design of individual hanging GET CGIs per user, and a single log file per chat room.  Even in C, we still couldn’t keep up with traffic.  Somehow I convinced management that buying some “real server hardware”, rather than our tiny single processor Pentium 133, would solve the scaling problems, and they went out and purchased a pair of SGI Origin 200 servers running Irix.  Today of course, all 3 of you reading this are screaming “but its the software that wasn’t scalable!”.  That’s what a decade of learning gets you.  :-)

So upon getting this shiny new server hardware, I start thinking about how I might have done things differently.  I knew I couldn’t rely on the (then pretty crappy and unscalable) dbm library for user data management, like I had with Perl.  So I started poking around at things like mSQL, PostgreSQL, and MySQL.  Back in 1996, mSQL was the best there was, but it was single threaded.  PostgreSQL was pretty slow and barely ran, it was more of a research project than it was a production database engine.  MySQL crashed more than it stayed up, severely lacked features compared to mSQL, and required nasty pthread libraries which weren’t exactly standard or well supported on UNIX systems back then.

So I toyed with writing my own.  As it turned out, Irix had a decent pthread implementation at the time.  I actually ended up with a thread safe, balanced B*-tree that stored user record data in fixed size leaf nodes, and used an arbitrary byte sequence as the record key.  To make the implementation easier on myself, I made every block of the file the same size (I think I used 2048 bytes, but I can’t remember), and the parent that pointed to the block told you the type using the lower bits of the file offset.  E.g. offset & 1 == 0 meant the block was another btree intermediate node, while offset & 1 == 1 meant it was an application data record.

Within the user data record, I didn’t want to write all of the glue required for a formal DDL like a proper SQL server would support.  After all, I just had to store a couple of values for each user account, like their email address, date of last login, and a handful of preference settings.  So I basically did what Google protocol buffers does, and stored a flexible bag of key/value pairs in binary form.  Schema changes were accomplished incrementally, as users were updated during normal application processing, rather than all up front when the software changed.

So, I solved my data storage problem by just rolling my own system.  Looking back on it, its not too different from the approach Google takes with BigTable.  Only they scale across machines by partitioning the key space, while I just tossed the entire key space onto a single node.

For the interactive web components, I finally got a clue and realized that the hanging GET CGI processes were my real scaling factor.  They consumed too much of the system’s resources per process.  My local UNIX system administrator finally got around to telling me about sar and ps and how to use them to figure out that I was being a moron.

Since we still didn’t have FastCGI, I wrote my own HTTP server.  From scratch.  In C.  First and last time I’ve ever done that.  Lesson learned.  There’s a reason I haven’t been involved in the Apache HTTPd project.  My scars still haven’t healed.

Initially, my HTTP server was using pthreads, spawning a pthread per browser request.  Which meant that my server used 1 thread per hanging GET.  At first I thought this would be OK, threads were lightweight compared to a process, right?  Until my local UNIX administrator brought out the clue stick and made me think about it for a few minutes.  In the process-per-connection implementation we had very little allocated memory, just a kilobyte or so of global state, and the entire program executable was mapped as shared memory between all of the instances.  The real memory cost was in the program’s stack, which the OS had a minimum size on.  With pthreads we didn’t gain much resource savings over the process-per-connection approach because we still had the same per-connection state, and the same minimum thread stack size.

I started digging around the Irix manual pages, and one day found this nifty thing called select().  I wrote a quick simple server using it, and realized non-blocking asynchronous IO was the best thing since sliced bread.  That night I completely tore apart the pthread based HTTP server and rewrote it as a non-blocking, asynchronous IO server.

By now I had completely abandoned the idea of the message distribution going through a local log file, and instead stored it in shared memory protected by pthread mutexes and condition variables.  Threads posting messages into a chat room appended their message object onto a distribution queue and signaled one of the IO threads to start streaming the messages out using asynchronous IO to each connected browser.

Along the way I also had to write an HTML parser, because we wanted to allow messages like “Here is a <font color=red>red rose</font>” to render as intended by the author, but we didn’t want to have unclosed tags like “<h1>Hi!” ruin the formatting for every subsequent message on the page.  End users also figured out cross site scripting before we did, with plenty of “<script>window.close()</script>” nosense being pasted into rooms and closing hundreds of browsers at once.  Yes, we learned about cross site scripting attacks the hard way.  Back in 1996/97, nobody really thought about this sort of stuff.

We relaunched in the winter of 1997.  The Wayback machine has a snapshot of the rewritten homepage dated April 13, 1997.

Of all of that though, the thing I’m still most excited about is the “non-Shockwave” version of the ChatterBox website.  Back then we didn’t have AJAX and XMLHttpRequest.  Heck, we didn’t even have iframe.  We had straight up boring <frameset>.  I rolled my own AJAX system from scratch using JavaScript, a hand rolled message queue, and a 1 pixel high frame in a frameset that performed POSTs driven by JavaScript.  In very early 1997.  Apparently I discovered parts of the web that we didn’t see again until XMLHttpRequest was widely supported.

Unfortunately management pulled the plug a year or so later, and that neat little HTTP server with its pre-AJAX AJAX and pre-Google Talk hanging GETs disappeared into the sands of time.

So, tonight, while reading Brendan Eich’s interview talking about building JavaScript in two weeks, I remembered just how much I learned that year in my part time job.  I went from being a Perl script monkey who couldn’t even compile a C program, to having written an asynchronous IO HTTP server from the ground up, mastered multi-threading, mutexes, condition variables, AJAX, cross site scripting, and a balanced binary tree implementation that bears a lot of resemblance to BigTable.  All without taking any CS classes in high school.

Maybe that’s why college, and my later jobs, were all so boring for me.  I’d done all of it already. Maybe that’s why I was less amazed than my peers when Google Maps launched. I didn’t make something nearly as awesome, or as empowering to so many people around the world as Lars and the Maps team did. But I had certainly seen, worked with, and discovered much of today’s web. In 1997.

What is HTML 5 going to bring us that we haven’t discovered yet?

Thanks rtyler.

However, he’s right.  EGit has to get better, fast.  We need more contributors who know and love the SWT/JFace/Resource APIs and can crank out the UI improvements necessary to bring the Git team provider up to the same level as the CVS team provider.

Last week I threw SpamAssassin off my domain’s server, which meant I finally had virtual memory free to run git daemon. So you can now find most of my projects at git.spearce.org, with proper git:// URLs available for efficient cloning.  Maybe sometime later this month I’ll get smart HTTP enabled as well.

I’ll continue to update the repositories on repo.or.cz, but I’ll primarily be using the ones hosted on git.spearce.org.

Fortunately Wayne Beaton helped us get Gerrit configured in a way that meets the foundation’s IP process guidelines, and has encouraged us to continue forward.  This is great, because it means we can rely on Git for attribution tracking, rather than Bugzilla.

Also, since the project moved homes we picked up 3 prolific contributors, and all of them have turned into committers on the project.