It behaves like http push does in C git in that it is transparent to the end-user. Transparent client-side encryption can also be enabled, in case the repository data must be protected from the operators of S3.

First you need to create a bucket using some sort of standard S3 tools. I used jets3t’s cockpit tool to create “gitney”. A bucket may hold any number of repositories and acts as a root directory. It may also be a domain name if you want to use S3 based virtual hosting.

Next you need to create a properties file containing your AWSAccessKeyId and AWSSecretAccessKey so that jgit can authenticate itself with the S3 service. Since the AWSSecretAccessKey should be maintained privately its a good idea to store this in a protected file within your home directory.

  $ touch ~/.jgit_s3_public
  $ chmod 600 ~/.jgit_s3_public
  $ cat >>~/.jgit_s3_public
  accesskey: AWSAccessKeyId
  secretkey: AWSSecretAccessKey
  acl: public
  EOF

We also include acl: public so all objects (files) created by jgit through this configuration file are readable by anyone. The default (if not specified) is acl: private, making the objects readable only by yourself, and those who manage the S3 service.

Next we configure the remote in Git and push to the S3 bucket:

  $ git remote add s3 amazon-s3://.jgit_s3_public@gitney/projects/egit.git/
  $ jgit push s3 refs/heads/master
  $ jgit push --tags s3

Future updates are just as easy:

  $ jgit push s3 refs/heads/master

(or)

  $ git config --add remote.s3.push refs/heads/master
  $ jgit push s3

Pushes are always incremental and consequently there is relatively little bandwidth usage during subsequent pushes.

Our repository is now cloneable directly over HTTP (assuming we used acl: public):

  $ git clone http://gitney.s3.amazonaws.com/projects/egit.git

A jgit amazon-s3 URL is organized as:

  amazon-s3://$config@$bucket/$prefix
  http://$bucket.s3.amazonaws.com/$prefix

where the three major components are:

• $config is the name of the configuration properties file stored in $GIT_DIR/$config or $HOME/$config (searched for in that order).
• $bucket is the name of the Amazon S3 bucket holding the objects.
• $prefix is the prefix to apply to all objects (files) within this repository. It implicitly ends in “/”. You may omit this portion of the URI if you want the bucket to contain only one repository.

This is something of an abuse of URI syntax as the traditional username field is holding the name of a file in either $GIT_DIR or $HOME, however it permits hiding the secret access key from prying eyes as well as supplies a way to carry more information (such as acl or encryption settings) than what can appear in a URI.

Transparent client-side encryption for a repository stored on S3 can be enabled by adding a password to the properties file:

  $ cp ~/.jgit_s3_public ~/.jgit_s3_private
  $ echo password: Sup3rS3cr3t >>~/.jgit_s3_private

and using .jgit_s3_private in the $config field of an amazon-s3:// URL. The encryption algorithm can also be specified in property crypto.algorithm, which defaults to PBEWithMD5AndDES.

The encryption format currently used by jgit matches the format used by jets3t (specifically format version 2), making it possible to download and decrypt a repository through cockpit in the event that jgit is not readily available.

If you aren’t reading this blog post, you may need to reconsider how you reached this location, since you can’t see it.

The first image is from a set of octopus merges that occurred in the history. This probably would look better if we had just used git-rebase to transplant the commits instead of merging them, but at the time the user who created these was still quite new to Git…

(larger version)

What’s even worse about that particular rendering is the branches on either side. This was taken with the following gitk preferences set:

Maximum graph width (lines): 80
Maximum graph width (% width of pane): 90

Even with those settings, gitk still does not have enough space to show all of the active branches passing through this particular point in time (see the lines cut off on the upper right corner). In case you are wondering, yes, nearly all of those merged together later on. A few haven’t yet.
This second image was taken from a more recent point in the project’s timeline, during the same gitk session, and with the same preferences. We’ve obviously reduced the number of active branches somewhat, and there is now space for the commit subject lines (which I had to redact) and tracking branch labels (also had to be redacted).

(larger version)

There is still an octopus here (”Cauterize prior batch m”), but this one was automatically generated by a script we developed and was not directly caused by a user. The script takes a single commit and merges it to all branches that share a common prefix. I don’t have the repository on hand right now, but I think this particular octopus was created while we were merging one commit to ~80 branches. In this case 4 branches were already fully merged with a 5th, and we wanted to keep it that way after the new commit was merged to all 80 branches. To do that the script remerged these 4 by way of an octopus.

Before you can ask, no, this is not an imported repository.  This was all created in git, using the core porcelain and git-gui.

The git.git repository (red line) appears to have a very stable history, but sees a huge spike right around now. This spike is an outlier caused by my development repository; I was very actively rebasing a branch on Feb 26th, resulting in hundreds of commits in my reflog.

The linux-2.6.git history seems to show a periodic cycle, with some days reaching up to 700 commits per day. The obvious correlation between number of KiB of disk space used and the number of commits per day is also clearly visible. As I am not a kernel developer the linux-2.6.git data came from a mirror of Linus’ repository. I ommitted the first day outlier to prevent the Y-axis scales from shooting through the roof.

I have linked the images to higher-resolution PDFs, clicking on them will give you the PDFs.

The main window:

Branch creation dialog:

Delete branch dialog:

You can currently get git-gui by cloning the repository from repo.or.cz:

git clone git://repo.or.cz/git-gui.git

or track it’s progress through gitweb at git-gui @ repo.or.cz.

I hate to admit it but I find Stargate Atlantis to be entertaining, so I record the new episode every Friday night. Because I may not be able to get around to watching the show rigtht away I asked MythTV retain two copies of the show, deleting the oldest.

So right now my MythTV box has the episodes from Aug 25, Sept 18, Sept 16. But not last night’s episode, Sept 22. First off, why do I have 3 episodes when I asked for 2? And secondly, why do I have an old episode but not the most recent?!?

Looking at the oldrecorded table in MySQL it shows that the Sept 22nd episode was in fact recorded. But it must have been deleted just a few minutes after it finished recording.

Thus far its worked rather well for us, despite some odd user interface quirks that probably have more to do with the way I designed my Pronto panels than with MythTV itself.

But its use of MySQL is another story…

Tonight MySQL filled the volume I have the database stored on. What was a 4 GB disk volume went down to 0% free at 9:21 pm on Sunday. MySQL didn’t handle this very well and crashed.

But to make matters worse I have the default Gentoo+MythTV MySQL setup, which means all MyISAM tables and binary logging. After the great crash I moved the database to a dedicated volume of larger size and mounted that volume at the old location (/var/lib/mysql). However MySQL found a way to corrupt its log files during the crash and was unable to start. It kept demanding a non-existant log file.

I wound up having to check every table by hand with myisamchk, wipe out the logs, move the database to a temporary location, create a new database, delete the new database, then move the real database back. Why? Because even though the tables were all recovered the server kept demanding a missing log file and there was no obvious way to create a new log file to fill in its place. If the server really was logging it wouldn’t have found a way to lose its log file in the first place.

I should make it clear that the host, the disks, the filesystem were all online and available. 62 days of uptime. This crash wasn’t due to a power failure or disk corruption. Its purely a software bug in MySQL.

Recovery is one of those areas that you just want to always work. Oracle has had it working for many years (though I still have seen Oracle databases not recover but its pretty rare). PostgreSQL has also had pretty good recovery for a long time. These MySQL yahoos are still in the stone ages when it comes to recovery. Yet MythTV’s only available database backend is MySQL. Arrgh!

I hate crappy database software. I had to give up on my OpenLDAP installation as it kept eating its Berkley DB database. I don’t have time or the energy to try and recover the thing every time I want to use it. Now its looking like I might have to give up MythTV because of its equally bad database backend. No wonder why I only use PostgreSQL for my projects.