My day job requires me to use PVCS Version Manager as our source code control system. The version we are using is a couple of years old (its the 6.8 release) and due to corporate security policies we are required to use the I-NET interface rather than the native Windows client. I-NET is essentially a Java applet which only works in Internet Explorer and provides very limited access to the PVCS Version Manager repository.
As a developer I have found PVCS Version Manager to be useless and a huge time sink. It is a horrible tool. Its retail price is more expensive then Perforce for a development team of 20 people. It lacks the most basic features. Its garbage and should be pulled from the market.
Developers can’t be given the security rights necessary to create new files; consequently I have to write new code then send a bug report for each file to our configuration management team, who creates the file for me in the version control system. Ditto for deletes. This creates a headache for me if I send a bug report to have the file created but then find and fix a bug in the file before it has been added to Version Manager. (Given that the average turn-around on new file creation is 48 hours, this is rather common.) I have no idea which files I need to now check out of Version Manager to check in bug fixes and which files are OK as-is.
Did I mention this is even worse than it would appear as Version Manager offers no way to compare your working copy of the source code to a version in the repository? Sure it offers a single file difference between your working file and the repository version of the file, but what it can’t do is give me the differences (if any) for all 8000+ files which make up our small product. Most of them don’t typically have differences, but how do I know which ones have them if Version Manager can’t tell me?
Did I mention that when another developer requests a file to be deleted from the repository and our configuration management team follows through with the bug report that was sent to them, Version Manager won’t remove it from my working directory? Yea… what’s up with that? It will happily leave all deleted content in my working directory until the cows come home, pigs fly, or I delete all source from my local machine and refetch it all from the version control system. I think we’ve lost some 100 man hours over the past 8 months to this problem alone.
Writing up the bug reports for file creations and file deletions is also a huge time sink. There’s no automated process to do this, so developers are writing down on pen and paper the names of the files they are creating as they create them; then a week or so later writing out bug reports for each one and submitting them to the configuration management team. We’ve estimated most developers are spending about 30 minutes to an hour each week doing this activity. Over the span of 8 months this is 240 man hours (0.75 hours * 10 developers * 32 weeks).
We also can’t have multiple developers editing the same files at the same time. If I’m adding a new constant at the bottom of a file and Bob is adding a new constant near the top of the file (at least 50+ lines from where I’m adding my new constant) I have to either wait for Bob to finish or make the change on my local system, then days later figure out my file is missing Bob’s change and redo Bob’s change manually. Between schedule slippage of developers having to wait for a file to be released for them to modify it, or wasted man-hours redoing work which had previously been done (but couldn’t be automatically merged by Version Manager as it lacks this ability) each developer has lost ~30 minutes each week for 8 months. That’s 160 man hours.
I’ve seen developers spend an entire day because they can’t get their local working copy to compile anymore. This typically happens because they have a partial update from Version Manager. Typical example is the developer has modified file X.java (but hasn’t checked in yet); another developer has modified A.java, B.java, C.java, … E.java, and checked all of those files in. The changes in A-E are dependent on each other, but the dependencies aren’t immediately obvious. The developer working on X.java realizes he also needs to modify E.java and checks it out for modification… now they have part of the A-E change, but not all of it, and suddenly they can’t compile code anymore or test. Because they can’t find out how their local directory differs from the version control tool they aren’t willing to just delete the directory and start over (for fear of losing their change in X.java and having to start over from scratch). But they also can’t find out that the change in E.java depends on A.java through D.java, as Version Manager can’t tell them these files were modified at the same time. This has cost the project at least 8 hours of a developer’s time once every couple of weeks over 8 months: 128 man hours.
By default Version Manager will set the modification date of source code files you are getting from the repository to be the modification date of the file when it was checked in by the last author. This is an [sarcastic]awesome feature[/sarcastic] when you are using Ant as a build tool, as Ant makes compilation decisions based on the *.java file being newer (has a later modification date) than the *.class file. If I compile A.java this morning, but Bob then checks in a new version of A.java which he wrote yesterday (and hasn’t modified since) and I then get the new version from Version Manager, Ant won’t recompile it as A.java is still older than A.class (the source has yesterday’s date on it, while the class has today’s date).
After discussing this with our local configuration management team they told me the Ant build system was broken; build systems which drive off modification dates on files can’t be trusted for incremental builds and another solution should be used. In classic form they couldn’t offer up any alternatives. I agree that file modification dates are problematic, but I’ve rarely had trouble when working only on a local filesystem (NFS without network time servers is a whole other story). Every other version control system that I’ve used in the past (except ClearCase) always sets the modification date of the source files to the date/time the file was gotten from the version control system, ensuring that a date/time based build system will recompile it properly. (ClearCase gets away with not doing so by providing ClearMake, a build process which drives off the version information stored by ClearCase.)
So given that I was forced to add an a task to Ant which compares file contents by MD5 checksum, then touches the files to move their modification date forward to the current date/time prior to running any other tasks (such as javac). This was 4 hours of development time wasted, and now our build process takes 5 minutes instead of 2 minutes. Given that I try to do about 10-15 builds per day I’m losing half an hour each day to waiting for the build process to update the file modification dates. Other developers are in the same boat.
Because we can’t do any reasonable differencing to determine which files we need to write bug reports for configuration management to modify for us, I’ve had to add special Ant tasks which scan for all files which aren’t read-only in a directory and copy them off to a holding area. The holding area serves as a place for the developer to run ‘diff’ from on the command line or from within the Eclipse IDE to determine what they need to send to our configuration management team. We’ve spent at least 1.5 person-days on this little set of scripts; 10 developers actively working with them (and producing on average 8 new holding areas per day – 1 per hour) has cost us nearly 20 GB of disk space on our file server. It has also entirely bypassed the security system of Version Manager, as now anyone with access to our file server can read the source code. (A wider user base has access to the file server than to PVCS Version Manager.)
With a (burdended) developer cost of ~$70/hour we’re talking about $92,960 (1,328 man hours) of wasted developer resources. Unfortunately management doesn’t see these hours as a real cost, but the project is 2 months behind right now. With 10 people working on the project we’re behind by at least 1 month thanks to our choice of version control tools. That’s just the last 8 months. We’ve had this tool for years.
Early on in this project we tried to branch our source code base. Version Manager can’t handle branches. The vendor claims it can, but it really can’t. Our local configuration management team was initially unable to setup a single branch of development properly. Developers were forced to spend a couple of weeks to get our configuration management team to correct the system; for the next couple of months we kept finding files which had been branched improperly. This likely cost us another 200 man hours between developers and configuration management: $14,000.
We could have purchased Perforce, ClearCase, BitKeeper, etc. for a fraction of the cost of the time we have thus far wasted, and we would have only been 1 month behind, not 2 (the other month loss is more likely due to a few ill-defined requirements which were incorrectly implemented early in the project). We can’t ship this project late, we’re contractally obligated to deliver it on time. If we don’t our customer will face a major revenue loss at a time which it can least afford to have one.
I miss the days of CVS. CVS for all its warts just freaking works. It could have saved us nearly $100,000 in man-hours in the past 8 months, kept the project one month less behind sechdule, and saved a whole lotta frustration.
To the PVCS VM developers: Have you even freaking looked at CVS? Perforce? ClearCase? Aegis? I realize some of these products are commercial and may be difficult for you to do competitive analysis on, but CVS is the defacto standard in open source development and its under the GPL. So long as you aren’t stealing source code from it it is perfectly acceptable to study. Now that Subversion, darcs, and GNU arch are all stable you might want to also look at those. Oh and don’t forget about git, which the Linux kernel team wrote in 1 month because of the whole BitKeeper fiasco. And lets not forget about BitKeeper, which is currently (in my opinion) the best version control tool available on the market.