A Creaky Old Man’s Defense of Creaky old Code

I’m beginning to wonder whether I’m turning into an “old fogey” of a computer programmer. You know, the elderly fellow who has an office near the server room; the one who always objects to the use of any new technology and goes on at length about “the good old days” you know, back when Real Programmers punched their cards out by hand.

Man with White BeardThe signs seem to be there: I am constantly having this same conversation with various people at my current job. It goes something like this: People say “Oh, why don’t we use ‘FooBaz’!” (some recent piece of fancy technology). Then I respond saying “Hmm… well, we could, but wouldn’t that be incompatible with ‘BarQuux’?” (some old custom-built thing that we have in our infrastructure). The person replies, “Oh, but BarQuux is an old custom-built piece of junk and it’s not standard, and we should just throw it out and replace it with FooBaz.” I try to reply “Can you actually compare the features of BarQuux and FooBaz? Does FooBaz do the things that we use BarQuux for?” but by this time they’re not listening anymore. My willful refusal to realize that technology marches onward and my untreated Not-Invented-Here syndrome have caused them to set the bozo bit.

It’s not that I’m skeptical of new technology just because it’s new; rather it’s that I’m conservative when it comes to things we have already built and are already working. I am not trying to defend the status quo (there is lots of room for improvement) but that I’m trying to insist that people take into account what is already working and not disturb the status quo unless they can show an absolute benefit to it. A particular example should illustrate what I mean.

Some years back, we built our own database connection pooling. We noticed that the only feature of WebSphere that we were actually using was its database connection pool and if we replaced that we could run on an open source stack instead and save some significant licensing fees. So we built our own connection pool. What we built wasn’t particularly fancy connection pooling isn’t really all that difficult a problem so Mark and I just spent a week or two and threw something together that would meet our needs. Over the years it’s gotten a couple of features added, but it has never really received much attention.

So I hear people say “Our current connection pooling is just some home-brewed thing: we should throw it out and use the connection pooling provided by JBoss.” On the surface, that sounds like a very sensible thing to do why not go with a standard solution that’s provided by somebody else and is well exercised rather than something that we threw together in-house? But it isn’t quite that simple. After all, the dinky little database connection pool that we threw together has quite a few features. For instance, it has failover so that if your database goes down it can make its connections to a secondary database instead and continue functioning. It has administrative screens that we built so you can log in and test the DB connection, see the current allocation of connections and a history of pool size over time. It provides simple wrappers around the connections, statements, and result sets that it returns and these do certain logging and also take care to ensure that even if the programmer forgets to properly close their result sets, that Oracle cursors will be released (before we added that one we used to have regular database outages every few months when someone got careless and failed to properly close a cursor and eventually Oracle would run out of cursors and bring the entire system to a halt).

So where are we going to find a database connection pool that has all these features? I’d be perfectly happy to switch to something that’s standard, that’s not something that we built ourselves, but the people who are proposing switching the pool are doing so without any evaluation of the benefits that are in place now from the system that we are currently using. There are genuine benefits: for instance, ours cannot be monitored by SNM; it’s impossible to hire a consultant who is already familiar with our pool.

And I’m not trying to suggest that we have built a database connection system that’s more effective than anything out there on the market that’s simply not true! I’m not even claiming that we have one that is better than what’s available as open source. If that were true, we should probably just release ours as open source and attract a little bit of free support and goodwill. The database connection pool systems that are out there need to be really robust in a way that ours currently isn’t. Ours works with Oracle, with SQL Server, and with an obscure JDBC wrapper for connecting into the Profile banking system but that’s really all it’s ever been tested with because that’s all that we’ve ever used it for. When we make calls into Oracle it’s our standard that we always use stored procedures and always return our results through out parameters not return values. Our wrapper code that closes Oracle cursors wouldn’t work properly if you returned a value from a stored procedure or did a direct SQL query instead.

This is commonly the case with software that is built in-house. They have lots of features that are interesting to you and that you use, and they are significantly lacking in certain kinds of flexibility for features that you do not use in your particular company. Whereas libraries and code for general use MUST have the flexibility (or they couldn’t survive in the marketplace) but they are often lacking in depth of specific features. Now, I am not saying that it’s wise to build everything in-house: there is a an enormous cost to doing so and it is often not the best choice. But people need to be aware that there are advantages to things that have “just been tossed together” in-house. And if you don’t even understand what is in place now, then you’re simply not qualified to be suggesting that we move to something standard.

Comments

3 Responses to “A Creaky Old Man’s Defense of Creaky old Code”

  1. Joe Campbell on 2008-04-01 11:15 am

    In theory I agree with you – but the value point that I think you are missing is, while this does work and in someways works as designed it is starting to get in the way. What I mean by this is

    1) specific to an upgrade paths, when attempting to change drivers to newer versions (which should be a simple drop in replacement) issues have arisen that can not seem to be worked around and are deeply ingrained in the way the pooling ‘was written’. Some have attempted to write this off as an oracle driver issue, you and I both know that it is more likely the home grown code causing issues.
    2) There has been alot of talk on the blog-o-sphere about things being intuitive and how much one has to keep in their head when doing work. How the connection items function in relation to my code should not be an issue I should have to worry about. I should be able to just assume it works as I would expect (it gives me a connection) and move on. Currently this is not true as the connection pool does alot of work that is protectionary – and one could argue ‘good’ but it does it in a way that ‘newer’ programmers are not familiar with and as such have to ‘bootstrap’ into their head when they are working on the project. I would much rather they work on Business functionality and not have to worry about ‘how’ a bit of plumbing works when that plumbing isn’t something they have ‘done’ in the past.

    Yes it works – and yes it has features but it just might be a good time to revisit and look for something that is ‘newer’ that meets a good number of the needs and extend that rather then continue to use something that we wrote FORCING the decision of weather to support it or not.

  2. Emil Lefkof on 2008-04-03 9:33 am

    I definitely see your points. However the real problem is “throwing something together in a week” and then it’s in production. Once it becomes pervasive in production it can be SO hard to extract it later on when you realize an API’s shortcomings. I think reaching the conclusion to home grow a piece of code should only be reached after you have researched the options out there. It is usually the first thing I do when I attack a problem…I ask “did someone else solve this problem already?”. And if the answer is NO then I look at writing something custom.

    What I have learned in the many many clients I have had is it’s not just your company. Lots of companies have similar problems with pervasive homegrown code and it is pretty much 90% of their problems and their cost in maintenance and downtime. A perfect example is my current client. They have decided to log all application to a central log like Syslog. Its high performance UDP based logging so their monitoring group can be watching gobs and gobs of data in one central location and eventing and alarming on that log with something like Tivoli. Great idea right? Sure, except they decided to home grow the API and its a total POS. They spent about 3 months working on it and in the end it was a steaming pile that we poked about 100 holes in on code review. I spent 15 minutes and had the exact functionality to Syslog done in both Log4J and Logback. Two open source API’s with Syslog capabilities already built in. They could have spent 25$ for my time and a great standard solution, instead they spent about 45,000$ over the two months.

    But they are going ahead with their homegrown solution because they can’t justify throwing it out. So now it will be pervasive in the company and when it rears its ugly head with deficiencies it will already be EVERYWHERE in the company. It just makes me shake my head.

    Hope all is well with you guys!

    Bud

  3. mcherm on 2008-04-06 1:02 pm

    Bud:

    Good point — I have certainly seen this pattern before. I guess that the key thing to keep in mind is the ratio between extra functionality provided by the custom system and extra effort required to maintain it.

    What’s most difficult about it is that these things are quite difficult to measure. Take the example from my original essay. The extra functionality provided is things like the failover, the admin screens and the protection against failure to close cursors. How much value do these things add? The admin screens — little to none (any good standard solution should have such tools); the others are harder to estimate.

    In some cases the maintenance effort is difficult to estimate also. For instance, Joe encountered a bug when upgrading oracle which he thinks (but as far as I know never actually demonstrated) may be caused by our connection pooling. So he thinks of it as requiring a significant amount of effort. I have 4+ years of experience using it with only the one reported issue that I know of (and that one just suspected), so I think of the effort as low. We could dig to the bottom of the issue to find out, but the effort it would take to do so is more than it’s worth! So either way we’re just guessing.

Leave a Reply