I work for a bank, and so I worry more about security than most programmers. After all, if a hacker were were truly motivated and competent who would they pick to go after? Probably a bank (the other good option is political or corporate espionage). Recently I saw two security-related stories which, when combined, form my ultimate nightmare: an effective attack for which I cannot imagine a possible defense.

The first story was a research article on microchip design written by some researchers at the University of Illinois at Urbana-Champaign. The researchers asked an interesting question: suppose a smart, competent attacker were to make some changes to the design of a CPU: could they make the CPU vulnerable to some kind of hacking attack without making it obvious? It is not a question I had ever heard before: I am used to thinking of the hardware as fixed and looking for vulnerabilities in the software instead.

The answer was a resounding “Yes!”. They came up with two attacks: each requiring around 1000 extra logic gates, on a chip that has around 1.8 million gates! One attack allowed a specially designed program to read anything in memory (including things like passwords), the other allowed extra programs to run that were invisible to the rest of the computer. Each could be activated by using a special “trigger phrase” — which could even be delivered by just sending a packet to the machine (you don’t necessarily have to hack into the machine to activate it). Once activated, the code you run can be completely invisible to the operating system, and can override any software running on the computer. It could collect passwords (allowing a conventional hack), modify data, replace programs with versions that have backdoors, and if done by a competent hacker, all of this would be impossible to detect.

An undetectable and unstoppable vulnerability: it’s a good thing that CPUs cost millions of dollars to manufacture (even for a very small run) and that we buy our computers from reputable companies like Dell who get their chips from reputable companies like Intel and AMG.

The second story is about a large multinational fraud that was being executed throughout parts of Europe. Over a period of months, MasterCard had noticed a suspicious pattern of credit card fraud throughout Northern England. Someone technically sophisticated was making fake duplicates of real credit cards, but the investigators could not determine how the fraudsters were obtaining the data. By one account, the break in the case came when a security guard at a grocery store noticed a burst of static on his cell phone; by another account the break came when someone duplicated a card that had only ever been used once. Either way, someone took apart one of the store scanners and discovered that it had an extra payload.

Someone had built in a device which stored data on a small portion of the cards that got scanned. Then once a day it would place a wireless call to someplace in Pakistan to drop off the data and pick up new instructions. Presumably from there an international criminal ring produced the fake cards.

Of course the fraud groups at MasterCard are well aware of this kind of scam, and they correlate stolen card numbers back to what stores and what devices those cards had been used in previously. Had it been just this one scanner with modified hardware, the folks at MasterCard would have identified it almost immediately. Scotland Yard noticed that the scanner with the bug in it weighed an extra 4 ounces. Soon investigators with scales were speeding to stores throughout Europe, checking for card-readers that weighed 4 oz more than they should. They found hundreds of compromised machines - even in major retailers like Wal-Mart, and there was no sign of tampering.

It turns out that the bugs were inserted not by agents in the store, but by someone in a factory in China back when the devices were first being manufactured. The criminals were quite smart in how they leveraged the information: never stealing so much from one site that it would give away the game. And they made off with at least 50-100 thousand dollars, perhaps more. No matter how careful the people at the stores were, they had no chance against devices that where compromised in the supply chain before the store ever received it.

. . .

So here is my nightmare senario: someone decides to target a bank (’cause that’s where the money is). They manufacture chips that allow them access below the operating system level, which is immune to any attempt to use virus checking, encryption or other such defenses. And they insert these into the supply chain somehow (how big a bribe would it take to get a couple of hours unobserved in a Dell warehouse?). Then they get competent programmers to slowly siphon off cash at levels that won’t be noticed. Sure, the costs of such a scheme probably run in the millions of dollars, but organized crime in places like Russia certainly have access to this kind of money.

And I cannot imagine anything that I, as a technical expert at a bank with a concern for security, could do to prevent this attack.

WARNING: Some local district lines were moved at the last minute (AFTER the primary). To verify your proper polling place, use this link.

President of the United States:

• Bob Barr/Wayne A. Root (Libertarian)
• John McCain/Sarah Palin (Republican)
• Ralph Nader/Matt Gonzales (Independent)
• Barack Obama/Joe Biden (Democrat)

Given the billion or so that has been spent by the candidates plus untold more spent by news media, you have almost certainly made up your mind already, and my opinion here means little. Suffice it to say, that I have great respect for John McCain — he is a man I would have voted for in a different election. But the positions he has taken to satisfy the Republican base disturb me, and his selection of an abundantly unqualified candidate for vice-president is unfortunate. On the other side, Barack Obama is exactly what we need. He is a charismatic leader, with an interest in bringing the country together. At the same time, his positions on many issues are much closer to mine than most politicians. He gives every indication of being a highly intelligent and thoughtful person — even in the book which he wrote before he considered entering politics. And his background as a community organizer and a constitutional lawyer are just what this country needs if we are to have any hope of recovering from the appalling damage of the past 8 years. I have contribute time and money to Obama’s campaign, and I will be out working for him on election day.

US House of Representatives

• Joe Sestak (Democrat)
• W. Craig Williams (Republican)

Joe Sestak is a former senior military leader (3-star admiral!) who has been serving for a term as a Democratic representative from a Republican-majority district. His stated priorities for this election are (1) economics, (2) health system, (3) economics, (4) energy (he wants to promote alternate energy AND drill more oil) and (5) defense. Frankly, I like this set of priorities. I have met Joe Sestak and I find that although I definitely do NOT agree with him on many issues, he was reasonable and thoughtful, had real reasons behind his choices, and was willing to explain those reasons to a constituent.

Craig Williams is his Republican challenger. Craig’s main points seem to be (1) Sara Palin endorses me, (2) we need to drill more oil to solve the energy crisis, (3) Government is too big and should regulate less — for instance, they shouldn’t allow airline traffic over our district [no... this doesn't make sense], (4) Health care should be fixed by privatizing it [yes, that IS what we already have]; specifically we should provide prescription drugs for senior citizens, “emphasize personal responsibility”, and put an end to “frivolous lawsuits”.

On the whole, Craig Williams’ proposals do not even seem sensible, and Joe Sestak has proven to be quite reasonable and effective in his time so far. I intend to support Joe Sestak.

Attorney General

• Tom Corbett (Republican)
• John M. Morganelli (Democrat)
• Marakay J. Rogers (Libertarian)

John Morganelli tells the League of Women Voters (and posts on his own website) that his #1 plan is to “Pass a gang statute that makes gang membership a crime so as to attack the gangs before the commit their next murder or drug deal”. Perhaps he does not quite understand what the constitutional guarantee of freedom of association means. Another proposal of his is to “Abolish parole for violent criminals.” Our prison system assigns one to serve a range of years for a crime depending on the judge and jury’s evaluation of the seriousness of the crime and on the individual’s behavior afterward. To completely ignore the latter in the name of “punishing violent criminals” is foolish and unwise.

Tom Corbett has been Attorney General for the past several years. As far as I can tell, all evidence suggests that he has done a reasonably good job.

Marakay Rogers is a bit more unusual — she has a history of running under various third parties. She has managed to get endorsements for various offices from the Libertarian party, the Green party, the Reform party and Ralph Nader. She feels evidence of racially unbalanced enforcement of the death penalty needs to be addressed. She questions Pennsylvania’s high rate of incarceration (3rd highest in the nation). She objects to the REAL ID act as violating PA constitutional privacy rights, and speaks out for the rights of immigrants.

I would feel very comfortable voting for Tom Corbett, who is qualified and competent. I would also feel comfortable voting for Marakay Rogers. Polling shows something in the neighborhood of 43% for Corbett to 31% for Morganelli to 3% for Marakay.

Auditor General

• Chet Beiler (Republican)
• Betsy Summers (Libertarian)
• Jack Wagner (Democrat)

Betsy Summers has not bothered to put up a website for her campaign, to populate the web page that her party provides for her, or even to answer questions for the League of Women Voters, so she is not even worthy of consideration.

Chet Beiler has a history as a successful businessman in a number of enterprises that he grew and ran successfully. He does not have a history of political involvement and bills himself as “not a career politician”. Jack Wagner is the currently serving Auditor General. I have read at least 5 different newspaper endorsements of Jack Wagner (I couldn’t find any newspapers endorsing Beiler). All of them said essentially the same thing: “In his current term, Jack Wagner has done audits that revealed serious issues (curiously, each newspaper seemed to list different audits) and has gone after mismanagement of funds. He should be returned to office to continue the good work.” With widespread endorsements like this, I intend to support Jack Wagner.

State Treasurer

• Tom Ellis (Republican)
• Berlie Etzel (Libertarian)
• Robert McCord (Democrat)

Tom Ellis failed to answer even the simplest of questions from the League of Women Voters; Berlie Etzel did that much but seems not to have a website (or event to have populated her page within her party’s site). Thus, these candidates were more difficult to research (Robert McCord has plenty of information available about himself and his positions.)

From what I can tell, Tom Ellis and Robert McCord have similar backgrounds. They are from the same area, both have political experience and business backgrounds, and both would be new to this office. McCord appears to have the ability to raise much more money - 23x as much - and to spend vast sums of his own money as well, but I mistrust that as an indicator in this case; I think it is likely to be due to large donors he knows rather than broad support.

Without any strong reason to prefer either candidate, I will probably select a candidate based on party endorsement (in my case, choosing McCord as the candidate of the Democratic party).

State Senator

• Daylin Leach (Democrat)
• Lance Rogers (Republican)

Lance Rogers and Daylin Leach must have spent a fair amount of money because I got several mailings from them. Unfortunately, all of the mailings were either attack ads or responses to attack ads. The gist of it seemed to be that Rogers accuses Leach of having tried to weaken laws against drunk driving, but independent 3rd parties seem to claim that Leach was actually just changing the laws in an attempt to strengthen not weaken them. Frankly, I’m not impressed either way, and I wish they’d found a better way to spend their campaign dollars.

Rogers says his top issues are: (1) cutting government perks, (2) cutting taxes and spending, (3) fixing healthcare, (4) promoting jobs, (5) promoting green space and green construction/energy, and several others. Leach says his top issues are: (1) education, (2) environment, (3) political reform, (4) healthcare (he worked to get prescription drugs for those receiving health insurance from the state, and he works on women’s health issues).

On the whole, there is not a lot there on which to base a decision, but I tend to favor Leach’s approach. And I don’t like politicians who start very dirty negative campaigning (which is how independent parties characterized Rogers’ mailings). So I’ll be voting for Daylin Leach.

State House of Representatives

• Stephen D’Emilio (Republican)
• Greg Vitali (Democrat)

Greg Vitali is a hero of mine. I still recall when the Pennsylvania legislature voted themselves a massive, undeserved and blatantly unconstitutional pay raise in the middle of the night. (The constitution forbids the legislature from raising its own pay during its term, so they effectively raised the amount of expenses allowed and said they wouldn’t be expecting to see any receipts.) Greg Vitali spoke out against this at the time. He spoke out so vocally that his own party stripped him of committee positions and power in an attempt to silence him — but he was not silent. Greg vitali has my vote and will continue to have my vote for a long, long time. It helps that I agree with him on many other issues, but ultimately, integrity is the most important issue for me.

And if you didn’t find THAT convincing enough, read our local newspaper’s article on the two candidates (which does not endorse a candidate).

Referendum on Bonds for Water and Sewer Improvement

Do you favor the incurring of indebtedness by the Commonwealth of $400,000,000 for grants and loans to municipalities and public utilities for the cost of all labor, materials, necessary operational machinery and equipment, lands, property, right and easements, plans and specifications, surveys, estimates of costs and revenues, prefeasibility studies, engineering and legal services and all other expenses necessary or incident to the acquisition, construction, improvement, expansion, extension, repair or rehabilitation of all or part of drinking water system, storm water, nonpoint source projects, nutrient credits and wastewater treatment system projects?

I intend to vote “Yes” on this measure. I have read newspaper editorials both for and against the measure. It was my opinion that the “yes” editorials backed up their position with sensible arguments about the state of our water and sewer infrastructure and an analysis showing that this would not affect the state’s bond rating, while the “no” editorials backed up their position by simply saying “government spending is bad… they should take the money from cutting government waste”. But ultimately, these were not what swayed my opinion — the economy was. During a recession, governments should be using their creditworthiness to borrow in order to pay for public works projects. (And during boom times they should pay down debt and set up rainy-day funds.) This kind of behavior is what smooths out the extremes and restores our economic stability, from which we all benefit.

costs.jsp

<% Cost[] costs = CostHelper.calculateCosts(loanData, currentDecision); %>
<h3>Current Costs:</h3>
<table border="0">
  <tr>
    <th>Cost</th>
    <th>Amount</th>
  </tr>
  <% for(int i=0; i<costs.length; i++) { %>
    <tr>
      <td><b><%costs[i].getName()%></b></td>
      <td>$ <%costs[i].getValue()%></td>
    </tr>
  <% } %>
</table>
<p style="text-align: center; font-style: italic">
  <a href="costHelp.html" 
      onclick="
        window.open(this.href, 'Help');
        return false;
      "
  >Explain these</a>
</p>

And it was confusing. But after a time we became enlightened and we realized that we should separate out the business logic from the rest of the presentation. We used tag libraries or velocity macros or any of a host of other technologies, and the code looked something like this:

CostController.java

Cost[] costs = CostHelper.calculateCosts(loanData, currentDecision);
modelAndView.addObject("costs", costs);

costs.vm

<h3>Current Costs:</h3>
<table border="0">
  <tr>
    <th>Cost</th>
    <th>Amount</th>
  </tr>
  #foreach( $cost in $costs )
    <tr>
      <td><b>$cost.name</b></td>
      <td>$ $cost.value</td>
    </tr>
  #end
</table>
<p style="text-align: center; font-style: italic">
  <a href="costHelp.html" 
      onclick="
        window.open(this.href, 'Help');
        return false;
      "
  >Explain these</a>
</p>

And it was confusing. But after a time we became enlightened and we realized that we should separate out the styling from the rest of the presentation. We used CSS markup in separate files, and the code looked something like this:

CostController.java

Cost[] costs = CostHelper.calculateCosts(loanData, currentDecision);
modelAndView.addObject("costs", costs);

costs.css

.costName {
  font-weight: bold;
}
.helpLink {
  text-align: center;
  font-style: italic;
}

costs.vm

<h3>Current Costs:</h3>
<table border="0">
  <tr>
    <th>Cost</th>
    <th>Amount</th>
  </tr>
  #foreach( $cost in $costs )
    <tr>
      <td class="costName">$cost.name</td>
      <td>$ $cost.value</td>
    </tr>
  #end
</table>
<p class="helpLink">
  <a href="costHelp.html" 
      onclick="
        window.open(this.href, 'Help');
        return false;
      "
  >Explain these</a>
</p>

And it was confusing. But after a time we became enlightened and we realized that we should separate out the dynamic JavaScript from the rest of the presentation. We called this unobtrusive JavaScript, and the code looked something like this:

CostController.java

Cost[] costs = CostHelper.calculateCosts(loanData, currentDecision);
modelAndView.addObject("costs", costs);

costs.css

.costName {
  font-weight: bold;
}
.helpLink {
  text-align: center;
  font-style: italic;
}

costs.js

$(function() {
  $('.popup').click(function() {
    window.open( $(this).attr('href'), 'Help');
    return false;
  });
})

costs.vm

<h3>Current Costs:</h3>
<table border="0">
  <tr>
    <th>Cost</th>
    <th>Amount</th>
  </tr>
  #foreach( $cost in $costs )
    <tr>
      <td class="costName">$cost.name</td>
      <td>$ $cost.value</td>
    </tr>
  #end
</table>
<p class="helpLink">
  <a href="costHelp.html" class="popup">Explain these</a>
</p>

And it was better. Things get a little bit confusing because each screen is rendered from several different files instead of from a single one, but separating out the concerns of business logic, styling, and dynamic behavior made each piece easier to understand and work with.

Recently, though, I became enlightened, and realized that we still have too many things going on in the same file. The HTML file still contains two independent concerns: the definition of the structure and the actual text. These facets are not really related, and they tend to be edited by different people. (HTML designers don’t necessarily produce the marketing copy and other text.) So this is a perfect candidate for another separation of concerns. That’s why my current project is organized like this instead:

CostController.java

Cost[] costs = CostHelper.calculateCosts(loanData, currentDecision);
modelAndView.addObject("costs", costs);

costs.css

.costName {
  font-weight: bold;
}
.helpLink {
  text-align: center;
  font-style: italic;
}

costs.js

$(function() {
  $('.popup').click(function() {
    window.open( $(this).attr('href'), 'Help');
    return false;
  });
})

cmstext.properties

cost-title: Current Costs:
cost-col1-label: Cost
cost-col2-label: Amount
cost-help: Explain these

costs.vm

<h3>#cms("cost-title")</h3>
<table border="0">
  <tr>
    <th>#cms("cost-col1-label")</th>
    <th>#cms("cost-col2-label")</th>
  </tr>
  #foreach( $cost in $costs )
    <tr>
      <td class="costName">$cost.name</td>
      <td>$ $cost.value</td>
    </tr>
  #end
</table>
<p class="helpLink">
  <a href="costHelp.html" class="popup">#cms("cost-help")</a>
</p>

The code that processes this allows no markup in the cmstext.properties file — any metacharacters will be properly escaped. So in principle it would be easy to give this file directly to the business to edit without IT help — or even to allow them to edit it on the live website without needing a QA cycle. The down-side is that we have yet another file to deal with: time will tell whether the benefits outweigh that cost.

SQL Injection

The most venerable is the SQL-injection attack (and related attacks for things other than databases). This is the danger that data entered by users will be treated as meta-characters not just as text, and will allow a visitor to your site to execute arbitrary code in your database (or some other system). Nothing illustrates this kind of attack better than this XKCD cartoon:

XSS

There are XSS attacks - which stands for “Cross Site Scripting” (the acronym “CSS” was already taken). XSS is the danger that your site might serve up some content which was entered by a user (or perhaps loaded from a 3rd party site). Normally, this makes sense: whole empires have been built on the concept of allowing users to customize their own pages on your site. But browsers assume that any code served up by your site is trusted, and will allow it to do things like accessing you site’s cookies. Particularly if users can get others to view their content, this can be quite dangerous. The first line of defense is to carefully sanitize and properly escape all content from users that it intended as plain text. As for content from users that is intended to contain markup, some sites just prohibit this, while others can use a solution like Google caja, which provides a secure sandbox for untrusted code to run in.

CSRF

XSS attacks exploited the fact that the user (or the user’s browser) trusted content coming from your website (even if that content wasn’t written by the site owner). The converse is a CSRF attack which exploits the fact that the site trusts content coming from it’s user’s browser. One of your users visits a malicious site which contains some sort of link to your site — preferably a link which causes some action to be taken. The request comes from the user’s browser, it has the user’s cookies and everything, so your site thinks the user intended to send the request. The idea is for a site containing pictures of cute cats to trick your banking site into transferring money to someone’s account. There are several defenses such as always using POST (helps, but doesn’t fix the problem), a per-session unique ID, and double-submit cookies.

Clickjacking

The latest hot issue in the web security community is “clickjacking“. This is a variant of CSRF in that it occurs when your site’s user navigates through a malicious website. But instead of the foreign site trying to generate a “click” on your website by embedding an image or executing some JavaScript, the foreign site embeds your site, perhaps within an IFrame. They then cover it up so you can’t SEE it, but when you click you perform an action on the attacked site. You can see an innocuous example at http://noscript.net/getit, where the “install now” button installs from Firefox’s site, not from noscript’s site. The most evil versions will continually move the attacked site so it stays under the cursor… no matter where you click, the button you don’t see will be under your cursor. Some of the details of this exploit have not been released yet, so it is too early to make clear statements about how a website (or a user) can protect themselves.

Plenty of others

I am sure that there are other fundamental forms of attack against web applications. Go ahead and add your own items in the comments below!

As far as I can tell, there IS no universal solution to this problem — only particular solutions that optimize for certain kinds of application behavior. In our case (we’re building an online mortgage application) nearly all of the interaction will occur with a logged-in user, or at the very least will care whether the user is logged in, which almost certainly means reading user information from the Session. So almost every request will be reading from the Session. However, many of the requests will be used for display not update — they will be reading from the Session but not updating it.

That observation immediately suggests the use of immutable data structures. Immutable data structures are a technique for concurrent programming that has been known for many years, but it has received a renewed spurt of interest recently sparked largely by the 1996 thesis of Chris Okasaki and others who expanded on it. What everyone had always known was that if a data structure is immutable (can’t be changed) then multiple threads can safely read it - essentially each thread gets a “frozen” copy of the data as it looked at the moment the thread began working with it. What Chris’s thesis added was a coherent way of analyzing the performance behavior of immutable data structures and some clever ideas that made them nearly as fast as traditional mutable data structures.

So if we use an immutable data structure, then we can be sure that all requests that are just reading data and not updating it can proceed at full speed. They will receive a copy of the data as it looked when the thread began processing the request. Any changes to session that happen during the time that the request is being processed will not be seen, but that is probably the best behavior we could ask for — it is equivalent to assuming that the read-only request was executed so quickly that it finished before the update occurred.

So we’ll use some sort of an immutable data structure to hold our session data — perhaps it’s a linked list, or an immutable map, so far it doesn’t really matter. And we’ll handle requests that just read data by grabbing a copy of the structure once — that way no matter how logn the request takes, it will be seeing a single, coherent view of the session data. For requests that must both read and write data we can do the reading the same way: take a copy of the data structure at the beginning of the processing and do all reads from that copy. The difference is that at the end of the service we will have done some writes to the data — with an immutable data structure this means we’ll have generated a new structure which is just like the original except for the small number of changes we have made. The question remaining to be solved is what to do with this changed data.

The “obvious” solution is to just update the pointer in the HTTPSession to point to our newly changed data, but this won’t work. This won’t be safe in the face of threading issues. Imagine two requests that both do writes are occuring simultaneously: one modifies the customer’s current requested loan amount while the other modifies the customer’s address. Whichevre is processed second may overwrite the changes of the first — so one thread would update loan amount then the second (unaware of the first) will set back the loan amount while updating the address. In the end, the user thinks the loan amount has been changed, but it hasn’t.

I can think of two solutions here: “one-writer-at-a-time” and “merging”. The general idea behind “one-at-a-time” would be for any thread that intends to make modifications to obtain a lock before beginning. Then we would allow as many “read-only” threads as desired, but only one “read-write” thread per user session could execute simultaneously. If the application consists of lots of simultaneous AJAX calls to obtain data but very few that perform updates, then this is a simple and straightforward approach.

To make this idea more clear, let me define an API that would achieve “one-writer-at-a-time” semantics. In the Session, we would place a single ThreadsafeSessionData object, and all programmers would be enjoined not to use the HTTPSession.putValue(). A method ThreadsafeSessionData.getReadonlyData() would return an immutable Map containing immutable data objects with the contents of the session. Servlets that only needed to read data could call this once to obtain a consistent view of the data. Another method, ThreadsafeSessionData.getEditableData() would obtain a lock on the session (blocking if needed) and return an EditableData object. The EditableData object would have a getReadonlyData() method returning the same immutable Map, but it would also have a setValue() method for editing the data (by creating a new immutable Map). And it would have a commitChanges() method which would take the current immutable Map, store it in the ThreadsafeSessionData object, release the lock, and then lock down the EditableData object (no calling setValue() after commitChanges()).

It is possible to go even further than this. Multiple writer threads could operate at the same time, so long as they are not modifying the same piece of data, or relying on a value which is simultaneously being modified. The concept is rather like version control systems that use merging rather than locking. But it requires merging the data after a “commit”, which adds a great deal of complexity to the solution. As a programming exercise, I think it would be fun to develop, but finding a reliable means of merging changes is challenging enough that I think for production use I’ll stop with “one-writer-at-a-time”.

However, threading issues for web servers are not limited to the fact that there are multiple simultaneous users — it is also possible to be processing multiple HTTP requests a single user at the same time. In fact, this happens all the time with requests for images. Most browsers today download images 4 at a time (or more) after the basic page has loaded. Fortunately, downloading images tends to be a read-only operation with little opportunity for threading problems. In “the old days” (in our industry that probably means last year) users would mostly view one page, wait while it finished loading, then follow a link or submit a form to a new page. So images were the only thing happening “simultaneously” for a given user, and there was little danger of data corruption.

The advent of AJAX and rich internet applications has changed all that. Modern websites don’t consist of a series of pages and forms, they consist of an interactive environment where the user modifies controls which communicate with the server. Think of Yahoo Mail, Google Maps, or NetVibes. And that means that the threading issues which formerly caused very little grief are now poised to become a major issue.

What, you may ask, is so special about multiple requests from the same user? If safely handling multiple simultaneous requests from different users is a solved problem, why would safely handling multiple requests from the same user be any different? Well, two threads can run safely as long as they do not share any objects. Avoiding instance variables in the servlet (or implementing SingleThreadModel, but that’s not as good a solution) prevents one possible kind of shared object. Avoiding static instance variables (except possibly those designed for threadsafe use) prevents another. And the specification for servlet says that the container is responsible for ensuring that the other objects available to the servlet are safe to use… except for one: the session.

The session (usually the HttpSession) is used to carry around data from one call to the next within a user’s session. (This is needed because the HTTP protocol is inherently stateless, but rich internet applications are quite stateful — the state is stored in the session). Some calls (like those downloading an image) won’t use the session, and those have no threading issues. But many calls will read some information from the session and write other information. If multiple threads are reading from and writing to the same object and that object is not specially designed to be threadsafe (and HttpSession is not), then it is a recipe for threading disasters.

So what is the solution? I will try to describe the some common “solutions” and why they are not satisfactory for the project I am starting on at the moment. Then (in a future post) I hope to sketch out a more complex but more capable solution which does meet my current needs.

The most common solution, the one used by the vast majority of Java web application programmers is called “pretend it doesn’t matter”. Whether through ignorance (probably), or a belief that “it won’t matter anyway”, they simply don’t do anything about the threading behavior of session. And that most of the time, they get away with it - after all, data corruption due to threading behavior is rare problem, occurring in an indeterminate fashion usually impossible to replicate in a test case. So the program “nearly always” works, and the few glitches are just ignored. For me, this approach (though tempting) is simply too dangerous: one should never rely on undefined behavior, particularly not if you work for a bank: people tend to get peeved when their bank makes small, random data errors.

Another solution is simply to avoid storing any data in the user’s Session. While this would work, it rather undermines the usefulness of the display tier. For most non-trivial web applications we don’t want to have to re-fetch data from the database on each and every HTTP request — we need to cache various pieces of information in the Session.

The next simplest solution is to obtain a lock on the user’s Session (or some standard object in it) before reading from or writing to it. The advantage of this approach is that it is straightforward, and that it is guaranteed to avoid concurrent access. There is even support for it in several major frameworks, such as the synchronizeOnSession property on the AbstractController in Spring MVC. The disadvantage is that different HTTP requests are handled one-at-a-time by the server. This means that we don’t get to take full advantage of the massive multi-threaded server that is serving our application: sure, all hardware allows us to support hundreds of simultaneous users, but it won’t speed up the experience for any one user. It also means that if we issue several requests in quick succession, they will “line up” at the server… I have seen cases where one busy page “locked up” the server with a whole series of requests so that even if the user navigated away to a different page there was still a noticable delay while the server performed the already-queued requests. And finally, it would prevent one thing which I certainly want, which is for certain long-running requests to go on in the background while the user’s interactions continue. So this solution, while simple and elegant, won’t meet my needs.

The next obvious idea is that instead of locking on the entire session, you can just lock on specific bits of data. So if the session contains variables “loginSuccessful”, “userName”, “cachedUserData”, and “cachedUserPreferences” (the latter two being complex structures in their own right), then instead of obtaining a lock on the entire Session before reading or writing any of the variables, you could instead have a policy of getting a lock on an individual object before accessing it. Unfortunately, this approach is fraught with problems. One is the race condition for object creation: if two threads both check for a “cachedUserData” and find that it is missing, they might both try to load and save it simultaneously. This can be avoided (somewhat awkwardly) by having a separate lock object for every possible variable and creating all of the lock objects at Session initialization.

Unfortunately, there is a more serious problem: that of deadlock. Suppose two different HTTP requests both need to access the “userName” and the “cachedUserData”. If one locks the “useName” while the other locks the “cachedUserData” then each tries to obtain the other lock, they will deadlock, and neither can continue (nor can any other thread using these locks!). The only solution that I know of for this problem is to always obtain all locks in a fixed order… but that is particularly difficult: in the general case it requires global knowledge of ALL code in the entire application.

And weighed against these tricky threading difficulties, the benefits of locking on specific bits of data are not very impressive. There will often be a single piece of data (”cachedUserPreferences” perhaps) that is used by nearly every command - in which case it reduces to being essentially a lock on the session.

So how CAN one solve this dilemma? I am hoping to come up with a solution by using immutable (copy-on-write) data structures. But this is long enough already: that will have to be a topic for a future essay.

I am sending email copies of the technical postings to this blog to an email list of people at my work (I suppose I’d accept others also) who expressed interest. It started as a bonus goal project. Here is the process I came up with for sending the email from Outlook:

1. First, publish the blog entry to the web normally.
2. Launch Outlook
3. Go to “Tools > Options > Mail Format” and set the email format to HTML while disabling the use of Microsoft Word for editing emails.
4. Go to “View > Toolbars > Web” to enable the web toolbar.
5. Navitage to the website (http://mcherm.com/) and follow the link to the story to be emailed. Manually add “?stylesheet=outlook” to the end of the URL and hit return to view it.
6. Go to “Actions > Send Web Page by E-mail”. This will launch Outlook’s terrible, almost unusable HTML editor. The page will look entirely wrong - for instance, it will ignore the stylesheet. Fortunately, only one thing really has to be fixed (everything else will look OK when viewed. That one thing is to delete the first three characters at the beginning of the page. (Sometimes is messes up quotes too… you can check the body of the story.)
7. Now copy the mailing list and add it to the BCC field. Be sure it’s the BCC field, not the CC or TO field, to protect the privacy of the message recipients.
8. Fix the email subject. It should read “Technical Essay Series: <title-of-article>”.
9. Hit send.

But those are just the new innovations — more impressive is the list of “standard” browser features that Chrome supports. It correctly renders the highly complex HTML and CSS standards, passing the ACID2 test (it fails ACID3, but still manages to attain a higher score than Firefox or Internet Explorer). It supports the HTTP protocol, with all its complexity like content negotiation and compression, as well as HTTPS, FTP, and probably a few others. It has special handling for downloads (showing downloads in progress, locating them, etc. It implements the entire JavaScript language, and can correctly display hundreds of thousands of popular websites. It dynamically generates possible options as you type in the URL bar (like the recently released Firefox 3 “awesome bar”). It comes packaged with abilities or plug-ins for displaying flash, PDF, and all common graphics formats. All in all, it is a full-featured browser, quite capable of competing on features with Internet Explorer 7, Firefox 3, Safari 3, or Opera 9. Whether it will compete in market share is something only time will tell.

If you stop and think about it, this is an astounding accomplishment. A browser of this complexity is a HUGE piece of software — Google Chrome is 1.6 gigabytes of code (that would take a typical person 20 years just to type in, much less debug!) Now, Google has a lot of extremely smart people working for them, but even so, this is an incredible achievement. Consider: at one point, Netscape was the “king of the internet”, the company poised to take on Microsoft for domination of the computer industry. Then Netscape decided that most of their browser needed to be rewritten. Three years later, they released it… and in the meantime, Microsoft had won the first of the browser wars, and Netscape was an irrelevant subdivision of AOL.

Given Google’s penchant for secrecy, it is difficult to say just how long they spent developing Chrome, but it is doubtful that it was under development for 3 years (perhaps more like 2). And many of the now “standard” features listed above didn’t exist when Netscape 6 was written. So what is Google’s secret, that allowed them to build this incredibly complex piece of software in an amazingly short period of time?

The answer is that Chrome is built on open source software. In fact, it uses at least 25 different software libraries! And we’re talking about simple little helper utilities, the open source dependencies include “webkit” (the same HTML rendering engine that powers Safari), NSS (which provides all of the encryption needed for SSL and more), pthreads (which supports building a multi-threaded application) and sqllite (an entire database).

This is the new face of software development. Open source is no longer an obscure, pseudo-communist hippie movement dedicated to putting professional programmers out of business (it never was, despite the fact that Microsoft once wanted us to believe it). Instead, it is an essential part of building modern software applications.

You probably have antivirus software protecting your computer from malicious computer viruses — in fact, you’d be a bit nervous about connecting to the internet if you didn’t have it. After all, viruses can make everything run more slowly, take over your computer and use it to send spam, or worst of all they can even delete files and destroy data. There are even a few viruses that take your data hostage — encrypting it, deleting the original, then offering to decrypt it if an anonymous payment is made to a certain account.

It is less common these days (hard drives have gotten much better), but occasionally you’ll have a catastrophic failure — your hard drive will crash with no way to recover it. This can be a real disaster: the cost of a new hard drive is nothing compared to the loss of all your precious data.

Or it’s even POSSIBLE (unlikely, to be sure), that while you were cooking dinner your five-year-old got onto the computer you had left on (despite KNOWING he wasn’t supposed to) and sort of accidentally dragged into the trash a folder containing some files — files you had been working on for quite a long time. And, unlikely though it sounds, it’s possible that you didn’t notice this until AFTER you had already emptied the trash can and deleted the files forever.

Of course there is one solution to ALL of these problems — backups. Everyone knows that they ought to back up their files. But not everyone gets around to doing it; in fact nearly everyone doesn’t - until right after they have an accident and lose data. In fact, I am guessing that you, yes YOU have data someplace (probably at home) that isn’t backed up properly.

There is an obvious “right” way to do backups; just follow a few simple rules. First, the backups have to be frequent — a 9-month old backup isn’t going to be very useful; every few days is about right for most people. Second, you MUST test your backup. A backup process, even one designed with the best of intentions, can’t be considered safe unless you have tried doing a restore from it. And succeeded. It doesn’t matter whether the backup is to CDs, tape, a spare hard drive, or over the internet, but you should store the backups someplace separate from the computer (otherwise they won’t be much help in case of a housefire).

And doing all this is a pain… particularly the part about doing it regularly. Murphy’s law says that you won’t lose data until sometime after you finally become lax and start skipping your backups. It is FAR better if the backups can be done automatically… so if you are lazy like the rest of us it will still get backed up. But for most of us, setting up some automated system is even harder than doing backups.

So what you really need is some product which will automatically back up your files for you every few days. It should be trivially easy to use, should take almost no time to install, and should work even if your computer isn’t on or net-connected 24/7. It should make restoring files a breeze (and not something you do only in case of catastrophe) and should automatically warn you if backups are NOT working for some reason. It should not require purchasing any special tape drives or other hardware. Oh yeah, and it should be free, or at least cheap.

Fortunately, such a product exists! Actually, several such products exist, but I’m just going to talk about my favorite which is called “Mozy” (it’s for Windows and Mac… you Linux users will need to find a different alternative). Mozy’s software installs quite easily. Once installed, it asks you to create an account: home use is FREE if you need to back up less than 2GB, and costs $5/month for unlimited amounts. (They have small business backup solutions also.) It will ask you to specify which directories need to be backed up and will send the data over the internet to Mozy’s servers. The documentation says that it is encrypted, so Mozy themselves cannot access your data without the password. (In case you were wondering, that’s a good thing.)

After the initial setup, Mozy runs without any supervision. It waits for a time when your internet connection is up and running but you are not currently working on your computer, and then it sends the changes (only the changes) from the backed-up directories over to Mozy’s servers. If it does not manage to do a backup within 7 days, then it begins to warn you that the backups are not current. And any time that you want to get files back, you can review the files stored on the server and restore any amount from a single file to the entire backup set.

There are a couple of features that I would want in an ideal backup system which Mozy lacks. I would want the entire disk backed up, including the OS so that I could completely swap in a new disk if needed. And I would want all versions of a backed-up file to be saved, not just the latest, so I could return to previous states of things rather like a version control system. But Mozy provides the most important features: it backs up my data, and it is very easy to use (good, because I’m too lazy to use it otherwise).

So now we come to the wager. At this point, as I predicted, you are thinking to yourself “Gee, this Mozy sounds like a good idea. I should really install it on my own computer.” But it’s one thing to THINK about it, and quite another to actually go DO it. There is a certain mental inertia to overcome. So I’m placing a small bet - shall we say $5? - that you won’t actually step up and do it. If I lose… if because of reading this article, you actually DO go install Mozy or another automated backup tool, then post a reply using the reply box below, and I’ll pay up on my bet. (For the record: Yes, Mozy does have an affiliate program but No, I am not participating. I get no kickbacks with this deal - just the warm fuzzy feeling of having saved someone else from losing their data. (Offer good only for the first 12 days after this is posted. Offer may be rescinded if this gets posted to the front page of Slashdot, Digg, reddit, or the like.))  You could have your data safe AND win $5 at the same time… what are you waiting for? Or are you too lazy?

Programming is like doing mathematical proofs.

Before the field of programming really came into existence, there was a metaphor for it: programming was like doing mathematical proofs. And before the invention of the electronic computer, it was a fairly good metaphor. After all, the first practical von Neumann machine wasn’t built until 1949, while Turing and Church were busy inventing the mathematical foundations of computer science in the early 1930s. The metaphor accurately captured the sense that programming required the same sort of logical precision as a mathematical proof. But it misleadingly implies that programs are perfectly error-free (most of us are not Knuth), that creating them is a solitary process, that they are inaccessible to all but the brightest minds, and many other misleading implications. Fortunately, this analogy has been thoroughly abandoned.

Programming is like writing a book.

The next analogy I know of dates from the earliest days of programming on computers, and it is still with us today in some of the language we use. We speak of “writing” a program, or refer to the “author’ of a particular piece of code because one of the early metaphors was that programming was like writing a book.

At a very shallow level, this metaphor is obvious: programming involves typing at a keyboard, just like writing. The metaphor actually has a bit more to offer than that (particularly for describing small programs and one-person efforts). Just like writing a book, the programmer begins with an outline or high-level plan. The programmer then works out the high level details in her head, but most of the details are not determined until the pen hits the page (or keyboard). Much like a book, it is important to have a good high-level structure (plot), but in the end it all comes down to proper execution of the details.

But the writing metaphor is also responsible for some of the most pernicious misunderstandings by non-programmers. It completely fails to capture the way in which the difficulty of programming scales more-than-linearly with the size of the program: writing a book that is 2x as long will take 2x as long but writing a program 2x as long may take 3x or 4x the effort. When poorly informed managers attempt to measure productivity in Lines Of Code, they are being mislead by the writing analogy. The writing metaphor fails to capture the vital role of testing and debugging in programming - which are nothing like editing a piece of writing. And perhaps most seriously of all, the writing metaphor does nothing to describe the fact that programming today is nearly always a team activity.

Programming is like building a bridge.

The “programming as writing” metaphor still lingers in some areas (particularly when speaking about the “lone programmer”), but the essential failure to capture the team nature of software development caused industry to adopt a different metaphor. They needed a metaphor for a large group of people coordinating to create something, and they cared deeply about the processes for measuring this process and making it repeatable; the analogy they chose was that programming is like an engineering project… the construction of a building or a bridge. The language of software projects is strongly influenced by this analogy: we speak of “building” our code, of creating scaffolding to test things.

In many ways, the metaphor is extremely effective. It does a particularly good job of describing the way in which teams of developers are managed. The mechanisms from managing construction projects, from managerial structures to Gantt charts have been applied to managing software projects, with a certain amount of success. The analogy also captures some of the role that solid requirements and good planning play in a software project.

But the bridge-building metaphor also fails in some important ways. One of the things that corporations want most from their programmers is predictability… most would willingly pay far more in time and cost if they could just be assured of predictable delivery. Yet a substantial portion of software projects fail, and it can often be attributed to attempts to run a software project as if it were the building of a bridge. But when building a bridge, one usually follows well-established engineering practices: most bridges follow an existing design. Software, by it’s nature, is constantly doing something NEW (otherwise, you’d just re-use the existing code). So problems like code running 100x slower than expected or detailed, or written requirements that bear little resemblance to what is actually needed… these problems are common in programming, yet are not even hinted at by the construction metaphor.

Buildings or bridges have ribbon-cutting ceremonies: the half-built bridge or incomplete building is unusable while work proceeds until a point when the project is finished and the edifice is available. This mindset has led to “big-bang delivery” in software projects… a common source of failure. And after the building is complete, the construction firm goes away — there are maintenance staff, but that is a different organization entirely. Programming effort don’t really “end” in the same way, but mislead by the construction metaphor, many software development projects do “end”.

Programming is like gardening.

In The Pragmatic Programmer, Hunt and Thomas propose a new metaphor: Programming is like gardening. I think that this metaphor offers significant benefits… enough that we should begin to make a conscious effort to use the gardening metaphor in our normal conversation. We should speak of “growing” the code, of “pruning” the technical debt, of “transplanting” libraries and of “weeding” out bugs.

The first big “truth” to the gardening metaphor is that gardens aren’t suddenly opened to the public on a given date; they are slowly grown over time. Sure, you might host a garden party (marketing push) on a particular date, but that’s not the first time a non-gardener views it - there are several individual visitors to the garden before the party.

Also, after you get your garden planted it requires ongoing care; the plants may grow on their own, but they need constant tending. This is true of programming too. One would think that, since software is built from abstract thoughtstuff closer to Plato’s perfect forms than to base materials, it would be LESS subject to the ravages of time than mere bridges. Curiously, the truth is quiet the opposite: the Golden Gate Bridge was built 70 years ago and it stands solidly today; end of life for Windows 2000 came in 2005. Software suffers from “bit rot” - the tendency to stop working because of changes in hardware, libraries, operating systems, standards, or other pieces of the environment. A certain amount of ongoing care is needed to keep the software healthy.

There are many smaller ways in which the gardening metaphor bears fruit. Growing a plant in place “incrementally” produces a healthier plant than transplanting, but it takes longer. Different plants grow better depending on the soil quality. Sometimes you have to take out an unhealthy plant so its neighbors can flourish. There is a wealth of useful insight to be gained from this metaphor, and we should give it a serious try.