1. Severity – mandatory: no logging system should be without this
2. Tree of Log Topics – mandatory: no logging system should be without this
3. Configurable – vital: configuring log levels at runtime is something we use often
4. Rotating Log Files – vital: our log files would be too big for the OS without this
5. Configurable Log Line Format – vital: it is unlikely that the off-the-shelf fields would be the ones we want to use
6. Logging of Exceptions – vital: getting stack traces from logs is one of our most productive debugging techniques
7. Delayed String Construction – vital: I consider this to be an very undervalued feature. Without it, software will be slowed significantly and also will be less readable.
8. Log to Multiple Locations – desirable: sometimes this is handy. We used to use it, but at the moment we don’t.
9. Configure where Logs are Directed – desirable: this, too, we have used in the past but are not using right now.
10. Standard or Widely Used – desirable: in the Java world, only Log4J (the most widely used library) and java.util logging (which is in the standard library).
11. Unique Messages – desirable: as suggested in the comments, an ability to identify each log usage uniquely (probably by source file and line number) would be handy.
12. Sensible Fallbacks – desirable: it’s nice that the library works OK when your config fails, because it helps in debugging the config problem.
13. Logging Queued to Avoid Delays – nice extra: although this seems like it would be a very useful feature, I do not know of any serious production logging tools that implement it.
14. Threadlocal Context Data – nice extra: theoretically, this is extremely useful. In practice, people usually live without it and find the data by reading back through the log.
15. Log Filtering – nice extra: if this were easy, we would use it to filter out SSNs and passwords from our logs, but we can live without it.
16. Internationalization – undesirable: I recommend against ever using this. Use logging ONLY for developers, NOT for end users; all developers should speak the same language.

Next, I will assemble a list of different logging libraries in Java to be evaluated.

• Log4J: Log4J by Apache is the most widely used logging framework in Java.
• Java util logging: Rather than adopting Log4J as the standard for logging in Java, Sun chose to clone it, creating something almost-but-not-quite the same as a standard Java library.
• Commons Logging/Log4J: Commons Logging is a wrapper from Apache which is designed for use in libraries. It simply delegates to an underlying logging framework. The purpose is so a library can be configured to use the same logging system as the rest of the application. I will consider Commons Logging backed by Log4J.
• SLF4J/Log4J: SLF4J is a project begun by the original author of Log4J. It is intended to provide a better API for calling into a logging framework, and it can connect to different logging back ends. I will consider SLF4J backed by Log4J.

There are others (logback, jLo, and many others), but I am fairly confident that one of these 4 will be the final choice, so at this point I am going to perform a full analysis just on these 4.

So, after considering all of these options, I have concluded that my preference is to use SLF4J as an interface, with the implementation from Log4J. Most of the options I have considered have more or less the same features. The deciding factors are (1) Threadlocal storage (MDC) is useful and not present in java.util.logging, and (2) The API for SLF4J provides an elegant solution to delay string construction, which is rather important for performance.

Conveniently, SLF4J publishes a tool for automatically converting existing java.util.logging and Log4J code to SLF4J, so the conversion should be relatively painless.

The ultimate goal in logging is to write out some messages that are solely for use in debugging and managing a system — output that is a side effect rather than the primary goal of the program. The simplest version of logging has always been this:

SplineList splineList = reticulateSplines(rawSplines);
System.out.println(splineList);

The simple “print” statement is a universal form of logging (I can name only a couple of languages that don’t have some basic syntax to support it) and I have used it far more times than I would choose to admit. Any logging framework should be evaluated in terms of what it adds to this capability. Here are the features I have been able to think of:

• Severity – the ability to mark some log messages as being more important than others. There is no need to go overboard: even the biggest systems I have seen have done quite well with 6 or seven different levels, but it is nice to distinguish between printing out the value of a variable for debugging and reporting a potentially fatal application error. Severity in logging looks like this:

  logger.log(Level.WARNING, "Spline reticulation returned 0 results.");

• Tree of Log Topics – in large or medium-sized programs, no one is interested in every single log message (at least not all at the same time). The best solution that has been found is to group messages into “topics”, and the best solutions organize these topics into trees. So you may write some logs to the topic “security.login”, others to “validation.TransferMoney” and a library may write to “org.apache.comcat.util”. Logging levels can be set to view even DEBUG level messages within “security.*” but only FATAL messages in “org.apache.*”. Often a sensible default is to take the name of the log topic from the full package-and-class of the class that is writing the log message. Use of log topics looks like this:

  private static Logger logger = new Logger("com.mcherm.sampleapp.ThisClass");

• Configurable – the choice of what to log, at what severity, and to what files (or other kinds of output) should not be embedded in code. It should be configurable, certainly at deploy time and perhaps at runtime as well. Configuration looks something like this:

  log4j.appender.LOGFILE=org.apache.log4j.DailyRollingFileAppender
  log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout
  log4j.appender.LOGFILE.layout.ConversionPattern=%d %p [%t] [%X{sesstag}] [%X{pmsessid}] [%c] - <%m>%n
  log4j.appender.LOGFILE.DatePattern='.'yyyy-MM-dd

• Rotating log files – When logs are written to a file (by far the most common approach), the files can very easily get too big to use. A useful feature is the ability to set a max size for the files. After the max size has been reached, subsequent writes go to a new file, and the old file is left in place until later. Rotating log files lead to directory listings like this:

  12/10/2009  03:24 PM         1,710,360 loanapp.0.log
  12/09/2009  11:41 AM                 0 loanapp.0.log.lck
  12/09/2009  11:40 AM           189,992 loanapp.1.log
  12/08/2009  06:06 PM         1,366,039 loanapp.2.log
  12/07/2009  03:08 PM         8,314,456 loanapp.3.log
  11/24/2009  01:34 PM           410,931 loanapp.4.log
  11/24/2009  11:34 AM           506,066 loanapp.5.log

  

  The other kind of "Log Rolling"

• Configurable Log Line Format – The key content of a log message is typically some string that was provided when the call was made to the logger. But there is a lot of other information that may be available: the exact time the log call was made, the logger topic that was used, the place in the code the call was made from, the thread that was in use… and lots of others if you keep thinking about it. A useful feature of some loggers is the ability to configure just what information gets written out along with that basic message. Typical output from a logger may look like this:

  * 2009-12-29T15:56:55.591-05:00 [22] com.company.util.security FINER (com.company.util.security.OneTimeCredential.validateCredential)
  clearCred = 00 00 01 25 DC 3A 23 8E 00 00 00 4E
   
  * 2009-12-29T15:56:55.591-05:00 [22] com.company.dg.util.OperatorImpl FINEST (com.company.dg.util.OperatorImpl.getOperator)
  Returning systemOperator = web
   
  * 2009-12-29T15:56:55.591-05:00 [22] com.company.util.ApplicationPropertiesImpl FINEST (com.company.util.ApplicationPropertiesImpl.getProperty)
  Property Name(AuthenHelper.custDatabase) has value (cust).
   
  * 2009-12-29T15:56:55.591-05:00 [22] com.company.util.ApplicationPropertiesImpl FINEST (com.company.util.ApplicationPropertiesImpl.getProperty)
  Property Name(AuthenHelper.getPartyId) has value (pkg_dg_party_login.sp_getPartyIdByString).

  Each entry has a timestamp, the thread number in square brackets, the logger name, the severity, the function logging it, and then the log message string. There are various other pieces of metadata that could have been recorded instead of or in addition to the timestamp, thread number and so forth, and this can all be controlled from the logging config file.

• Logging of Exceptions – Most logging has been described in terms of the string to be logged, but there is no reason why it necessarily has to be a string. In theory, a logging framework could support producing sensible logs from other kinds of objects that are passed in to it. In practice, I have only seen one such object which is useful (but that one is quite helpful) – Exceptions. So it is useful if a logging framework allows the user to specify an exception, along with the string, when that is appropriate, and includes the stack trace from that exception in the data recorded for the log.
• Delayed String Construction – The content for a log message can be just a simple string:

  logger.warn("Spline reticulation returned 0 results.");

  but it is far more common for it to be a string constructed with data available at runtime:

  logger.warn("Spline reticulation returned " + numResults + " results.");

  Also, it is not at all unusual for there to be significant amounts of work in constructing the string:

  logger.warn("Splines returned: " + splineList.toString());

  Because Java does early binding of parameters (as do most languages other than Haskell), the work to generate the string to be logged will be done EACH AND EVERY TIME the log statement is encountered, EVEN IF LOGGING IS DISABLED. So even if the log levels are set to warnings and debug logs won’t be printed, the examples above would still perform all the work to construct the string. This overhead can be enormous: I personally worked on a project where we sped up the entire application by a factor of 3 just be avoiding the creation of log lines for statements that would not be logged.

  There are three ways to avoid this overhead. One is specify the log level at compile time (lots of logging systems for C do this), but then it cannot be changed without recompiling. Another is to manually perform the (efficient) check of logging level before performing the work:

  if (logger.isLoggable(Level.DEBUG) {
      logger.debug("Splines returned: " + splineList.toString());
  }

  but this requires burying each logging statement inside of an if. The final option is what I call “Delayed String Construction”, or sometimes “parameterized log messages” and that is to have library level support for delaying the construction:

  logger.debug("Splines returned: {}", splineList);

• Log to Multiple Locations – Sometimes you want certain log entries to go to more than one file. For instance, you might want to send all logs to a single master log file, but also send logs about login errors to a separate security log which gets reviewed by a different group of people.
• Configure Where Logs are Directed – Usually you want logs to go to a file, but other times you want something more exotic like storing logs from multiple servers on a single remote location, sending logs to a messaging system, or writing logs to a database. This is particularly useful in conjunction with logging to multiple locations. So one useful feature, called “Appenders” in some systems, is the ability to provide a custom piece of code which processes the log entry: it could write it to a file OR do something else more exotic.
• Standard, or Widely Used – It is a big advantage if the logging system you use is widely used by others. One reason is the usual: standardized or widely used products tend to get improved over time while more obscure products tend to be forgotten and lose support. But in logging there is an additional advantage: if a library you are using or the server that is hosting your code also uses the same logging facility, then it may be possible (even easy) to comingle the log messages with your application log messages, which can be extremely useful when debugging issues which cross library boundaries.
• Sensible Fallbacks – As previous features have mentioned, logging can involve all kinds of complicated configuration and setup. In the real world, these will sometimes fail. Perhaps your logging configuration file cannot be found, or the log directory cannot be written to, or the logging database password fails — and the most common of these errors is to have no configuration at all. Different logging frameworks have different fallback behavior in the face of problems like this: some will disable all logging, others will fall back on a more primitive form of logging such as writing all logs to stdout if the logging is not configured. It is desirable for a logging system to have sensible fallback behavior.
• Logging Queued to Avoid Delays – In nearly all cases, logging is not the most important thing that your application is doing. The primary goal of the program is to reticulate splines, respond to user keystrokes, respond to web requests, or whatever it is that your application does; logging is a secondary concern. That being said, it is a real shame that writing logs slows down the application. It is quite common that the work done in response to a request can be quick and simple enough (retrieving some data that is already cached) that it is FAR less than the time required to open a file on disk and write even a single line of logging.One solution (with a few drawbacks) is that instead of actually writing the log immediately, logs that are to be written get put onto a queue, and they are written out by a background thread. The big advantage is performance: your program can quickly stash a few strings in memory and get on with its work faster; the log entry gets written later, when there are spare CPU cycles available. It can even save time overall by batching writes. There are a few disadvantages to be aware of: it will increase memory requirements (but typically by an insignificant amount); the queue typically needs a max size, and you need a policy for what to do when logs are being created faster than they can be written: typically the choices are either to discard some of the logs or to slow down the application. And it makes it difficult to debug issues that actually lead to crashes (because the logs don’t get written due to the crash). But significantly faster application performance is often worth these costs.
• Threadlocal Context Data – Imagine building a web application, which takes requests from logged-in users. And consider logs which are written from deep inside the spline reticulation code. For tracking down certain kinds of bugs, it would nice to know which user’s data contained the spline that was being reticulated. But we can’t just add that to the log message, because the spline reticulation code isn’t passed the name of the user. Some function 7 levels up the stack may have known it, but that doesn’t help to pass it into the log message.So a useful feature would be to have a command we could call at the point where we first identify the user submitting this request. This command could tell the logging system the name of the user and the logging system could store it in threadlocal storage. Then all future logs written from this thread could be annotated with “user=JSmith”, until this service call is completed.

  Two versions of this feature are offered by Log4J: “Nested Diagnostic Context”, in which multiple pieces of data can be kept in a stack, and “Mapped Diagnostic Context” in which multiple pieces of data are kept in a map.

• Log Filtering – Not all applications will have this requirement, but at large institutions with a concern for security, there are some things which ought not be written into a log. For instance, there may be a policy that passwords are not recorded, or that customer social security numbers never be stored in an unencrypted fashion. A fairly rare, and often imperfect but still useful, feature is the ability to provide some sort of filter which will recognize these kinds of data and remove them before the log data is recorded to disk. The difficult part, of course, is in recognizing the data.
• Internationalization – Many loggers have support for writing messages in unicode (particularly in languages like Java where the fundamental String type supports unicode). But some go further and provide support for loading the messages from a language-specific file and may have features like localization of dates and other information appearing in log messages.

So that’s my list of features; see my next entry where I use this list of features to evaluate some Java logging libraries.

My bank on the online banking login instead of having a password field it presents you with 3 password fields 1 character each where it asks you for 3 characters from your password, chosen randomly. E.g. the 2nd, 4th and 7th.

I wanted to respond to this, because not only is it an incredibly misguided attempt at security which seriously weakens actual security, it also sounds familiar. Because a few months ago my employer considered doing something just like this. Let me recount the story:

I work for a bank, so we care a LOT about security. Customers call into our call center and to identify themselves they get connected to the IVR (interactive voice response unit… telephone system) to enter their PIN (a 4 to 8 digit passcode). An important feature is that we cut out the phone reps from hearing this… because we want your password to be a secret EVEN FROM OURSELVES. All of this is good security design.

We opened up a new call center in Hawaii, and they had some problems. Apparently the phone system we were using had a time limit when transferring a call — if it wasn’t picked up by the remote phone switch within a few milliseconds then it was disconnected. The ping time between the Hawaii call center and our east-coast data center was just a little too long and many of the calls were being disconnected when they were transferred to the IVR to enter the PIN.

The first solution that they thought of was to stop using the IVR to enter PIN numbers. Instead, the idea was that they would instead create a system where the phone reps asked the customers for certain digits out of their PIN (just as described in enanoretozon’s reddit question). They would type this in and then the customer could log in. Apparently, this was the standard practice at our German subsidiary, and had somehow become blessed as the official corporate-wide best practice.

Well, it may be an official “best practice”, but it’s still a very bad idea, for two reasons. The first reason should be completely obvious if you just try it. First, say your phone number out loud. Now say the 3rd, 6th, and 4th characters of it. For most normal people, the second will take many times longer, and be much harder, even though it is only 3 digits. There is always a tradeoff between security and usability (We could provide perfect security if we never allowed anyone to take their money out of the bank. Of course, usability would have dropped to zero.), and entering random digits has SUCH poor usability that it is not worth it.

Random Digits

Besides that, it is also less secure. There are, if you consider it, multiple different kinds of attacks that we need to protect against. One kind, certainly, is attacks by unscrupulous bank employees who might misuse a customer’s login credentials. But another far more likely attack is a third-party who wants to steal from a customer’s account.

Such an attacker, if they didn’t know the customer’s PIN, would have to guess. To prevent repeated guessing, we will temporarily lock out a customer’s account after a certain number of incorrect login attempts. But a clever attacker would just try different customers, making just one guess for each one.

With (for instance) a 6-digit pin, the expected number of guesses before the attacker got one right is around 100,000. Long before an attacker managed to try even a small fraction of 100,000 guesses, we would have noticed what they were doing and put a stop to it. But we only ask for 3 particular digits out of the password, then the attacker only needs to try about 1000 times before she is expected to guess correctly. There is a good chance that we would catch that, but (particularly if they spoof their phone number) we might not.

So we traded better defense against a rare attack (we don’t hire a lot of employees who commit bank fraud) for much worse defense against a common attack (we detect and stop attempted attacks of various sorts every single week!). It is NOT an improvement.

So… after these points were raised, we chose not to implement our German counterpart’s policy. What did we do instead?

Very simple: we fixed the phone system so it could transfer calls properly.

The feature that we need to implement (or “story” in Scrum parlance) is an increase in the maximum number of customers that can be set at once. You see, there is a “feature” that limits the number of IDs that can be set at one time to about 200. (“About” 200 because most id’s are 9 digits long and they are separated by whitespace; the actual limit is 2000 characters, enforced in Javascript as the field is input.) So when they need to set an alert on 600 IDs, they run through the screen 3 times. When they have 2.5 million IDs to update they open up a “story” for the development team.

I think we asked someone why it was limited to 200 IDs. No one is quite sure, but it’s probably to avoid overtaxing the database query or running a middleware service that takes too long… something like that. “Sure,” we say, “we can increase the limit.” We figure maybe we’ll group it in chunks of 200 and call it in a loop or something. We schedule it to be worked on in this month’s “sprint”.

A couple of man-days of effort go into building it. Some testing determines that (on much less powerful dev hardware) a single call can easily handle thousands of IDs without running into timeout issues — more than that, actually, as we left a factor of 4 or 5 for safety. So the front end breaks the list into chunks of that size. We thought we’d build it to handle unlimited capacity, but there’s an IE6 bug (yes, our corporate overlords require the use of and obsolete broken browser) that limits us to about 60,000 IDs.

Our Corporate Overlords

So we have completed the feature and the business can now enter more than 50x as many IDs at a time. But that’s not quite the end of the story. Because as part of regression testing, our QA staff does some exhaustive testing of the screen, and they discover that there apparently isn’t a limit on the size of other field, the one that contains the alert message. We check the database table for the appropriate max message length, and it turns out to be exactly 2000 characters.

Wait… I think I’ve heard that number before.

Apparently, whoever built this page in the very first place accidentally limited the length of the wrong field. There never was a reason for a limit on the number of IDs processed at once… the limit came entirely because of a bug. Yet we’ve been living with this absurd limitation for several years, simply because no one ever questioned the limit. (Or if they did question it, they got some vague answer like “I assume it’s for performance reasons.”)

I’m sure there is some lesson we should draw from this experience… I’ll leave it to you to figure out what the lesson is.

Well, they released a new version and I hit the “upgrade” button. After that, my project didn’t work anymore. I tried for a day to resolve it and I just couldn’t understand anything. Finally I “solved” it by uninstalling Eclipse and reinstalling it, then following the tutorial steps to create a brand new project and copying in my old files one-by-one. Another full day lost (I can only work a couple of hours per day on hobby projects).

Surely they wouldn’t do it again, right? So I carefully saved everything and held my breath the next time Google released an upgrade. It promptly broke everything like last time. Only this time I solved it differently: I uninstalled Eclipse and did NOT reinstall it.

The big difference is that in the intervening month JetBrains had announced that a slightly-impaired version of IntelliJ IDEA would be available for free. The stripped down version doesn’t have support for GWT and App Engine (which the paid version does have), but it’s something I can use. At work, I use IntelliJ (properly paid for) but it’s awfully expensive to pay for my own copy at home. (Can’t use the same copy because that would disturb the corporate bean-counters, even though it is allowed by the license.) The stripped down version is fine if I can run from the command line.

There are instructions for running GWT via ant. And there are instructions for adding support for App Engine. But they are broken in (what I think is) exactly the same way that the Eclipse plugin is broken. Details from a forum posting led me to realize the problem was that it now needs a “javaagent” specified. A “javaagent” is some sort of a pre-processor that runs before main() — apparently introduced with Java 1.5.

So after following Google’s instructions, I now add the following: In my <hosted> target, along with the other <jvmarg> elements, I add a new one which looks like this:

<jvmarg value="-javaagent:${appengine.sdk}/lib/agent/appengine-agent.jar"/>

After that, I can build it using ant. I’ll also need to use the command line for deploys, that looks like this:

"C:\Program Files\appengine-java-sdk-1.2.6\bin\appcfg" update war

And now it works again.

But there is one catch: the tools are “dumb”, and there always needs to be a way for a knowledgeable human to override it. Usually it’s a special comment that tells the tool not to report a particular infraction. The override is used in that rare special case where there is a good reason why the rule does not apply.

Now, I know a certain portion of my readers are already forming objections, certain that THEIR pet peeve is the one case to which there should never be an exception. For instance, rules requiring consistent indentation throughout the project are good, but suppose some of your source files are from an external source? It’s not wise to reformat them just to placate the code scanner — that will make it harder to merge in changes when the next revision is released.

Or for another example, the capitalization rule that Java instance variables always begin with a lowercase letter seems very good, but when my team created an XML binding mechanism that mapped XML fields (which are case sensitive and often begin with a capital letter) to instance variables, being consistent with the XML file was more important than following the standard Java conventions.

I can still hear someone arguing. Some reader out there is still complaining that there is a special reason (security? correctness checks?) why some rules must be absolute. Frankly, I think this reader is just a control freak and they should learn to let programmers act like the professionals they are, but in order to convince you, I’m going to tell a story.

Larry Osterman

This is the story of a programming rule so VERY absolute that before you read the tale you’ll agree that there is NO possible reason to violate this rule. Yet after an automated code scanner discovered the violation, the catastrophic result was that for 2 years, all cryptography originating from a major operating system was completely insecure.

So… on with the story. (And many thanks to Larry Osterman, pictured here, from whom I first learned this sordid tale.)

Now, Linus Torvalds and his compatriots manage the source code for the kernel of the Linux operating system, but a usable system requires much more — a whole set of systems and applications which are tested and configured to work together. This is called a “Linux distribution” and there are several major distributions in wide use: SUSE, Fedora, and Mandriva among others.  The single most popular today is Ubuntu, which is based on Debian.

The folks at Debian collect code from a number of different projects and carefully review it, test for compatibility, and then certify the resulting distribution and provide a place to download the finished product. One of those components is OpenSSL, an open-source implementation of some cryptography libraries that provide support for basic functions like secure random number generation. And one of the tests that the Debian folks perform is to analyze the code with Valgrind, an automated code scanner that detects memory leaks and similar problems.

Valgrind detected that there was a problem in one of the OpenSSL routines that generated random numbers — it accessed uninitialized memory. Not knowing how to suppress the warning, someone at Debian decided to “fix” the code.

What they didn’t realize is that the OpenSSL developers had known exactly what they were doing. The crux of a cryptographically secure random number generator is an entropy collector. You see, the problem with generating “random” numbers on a computer is that nothing on the computer is truly random. You can mix and mash bits with the fanciest hash function in the world, but if the seed you start it off with is just the current time off the clock and an attacker can guess that (all but the last few digits are awfully easy to guess), then the attacker can repeat the same process and determine what “random” key was chosen, thus completely cracking your security.

So a cryptographic random number generator (as opposed to a garden variety RNG) goes to great lengths to collect “entropy”. It may start with the time off the clock, but it mixes in the number of milliseconds between key presses on the keyboard. And it mixes in the process ID of the OpenSSL process, data traveling over the network socket, micro-timing of the hard-drive motor and of mouse movements, and anything else they can use as a source of “randomness”. When they are first creating the data structure for the entropy pool, they intentionally left the memory uninitialized, because it’s yet another source of randomness.

Unfortunately, a Debian maintainer didn’t realize this. And they also made a minor error when changing the code: they accidentally commented it out in such a way that all sources of entropy other than the first one (the process ID) were multiplied by zero before being mixed in. So there was NO randomness except the process ID (which is very easily guessed).

Ubuntu

Then Debian was released with this bug, and Ubuntu picked it up and distributed it further. And 2 years went by. During that 2 years, everything done using the RNG on Debian or Ubuntu Linux is insecure because the keys are guessable. Everything! Any SSL connection made from such a machine. Any secure certificate signed by such a machine. And no one noticed for two whole years.

So the moral of the story is, don’t behave like that  ignominious Debian developer and change code that you don’t understand. But also realize that for any supposedly-universal rule, there is some special case exception. In almost all circumstances, the rule is good, but it is still wise to provide some way for the experts who do need to violate the rule to declare that they are doing so on purpose (preferably with a required explanation of why). Otherwise, someone who doesn’t know what they are doing is likely to break things.

Then with a single innovation, the entire science of command language design became forever irrelevant. Command languages disappeared completely except in a few odd cases; even those depended more on its use as a scripting or programming language rather than a way for a normal user to manipulate the program.

Lotus 123 with Menus

That innovation was the invention of the menu, which I first remember encountering in Lotus 123 during the early 1980s. No longer did the user have to keep the manual handy or memorize a large vocabulary of commands, trying to remember whether this particular piece of software used ‘open’ or ‘load’; used ‘close’ or ‘quit’, instead she simply selected the command from a list of available options displayed on the screen. No longer was there a complex structure of command arguments and command modifiers, instead many commands launched a dialog box where the user could specify options or select a target.

The old approach had a few advantages — for instance, it was more amenable to scripting — but the ease of use of menus won out and it was only a few years before every program had menus. Those who had studied the somewhat arcane skill of designing good command languages soon found themselves working side-by-side with the buggy-whip repairmen.

I see a similar transformation occurring right now. When my wife was studying “Information Science” (librarians, for instance, may take this), she spent a whole unit studying “Ontology“. Ontology, as I understand it, is the creation of a taxonomy and/or specific vocabulary for describing a field of knowledge. For instance, “Cooking” is a field of knowledge, and if you were compiling an index for a cookbook or a cooking site, you might want to decide whether to group recipes into “soups, salads, entrees, deserts” or “breakfast, lunch, dinner”, or any of hundreds of other ways they could be organized.

But the innovation that will obsolete the field of ontology has already appeared. The first of these was good, quick search engines. The fundamental trick for rapid searching is the hash table, but when that is scaled up to the size of Google’s index, it takes on a whole different meaning. I often don’t bother to look in a manual or help pages because (astonishingly) it is actually easier and quicker to type few words into Google and search the entire internet than it is to activate a program’s help features.

It used to be that an ontologist could argue that searching still required one to know what key words to search by. But a couple of days ago I heard a friend describe how she learned to move archived emails from one program to another. Searching for “eudora email export” didn’t turn up any useful results, but when she simply typed “eudora email”, then waited to see what the “Google suggest” feature displayed. Right there was “eudora email transfer”, and now that she knew what keyword to use, finding the information was easy.

The massive scale of the data available on the Internet or to a company like Google makes it unnecessary to have an expert produce a carefully designed vocabulary for a topic. Instead, we can effectively mine the collective mind of all of humanity (or at least that tiny portion of it that gets typed into searchable form on a network somewhere). Those words that are frequently used to define the topic become the correct vocabulary for describing it. Like menus, this requires a two-way flow of information between the user and the system, but my friend’s example shows that this two-way flow already exists. And a professionally designed ontology is likely to be more effective in many ways (just as a command language could be more powerful and scriptable than menus), but I think that is unlikely to stop people from abandoning well-organized bodies of knowledge in favor of those that handy to search.

In Oracle (what I use most often), the correct type to use for an email is VARCHAR2(x) for some value of x. Oracle stores VARCHAR2 in an space-efficient manner — essentially it stores a null-terminated string. So if you declare the field to be a VARCHAR2(600), then it won’t take 600 bytes to store “JoeSmith@example.com”. It will take 21 or 42 bytes (I forget whether it stores characters in 8 bits or 16), one (or 2?) for each character plus one for the null terminator.

Given that, we don’t mind creating a really large field if that’s appropriate: it won’t take up extra space and will prevent someone’s long email address from being cut off. So my first instinct is to figure out the maximum legal length of an email address, then use this length.

Unfortunately, that turns out to be trickier than I had thought. Email was one of the first things standardized on the internet (actually, before there was an internet!) so the specifications have gone through several revisions. I believe the current specification would be RFC 5322. That does not define any length limit on email addresses, but it DOES define a maximum line length.

To get from that to email addresses, we need to divert to discuss what an email address is. The official definition says that an email address looks something like this: “Michael Chermside <mcherm@mcherm.com>”. However, in common parlance, people use the term “email address” to refer to just the “mcherm@mcherm.com” portion. Since we are storing email addresses entered by customers we’ll assume we are only storing the latter portion.

With that restriction, email addresses cannot contain whitespace. And RFC 5322 says header fields can only be line-wrapped at whitespace, so you can’t line wrap within an “email address” (of the sort we’ll be storing). So start with a max line length of 1000 characters, subtract 2 characters for the “\r\n” and 3 more for “TO:” and we can say that an email address over 995 characters could never receive email.

It was fun finding a theoretical limit, but to size my actual database column, I will use a much more practical approach. I really only know of one service on the internet that specializes in creating extremely long email addresses. (Yes, you can find anything on the internet.) That is the euphoniously named “http://www.abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk.com/“. Basically, somebody realized that although there is no length limit on domain names, there IS a limit of 64 characters in the component before the “.com” . So he registered a maximum-lenght name and started offering webmail with very long email addresses. The longest username they allow is 32 characters, with the @ and .com that comes to a nice round 100 characters.

So I will declare my email column to be VARCHAR2(100).

I work for a bank, and so I worry more about security than most programmers. After all, if a hacker were were truly motivated and competent who would they pick to go after? Probably a bank (the other good option is political or corporate espionage). Recently I saw two security-related stories which, when combined, form my ultimate nightmare: an effective attack for which I cannot imagine a possible defense.

The first story was a research article on microchip design written by some researchers at the University of Illinois at Urbana-Champaign. The researchers asked an interesting question: suppose a smart, competent attacker were to make some changes to the design of a CPU: could they make the CPU vulnerable to some kind of hacking attack without making it obvious? It is not a question I had ever heard before: I am used to thinking of the hardware as fixed and looking for vulnerabilities in the software instead.

The answer was a resounding “Yes!”. They came up with two attacks: each requiring around 1000 extra logic gates, on a chip that has around 1.8 million gates! One attack allowed a specially designed program to read anything in memory (including things like passwords), the other allowed extra programs to run that were invisible to the rest of the computer. Each could be activated by using a special “trigger phrase” — which could even be delivered by just sending a packet to the machine (you don’t necessarily have to hack into the machine to activate it). Once activated, the code you run can be completely invisible to the operating system, and can override any software running on the computer. It could collect passwords (allowing a conventional hack), modify data, replace programs with versions that have backdoors, and if done by a competent hacker, all of this would be impossible to detect.

An undetectable and unstoppable vulnerability: it’s a good thing that CPUs cost millions of dollars to manufacture (even for a very small run) and that we buy our computers from reputable companies like Dell who get their chips from reputable companies like Intel and AMG.

The second story is about a large multinational fraud that was being executed throughout parts of Europe. Over a period of months, MasterCard had noticed a suspicious pattern of credit card fraud throughout Northern England. Someone technically sophisticated was making fake duplicates of real credit cards, but the investigators could not determine how the fraudsters were obtaining the data. By one account, the break in the case came when a security guard at a grocery store noticed a burst of static on his cell phone; by another account the break came when someone duplicated a card that had only ever been used once. Either way, someone took apart one of the store scanners and discovered that it had an extra payload.

Someone had built in a device which stored data on a small portion of the cards that got scanned. Then once a day it would place a wireless call to someplace in Pakistan to drop off the data and pick up new instructions. Presumably from there an international criminal ring produced the fake cards.

Of course the fraud groups at MasterCard are well aware of this kind of scam, and they correlate stolen card numbers back to what stores and what devices those cards had been used in previously. Had it been just this one scanner with modified hardware, the folks at MasterCard would have identified it almost immediately. Scotland Yard noticed that the scanner with the bug in it weighed an extra 4 ounces. Soon investigators with scales were speeding to stores throughout Europe, checking for card-readers that weighed 4 oz more than they should. They found hundreds of compromised machines – even in major retailers like Wal-Mart, and there was no sign of tampering.

It turns out that the bugs were inserted not by agents in the store, but by someone in a factory in China back when the devices were first being manufactured. The criminals were quite smart in how they leveraged the information: never stealing so much from one site that it would give away the game. And they made off with at least 50-100 thousand dollars, perhaps more. No matter how careful the people at the stores were, they had no chance against devices that where compromised in the supply chain before the store ever received it.

. . .

So here is my nightmare senario: someone decides to target a bank (’cause that’s where the money is). They manufacture chips that allow them access below the operating system level, which is immune to any attempt to use virus checking, encryption or other such defenses. And they insert these into the supply chain somehow (how big a bribe would it take to get a couple of hours unobserved in a Dell warehouse?). Then they get competent programmers to slowly siphon off cash at levels that won’t be noticed. Sure, the costs of such a scheme probably run in the millions of dollars, but organized crime in places like Russia certainly have access to this kind of money.

And I cannot imagine anything that I, as a technical expert at a bank with a concern for security, could do to prevent this attack.

As far as I can tell, there IS no universal solution to this problem — only particular solutions that optimize for certain kinds of application behavior. In our case (we’re building an online mortgage application) nearly all of the interaction will occur with a logged-in user, or at the very least will care whether the user is logged in, which almost certainly means reading user information from the Session. So almost every request will be reading from the Session. However, many of the requests will be used for display not update — they will be reading from the Session but not updating it.

That observation immediately suggests the use of immutable data structures. Immutable data structures are a technique for concurrent programming that has been known for many years, but it has received a renewed spurt of interest recently sparked largely by the 1996 thesis of Chris Okasaki and others who expanded on it. What everyone had always known was that if a data structure is immutable (can’t be changed) then multiple threads can safely read it – essentially each thread gets a “frozen” copy of the data as it looked at the moment the thread began working with it. What Chris’s thesis added was a coherent way of analyzing the performance behavior of immutable data structures and some clever ideas that made them nearly as fast as traditional mutable data structures.

So if we use an immutable data structure, then we can be sure that all requests that are just reading data and not updating it can proceed at full speed. They will receive a copy of the data as it looked when the thread began processing the request. Any changes to session that happen during the time that the request is being processed will not be seen, but that is probably the best behavior we could ask for — it is equivalent to assuming that the read-only request was executed so quickly that it finished before the update occurred.

So we’ll use some sort of an immutable data structure to hold our session data — perhaps it’s a linked list, or an immutable map, so far it doesn’t really matter. And we’ll handle requests that just read data by grabbing a copy of the structure once — that way no matter how logn the request takes, it will be seeing a single, coherent view of the session data. For requests that must both read and write data we can do the reading the same way: take a copy of the data structure at the beginning of the processing and do all reads from that copy. The difference is that at the end of the service we will have done some writes to the data — with an immutable data structure this means we’ll have generated a new structure which is just like the original except for the small number of changes we have made. The question remaining to be solved is what to do with this changed data.

The “obvious” solution is to just update the pointer in the HTTPSession to point to our newly changed data, but this won’t work. This won’t be safe in the face of threading issues. Imagine two requests that both do writes are occuring simultaneously: one modifies the customer’s current requested loan amount while the other modifies the customer’s address. Whichevre is processed second may overwrite the changes of the first — so one thread would update loan amount then the second (unaware of the first) will set back the loan amount while updating the address. In the end, the user thinks the loan amount has been changed, but it hasn’t.

I can think of two solutions here: “one-writer-at-a-time” and “merging”. The general idea behind “one-at-a-time” would be for any thread that intends to make modifications to obtain a lock before beginning. Then we would allow as many “read-only” threads as desired, but only one “read-write” thread per user session could execute simultaneously. If the application consists of lots of simultaneous AJAX calls to obtain data but very few that perform updates, then this is a simple and straightforward approach.

To make this idea more clear, let me define an API that would achieve “one-writer-at-a-time” semantics. In the Session, we would place a single ThreadsafeSessionData object, and all programmers would be enjoined not to use the HTTPSession.putValue(). A method ThreadsafeSessionData.getReadonlyData() would return an immutable Map containing immutable data objects with the contents of the session. Servlets that only needed to read data could call this once to obtain a consistent view of the data. Another method, ThreadsafeSessionData.getEditableData() would obtain a lock on the session (blocking if needed) and return an EditableData object. The EditableData object would have a getReadonlyData() method returning the same immutable Map, but it would also have a setValue() method for editing the data (by creating a new immutable Map). And it would have a commitChanges() method which would take the current immutable Map, store it in the ThreadsafeSessionData object, release the lock, and then lock down the EditableData object (no calling setValue() after commitChanges()).

It is possible to go even further than this. Multiple writer threads could operate at the same time, so long as they are not modifying the same piece of data, or relying on a value which is simultaneously being modified. The concept is rather like version control systems that use merging rather than locking. But it requires merging the data after a “commit”, which adds a great deal of complexity to the solution. As a programming exercise, I think it would be fun to develop, but finding a reliable means of merging changes is challenging enough that I think for production use I’ll stop with “one-writer-at-a-time”.