Race #1: Governor:

Dan Onorato

Dan is the front-runner for the governor’s race: polls suggest that he has all-but-clinched the Democratic nomination. One of his strongest advantages is that polls show him doing well against the likely Republican nominee. Dan is a former county executive who appears, based on his position papers and newspaper articles to be a reasonable, competent man who I would be happy to support for governor.

Joe Hoeffel

Joe Hoeffel is an unabashed liberal (sorry: conservatives have made that a bad word, so now they call them “progressives”). His publicly taken positions include support for gay marriage and imposing a progressive income tax. What can I say… I liked most all of his positions.

Anthony Williams

I advice anyone NOT to vote for Mr. Williams. It appears to me that he came from nowhere with significant amounts of funding from what was basically a single rich source. And his major issue appears to be school vouchers: moving money out of public schools when students go to private schools.

Jack Wagner

I learned much less about Mr. Wagner in my research. He presented himself rather well in the debate, but did not (in my mind) distinguish himself.

I will be voting for *Joe Hoeffel*. I agree with his positions on basically every issue for which he had a position paper — including some which were rather controversial.  And since it appears that Dan Onorato has the nomination sewed up I feel I can vote my heart rather than trying to vote strategically.

Race #2: Lieutenant Governor:

Jonathan Saidel

Jonathan Saidel, former Philadelphia comptroller, has political experience, money and democratic endorsements.

Scott Conklin

Scott Conklin is a state House representative from a rural area. He seems generally to be thought well of, and he secured the endorsement of the Philadelphia Inquirer.

Doris Smith-Ribner

Doris Smith-Ribner is a former judge who is running for the office. Her campaign seems less “professional” than the others.

This article, from the Pittsburg City Paper seemed to lay it out best: Smith-Ribner is not a serious contender, Conklin may be up-and-coming, but Saidel is tried-and-true and will be an asset to the ticket. I’m planning to vote for *Jonathan Saidel*, but wouldn’t be disappointed if Conklin won.

Race #3: US Senator: This may be the most important race on the ballot!

Arlen Specter

Look, if you don’t know who Arlen Specter is, then you shouldn’t be voting. He has been a Senator from PA for 30 years! Of course, he was Republican for all but the last year or so. Here’s the deal: I always respected Specter as a member of the “other side” who was willing to be reasonable and to cross party lines. He says that he changed parties to have a chance of getting re-elected (I believe this) and he also says that the Republican party moved to the right rather than he moving to the left (I believe this too). And I strongly want to encourage moderates to leave the Republicans and join the Democratic “big tent”. So I decided long ago to support Arlen Specter.

Joe Sestak

If you are not from my district, you may not know Joe Sestak. He’s a former 3-star Navy Admiral who went into politics in the US House from my district. He has his flaws (he’s more conservative on some issues than I would like, and he seems to have a bossy personal style with his subordinates), but he is very smart, very reasonable, and rather charismatic. He represents the up-and-coming future of the Democratic party.

All other races in my district are uncontested. I’ll happily vote for anyone brave enough to put their name on the ballot.

Late Breaking Correction:

I just got back from the polls, and it turns out there is another contested item on the ballot: selection of 5 men and 5 women for the “democratic committee” (what’s that?). There are just 5 women running, but 8 men. I didn’t do any research on this so you’re on your own. (The democratic party has official endorsements.)

To the FCC:

Some form of “Net Neutrality” is essential, and it is up to the governing agencies to determine what form and how.

The general principle holds that government should regulate as little as possible in order to allow citizens to innovate. However, there are some places where that principle must give way to other important principles. The building and deploying of a worldwide network is a vital public good, but because of very strong network effects (please pardon the pun) the network itself will tend to be run by a very small number of large corporations.

Soon, that network will transport nearly all communications in this country. It will be (already is!) vital to citizens’ ability to communicate: to speak with each other, to associate freely, to petition their government, and so forth. It will also be (already is!) vital to business: many businesses exist solely on the internet (YouTube, online retailers, eBay sellers, iPhone applications) and even more physical industries rely on the internet for communications.

The important point is this: the small number of companies that are responsible for the infrastructure of the internet do not have the ability to completely silence someone: if AT&T or Verizon were to cut off internet access for anyone who took a particular political position the resulting hue and cry would lead to legislation to prevent such behavior. But they WILL have — in fact, already DO have — the ability to have a more subtle effect.

Companies could, with very plausible technical excuses, provide a small degree of preference for one customer over another. Perhaps they would deliver video slightly faster (so it works in realtime) for a major studio who paid them but not for the independent film producer. Perhaps they would block certain protocols that allowed anonymous communication while allowing others (this has already happened). The point is that the fundamental ability to communicate is too important to be allowed in the hands of a small number of companies, no matter how well intentioned.

This, then, is one of those rare situations where minimal government regulation IS called for. Rules are needed which will still permit companies to manage their traffic, but will prohibit them from discriminating on the basis of the origin, destination, format, or content of that traffic. And now is the time for such rules to be put in place.

Michael Chermside

If you feel similarly, I would encourage you to send your own message. But hurry. The last day on which they will accept public comment is April 8th.

1. Severity – mandatory: no logging system should be without this
2. Tree of Log Topics – mandatory: no logging system should be without this
3. Configurable – vital: configuring log levels at runtime is something we use often
4. Rotating Log Files – vital: our log files would be too big for the OS without this
5. Configurable Log Line Format – vital: it is unlikely that the off-the-shelf fields would be the ones we want to use
6. Logging of Exceptions – vital: getting stack traces from logs is one of our most productive debugging techniques
7. Delayed String Construction – vital: I consider this to be an very undervalued feature. Without it, software will be slowed significantly and also will be less readable.
8. Log to Multiple Locations – desirable: sometimes this is handy. We used to use it, but at the moment we don’t.
9. Configure where Logs are Directed – desirable: this, too, we have used in the past but are not using right now.
10. Standard or Widely Used – desirable: in the Java world, only Log4J (the most widely used library) and java.util logging (which is in the standard library).
11. Unique Messages – desirable: as suggested in the comments, an ability to identify each log usage uniquely (probably by source file and line number) would be handy.
12. Sensible Fallbacks – desirable: it’s nice that the library works OK when your config fails, because it helps in debugging the config problem.
13. Logging Queued to Avoid Delays – nice extra: although this seems like it would be a very useful feature, I do not know of any serious production logging tools that implement it.
14. Threadlocal Context Data – nice extra: theoretically, this is extremely useful. In practice, people usually live without it and find the data by reading back through the log.
15. Log Filtering – nice extra: if this were easy, we would use it to filter out SSNs and passwords from our logs, but we can live without it.
16. Internationalization – undesirable: I recommend against ever using this. Use logging ONLY for developers, NOT for end users; all developers should speak the same language.

Next, I will assemble a list of different logging libraries in Java to be evaluated.

• Log4J: Log4J by Apache is the most widely used logging framework in Java.
• Java util logging: Rather than adopting Log4J as the standard for logging in Java, Sun chose to clone it, creating something almost-but-not-quite the same as a standard Java library.
• Commons Logging/Log4J: Commons Logging is a wrapper from Apache which is designed for use in libraries. It simply delegates to an underlying logging framework. The purpose is so a library can be configured to use the same logging system as the rest of the application. I will consider Commons Logging backed by Log4J.
• SLF4J/Log4J: SLF4J is a project begun by the original author of Log4J. It is intended to provide a better API for calling into a logging framework, and it can connect to different logging back ends. I will consider SLF4J backed by Log4J.

There are others (logback, jLo, and many others), but I am fairly confident that one of these 4 will be the final choice, so at this point I am going to perform a full analysis just on these 4.

So, after considering all of these options, I have concluded that my preference is to use SLF4J as an interface, with the implementation from Log4J. Most of the options I have considered have more or less the same features. The deciding factors are (1) Threadlocal storage (MDC) is useful and not present in java.util.logging, and (2) The API for SLF4J provides an elegant solution to delay string construction, which is rather important for performance.

Conveniently, SLF4J publishes a tool for automatically converting existing java.util.logging and Log4J code to SLF4J, so the conversion should be relatively painless.

The ultimate goal in logging is to write out some messages that are solely for use in debugging and managing a system — output that is a side effect rather than the primary goal of the program. The simplest version of logging has always been this:

SplineList splineList = reticulateSplines(rawSplines);
System.out.println(splineList);

The simple “print” statement is a universal form of logging (I can name only a couple of languages that don’t have some basic syntax to support it) and I have used it far more times than I would choose to admit. Any logging framework should be evaluated in terms of what it adds to this capability. Here are the features I have been able to think of:

• Severity – the ability to mark some log messages as being more important than others. There is no need to go overboard: even the biggest systems I have seen have done quite well with 6 or seven different levels, but it is nice to distinguish between printing out the value of a variable for debugging and reporting a potentially fatal application error. Severity in logging looks like this:

  logger.log(Level.WARNING, "Spline reticulation returned 0 results.");

• Tree of Log Topics – in large or medium-sized programs, no one is interested in every single log message (at least not all at the same time). The best solution that has been found is to group messages into “topics”, and the best solutions organize these topics into trees. So you may write some logs to the topic “security.login”, others to “validation.TransferMoney” and a library may write to “org.apache.comcat.util”. Logging levels can be set to view even DEBUG level messages within “security.*” but only FATAL messages in “org.apache.*”. Often a sensible default is to take the name of the log topic from the full package-and-class of the class that is writing the log message. Use of log topics looks like this:

  private static Logger logger = new Logger("com.mcherm.sampleapp.ThisClass");

• Configurable – the choice of what to log, at what severity, and to what files (or other kinds of output) should not be embedded in code. It should be configurable, certainly at deploy time and perhaps at runtime as well. Configuration looks something like this:

  log4j.appender.LOGFILE=org.apache.log4j.DailyRollingFileAppender
  log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout
  log4j.appender.LOGFILE.layout.ConversionPattern=%d %p [%t] [%X{sesstag}] [%X{pmsessid}] [%c] - <%m>%n
  log4j.appender.LOGFILE.DatePattern='.'yyyy-MM-dd

• Rotating log files – When logs are written to a file (by far the most common approach), the files can very easily get too big to use. A useful feature is the ability to set a max size for the files. After the max size has been reached, subsequent writes go to a new file, and the old file is left in place until later. Rotating log files lead to directory listings like this:

  12/10/2009  03:24 PM         1,710,360 loanapp.0.log
  12/09/2009  11:41 AM                 0 loanapp.0.log.lck
  12/09/2009  11:40 AM           189,992 loanapp.1.log
  12/08/2009  06:06 PM         1,366,039 loanapp.2.log
  12/07/2009  03:08 PM         8,314,456 loanapp.3.log
  11/24/2009  01:34 PM           410,931 loanapp.4.log
  11/24/2009  11:34 AM           506,066 loanapp.5.log

  

  The other kind of "Log Rolling"

• Configurable Log Line Format – The key content of a log message is typically some string that was provided when the call was made to the logger. But there is a lot of other information that may be available: the exact time the log call was made, the logger topic that was used, the place in the code the call was made from, the thread that was in use… and lots of others if you keep thinking about it. A useful feature of some loggers is the ability to configure just what information gets written out along with that basic message. Typical output from a logger may look like this:

  * 2009-12-29T15:56:55.591-05:00 [22] com.company.util.security FINER (com.company.util.security.OneTimeCredential.validateCredential)
  clearCred = 00 00 01 25 DC 3A 23 8E 00 00 00 4E
   
  * 2009-12-29T15:56:55.591-05:00 [22] com.company.dg.util.OperatorImpl FINEST (com.company.dg.util.OperatorImpl.getOperator)
  Returning systemOperator = web
   
  * 2009-12-29T15:56:55.591-05:00 [22] com.company.util.ApplicationPropertiesImpl FINEST (com.company.util.ApplicationPropertiesImpl.getProperty)
  Property Name(AuthenHelper.custDatabase) has value (cust).
   
  * 2009-12-29T15:56:55.591-05:00 [22] com.company.util.ApplicationPropertiesImpl FINEST (com.company.util.ApplicationPropertiesImpl.getProperty)
  Property Name(AuthenHelper.getPartyId) has value (pkg_dg_party_login.sp_getPartyIdByString).

  Each entry has a timestamp, the thread number in square brackets, the logger name, the severity, the function logging it, and then the log message string. There are various other pieces of metadata that could have been recorded instead of or in addition to the timestamp, thread number and so forth, and this can all be controlled from the logging config file.

• Logging of Exceptions – Most logging has been described in terms of the string to be logged, but there is no reason why it necessarily has to be a string. In theory, a logging framework could support producing sensible logs from other kinds of objects that are passed in to it. In practice, I have only seen one such object which is useful (but that one is quite helpful) – Exceptions. So it is useful if a logging framework allows the user to specify an exception, along with the string, when that is appropriate, and includes the stack trace from that exception in the data recorded for the log.
• Delayed String Construction – The content for a log message can be just a simple string:

  logger.warn("Spline reticulation returned 0 results.");

  but it is far more common for it to be a string constructed with data available at runtime:

  logger.warn("Spline reticulation returned " + numResults + " results.");

  Also, it is not at all unusual for there to be significant amounts of work in constructing the string:

  logger.warn("Splines returned: " + splineList.toString());

  Because Java does early binding of parameters (as do most languages other than Haskell), the work to generate the string to be logged will be done EACH AND EVERY TIME the log statement is encountered, EVEN IF LOGGING IS DISABLED. So even if the log levels are set to warnings and debug logs won’t be printed, the examples above would still perform all the work to construct the string. This overhead can be enormous: I personally worked on a project where we sped up the entire application by a factor of 3 just be avoiding the creation of log lines for statements that would not be logged.

  There are three ways to avoid this overhead. One is specify the log level at compile time (lots of logging systems for C do this), but then it cannot be changed without recompiling. Another is to manually perform the (efficient) check of logging level before performing the work:

  if (logger.isLoggable(Level.DEBUG) {
      logger.debug("Splines returned: " + splineList.toString());
  }

  but this requires burying each logging statement inside of an if. The final option is what I call “Delayed String Construction”, or sometimes “parameterized log messages” and that is to have library level support for delaying the construction:

  logger.debug("Splines returned: {}", splineList);

• Log to Multiple Locations – Sometimes you want certain log entries to go to more than one file. For instance, you might want to send all logs to a single master log file, but also send logs about login errors to a separate security log which gets reviewed by a different group of people.
• Configure Where Logs are Directed – Usually you want logs to go to a file, but other times you want something more exotic like storing logs from multiple servers on a single remote location, sending logs to a messaging system, or writing logs to a database. This is particularly useful in conjunction with logging to multiple locations. So one useful feature, called “Appenders” in some systems, is the ability to provide a custom piece of code which processes the log entry: it could write it to a file OR do something else more exotic.
• Standard, or Widely Used – It is a big advantage if the logging system you use is widely used by others. One reason is the usual: standardized or widely used products tend to get improved over time while more obscure products tend to be forgotten and lose support. But in logging there is an additional advantage: if a library you are using or the server that is hosting your code also uses the same logging facility, then it may be possible (even easy) to comingle the log messages with your application log messages, which can be extremely useful when debugging issues which cross library boundaries.
• Sensible Fallbacks – As previous features have mentioned, logging can involve all kinds of complicated configuration and setup. In the real world, these will sometimes fail. Perhaps your logging configuration file cannot be found, or the log directory cannot be written to, or the logging database password fails — and the most common of these errors is to have no configuration at all. Different logging frameworks have different fallback behavior in the face of problems like this: some will disable all logging, others will fall back on a more primitive form of logging such as writing all logs to stdout if the logging is not configured. It is desirable for a logging system to have sensible fallback behavior.
• Logging Queued to Avoid Delays – In nearly all cases, logging is not the most important thing that your application is doing. The primary goal of the program is to reticulate splines, respond to user keystrokes, respond to web requests, or whatever it is that your application does; logging is a secondary concern. That being said, it is a real shame that writing logs slows down the application. It is quite common that the work done in response to a request can be quick and simple enough (retrieving some data that is already cached) that it is FAR less than the time required to open a file on disk and write even a single line of logging.One solution (with a few drawbacks) is that instead of actually writing the log immediately, logs that are to be written get put onto a queue, and they are written out by a background thread. The big advantage is performance: your program can quickly stash a few strings in memory and get on with its work faster; the log entry gets written later, when there are spare CPU cycles available. It can even save time overall by batching writes. There are a few disadvantages to be aware of: it will increase memory requirements (but typically by an insignificant amount); the queue typically needs a max size, and you need a policy for what to do when logs are being created faster than they can be written: typically the choices are either to discard some of the logs or to slow down the application. And it makes it difficult to debug issues that actually lead to crashes (because the logs don’t get written due to the crash). But significantly faster application performance is often worth these costs.
• Threadlocal Context Data – Imagine building a web application, which takes requests from logged-in users. And consider logs which are written from deep inside the spline reticulation code. For tracking down certain kinds of bugs, it would nice to know which user’s data contained the spline that was being reticulated. But we can’t just add that to the log message, because the spline reticulation code isn’t passed the name of the user. Some function 7 levels up the stack may have known it, but that doesn’t help to pass it into the log message.So a useful feature would be to have a command we could call at the point where we first identify the user submitting this request. This command could tell the logging system the name of the user and the logging system could store it in threadlocal storage. Then all future logs written from this thread could be annotated with “user=JSmith”, until this service call is completed.

  Two versions of this feature are offered by Log4J: “Nested Diagnostic Context”, in which multiple pieces of data can be kept in a stack, and “Mapped Diagnostic Context” in which multiple pieces of data are kept in a map.

• Log Filtering – Not all applications will have this requirement, but at large institutions with a concern for security, there are some things which ought not be written into a log. For instance, there may be a policy that passwords are not recorded, or that customer social security numbers never be stored in an unencrypted fashion. A fairly rare, and often imperfect but still useful, feature is the ability to provide some sort of filter which will recognize these kinds of data and remove them before the log data is recorded to disk. The difficult part, of course, is in recognizing the data.
• Internationalization – Many loggers have support for writing messages in unicode (particularly in languages like Java where the fundamental String type supports unicode). But some go further and provide support for loading the messages from a language-specific file and may have features like localization of dates and other information appearing in log messages.

So that’s my list of features; see my next entry where I use this list of features to evaluate some Java logging libraries.

My bank on the online banking login instead of having a password field it presents you with 3 password fields 1 character each where it asks you for 3 characters from your password, chosen randomly. E.g. the 2nd, 4th and 7th.

I wanted to respond to this, because not only is it an incredibly misguided attempt at security which seriously weakens actual security, it also sounds familiar. Because a few months ago my employer considered doing something just like this. Let me recount the story:

I work for a bank, so we care a LOT about security. Customers call into our call center and to identify themselves they get connected to the IVR (interactive voice response unit… telephone system) to enter their PIN (a 4 to 8 digit passcode). An important feature is that we cut out the phone reps from hearing this… because we want your password to be a secret EVEN FROM OURSELVES. All of this is good security design.

We opened up a new call center in Hawaii, and they had some problems. Apparently the phone system we were using had a time limit when transferring a call — if it wasn’t picked up by the remote phone switch within a few milliseconds then it was disconnected. The ping time between the Hawaii call center and our east-coast data center was just a little too long and many of the calls were being disconnected when they were transferred to the IVR to enter the PIN.

The first solution that they thought of was to stop using the IVR to enter PIN numbers. Instead, the idea was that they would instead create a system where the phone reps asked the customers for certain digits out of their PIN (just as described in enanoretozon’s reddit question). They would type this in and then the customer could log in. Apparently, this was the standard practice at our German subsidiary, and had somehow become blessed as the official corporate-wide best practice.

Well, it may be an official “best practice”, but it’s still a very bad idea, for two reasons. The first reason should be completely obvious if you just try it. First, say your phone number out loud. Now say the 3rd, 6th, and 4th characters of it. For most normal people, the second will take many times longer, and be much harder, even though it is only 3 digits. There is always a tradeoff between security and usability (We could provide perfect security if we never allowed anyone to take their money out of the bank. Of course, usability would have dropped to zero.), and entering random digits has SUCH poor usability that it is not worth it.

Random Digits

Besides that, it is also less secure. There are, if you consider it, multiple different kinds of attacks that we need to protect against. One kind, certainly, is attacks by unscrupulous bank employees who might misuse a customer’s login credentials. But another far more likely attack is a third-party who wants to steal from a customer’s account.

Such an attacker, if they didn’t know the customer’s PIN, would have to guess. To prevent repeated guessing, we will temporarily lock out a customer’s account after a certain number of incorrect login attempts. But a clever attacker would just try different customers, making just one guess for each one.

With (for instance) a 6-digit pin, the expected number of guesses before the attacker got one right is around 100,000. Long before an attacker managed to try even a small fraction of 100,000 guesses, we would have noticed what they were doing and put a stop to it. But we only ask for 3 particular digits out of the password, then the attacker only needs to try about 1000 times before she is expected to guess correctly. There is a good chance that we would catch that, but (particularly if they spoof their phone number) we might not.

So we traded better defense against a rare attack (we don’t hire a lot of employees who commit bank fraud) for much worse defense against a common attack (we detect and stop attempted attacks of various sorts every single week!). It is NOT an improvement.

So… after these points were raised, we chose not to implement our German counterpart’s policy. What did we do instead?

Very simple: we fixed the phone system so it could transfer calls properly.

The feature that we need to implement (or “story” in Scrum parlance) is an increase in the maximum number of customers that can be set at once. You see, there is a “feature” that limits the number of IDs that can be set at one time to about 200. (“About” 200 because most id’s are 9 digits long and they are separated by whitespace; the actual limit is 2000 characters, enforced in Javascript as the field is input.) So when they need to set an alert on 600 IDs, they run through the screen 3 times. When they have 2.5 million IDs to update they open up a “story” for the development team.

I think we asked someone why it was limited to 200 IDs. No one is quite sure, but it’s probably to avoid overtaxing the database query or running a middleware service that takes too long… something like that. “Sure,” we say, “we can increase the limit.” We figure maybe we’ll group it in chunks of 200 and call it in a loop or something. We schedule it to be worked on in this month’s “sprint”.

A couple of man-days of effort go into building it. Some testing determines that (on much less powerful dev hardware) a single call can easily handle thousands of IDs without running into timeout issues — more than that, actually, as we left a factor of 4 or 5 for safety. So the front end breaks the list into chunks of that size. We thought we’d build it to handle unlimited capacity, but there’s an IE6 bug (yes, our corporate overlords require the use of and obsolete broken browser) that limits us to about 60,000 IDs.

Our Corporate Overlords

So we have completed the feature and the business can now enter more than 50x as many IDs at a time. But that’s not quite the end of the story. Because as part of regression testing, our QA staff does some exhaustive testing of the screen, and they discover that there apparently isn’t a limit on the size of other field, the one that contains the alert message. We check the database table for the appropriate max message length, and it turns out to be exactly 2000 characters.

Wait… I think I’ve heard that number before.

Apparently, whoever built this page in the very first place accidentally limited the length of the wrong field. There never was a reason for a limit on the number of IDs processed at once… the limit came entirely because of a bug. Yet we’ve been living with this absurd limitation for several years, simply because no one ever questioned the limit. (Or if they did question it, they got some vague answer like “I assume it’s for performance reasons.”)

I’m sure there is some lesson we should draw from this experience… I’ll leave it to you to figure out what the lesson is.

Well, they released a new version and I hit the “upgrade” button. After that, my project didn’t work anymore. I tried for a day to resolve it and I just couldn’t understand anything. Finally I “solved” it by uninstalling Eclipse and reinstalling it, then following the tutorial steps to create a brand new project and copying in my old files one-by-one. Another full day lost (I can only work a couple of hours per day on hobby projects).

Surely they wouldn’t do it again, right? So I carefully saved everything and held my breath the next time Google released an upgrade. It promptly broke everything like last time. Only this time I solved it differently: I uninstalled Eclipse and did NOT reinstall it.

The big difference is that in the intervening month JetBrains had announced that a slightly-impaired version of IntelliJ IDEA would be available for free. The stripped down version doesn’t have support for GWT and App Engine (which the paid version does have), but it’s something I can use. At work, I use IntelliJ (properly paid for) but it’s awfully expensive to pay for my own copy at home. (Can’t use the same copy because that would disturb the corporate bean-counters, even though it is allowed by the license.) The stripped down version is fine if I can run from the command line.

There are instructions for running GWT via ant. And there are instructions for adding support for App Engine. But they are broken in (what I think is) exactly the same way that the Eclipse plugin is broken. Details from a forum posting led me to realize the problem was that it now needs a “javaagent” specified. A “javaagent” is some sort of a pre-processor that runs before main() — apparently introduced with Java 1.5.

So after following Google’s instructions, I now add the following: In my <hosted> target, along with the other <jvmarg> elements, I add a new one which looks like this:

<jvmarg value="-javaagent:${appengine.sdk}/lib/agent/appengine-agent.jar"/>

After that, I can build it using ant. I’ll also need to use the command line for deploys, that looks like this:

"C:\Program Files\appengine-java-sdk-1.2.6\bin\appcfg" update war

And now it works again.

As I see it, the big advantage of estimating in hours is that if you THINK in hours, you tend to get a more accurate estimate. There are lots of development tasks which will seem like they should take “no more than 2 days”, but if you think about all the individual steps (I have write create the page and the new service. And the stored procedure. And I’ll have to get a security review and a code review. And I have to remember to do the unit tests. Oh yes, and save time for bug fixes), the total comes out a big bigger.

As I see it, the big advantage of estimating in days is that it’s quicker and simpler. If you team is sitting there arguing whether a task is 3 hours or 4 hours, then you’re wasting time — after all, development estimates are never THAT accurate anyway: we always need to allow for the unexpected.

Considering these, I could be persuaded to do it either way. What is NOT useful is to think how many days it will take, multiply by the number of hours per day, then spend time arguing about whether it is one more or one less than this number.

But there is one catch: the tools are “dumb”, and there always needs to be a way for a knowledgeable human to override it. Usually it’s a special comment that tells the tool not to report a particular infraction. The override is used in that rare special case where there is a good reason why the rule does not apply.

Now, I know a certain portion of my readers are already forming objections, certain that THEIR pet peeve is the one case to which there should never be an exception. For instance, rules requiring consistent indentation throughout the project are good, but suppose some of your source files are from an external source? It’s not wise to reformat them just to placate the code scanner — that will make it harder to merge in changes when the next revision is released.

Or for another example, the capitalization rule that Java instance variables always begin with a lowercase letter seems very good, but when my team created an XML binding mechanism that mapped XML fields (which are case sensitive and often begin with a capital letter) to instance variables, being consistent with the XML file was more important than following the standard Java conventions.

I can still hear someone arguing. Some reader out there is still complaining that there is a special reason (security? correctness checks?) why some rules must be absolute. Frankly, I think this reader is just a control freak and they should learn to let programmers act like the professionals they are, but in order to convince you, I’m going to tell a story.

Larry Osterman

This is the story of a programming rule so VERY absolute that before you read the tale you’ll agree that there is NO possible reason to violate this rule. Yet after an automated code scanner discovered the violation, the catastrophic result was that for 2 years, all cryptography originating from a major operating system was completely insecure.

So… on with the story. (And many thanks to Larry Osterman, pictured here, from whom I first learned this sordid tale.)

Now, Linus Torvalds and his compatriots manage the source code for the kernel of the Linux operating system, but a usable system requires much more — a whole set of systems and applications which are tested and configured to work together. This is called a “Linux distribution” and there are several major distributions in wide use: SUSE, Fedora, and Mandriva among others.  The single most popular today is Ubuntu, which is based on Debian.

The folks at Debian collect code from a number of different projects and carefully review it, test for compatibility, and then certify the resulting distribution and provide a place to download the finished product. One of those components is OpenSSL, an open-source implementation of some cryptography libraries that provide support for basic functions like secure random number generation. And one of the tests that the Debian folks perform is to analyze the code with Valgrind, an automated code scanner that detects memory leaks and similar problems.

Valgrind detected that there was a problem in one of the OpenSSL routines that generated random numbers — it accessed uninitialized memory. Not knowing how to suppress the warning, someone at Debian decided to “fix” the code.

What they didn’t realize is that the OpenSSL developers had known exactly what they were doing. The crux of a cryptographically secure random number generator is an entropy collector. You see, the problem with generating “random” numbers on a computer is that nothing on the computer is truly random. You can mix and mash bits with the fanciest hash function in the world, but if the seed you start it off with is just the current time off the clock and an attacker can guess that (all but the last few digits are awfully easy to guess), then the attacker can repeat the same process and determine what “random” key was chosen, thus completely cracking your security.

So a cryptographic random number generator (as opposed to a garden variety RNG) goes to great lengths to collect “entropy”. It may start with the time off the clock, but it mixes in the number of milliseconds between key presses on the keyboard. And it mixes in the process ID of the OpenSSL process, data traveling over the network socket, micro-timing of the hard-drive motor and of mouse movements, and anything else they can use as a source of “randomness”. When they are first creating the data structure for the entropy pool, they intentionally left the memory uninitialized, because it’s yet another source of randomness.

Unfortunately, a Debian maintainer didn’t realize this. And they also made a minor error when changing the code: they accidentally commented it out in such a way that all sources of entropy other than the first one (the process ID) were multiplied by zero before being mixed in. So there was NO randomness except the process ID (which is very easily guessed).

Ubuntu

Then Debian was released with this bug, and Ubuntu picked it up and distributed it further. And 2 years went by. During that 2 years, everything done using the RNG on Debian or Ubuntu Linux is insecure because the keys are guessable. Everything! Any SSL connection made from such a machine. Any secure certificate signed by such a machine. And no one noticed for two whole years.

So the moral of the story is, don’t behave like that  ignominious Debian developer and change code that you don’t understand. But also realize that for any supposedly-universal rule, there is some special case exception. In almost all circumstances, the rule is good, but it is still wise to provide some way for the experts who do need to violate the rule to declare that they are doing so on purpose (preferably with a required explanation of why). Otherwise, someone who doesn’t know what they are doing is likely to break things.