As I see it, the big advantage of estimating in hours is that if you THINK in hours, you tend to get a more accurate estimate. There are lots of development tasks which will seem like they should take “no more than 2 days”, but if you think about all the individual steps (I have write create the page and the new service. And the stored procedure. And I’ll have to get a security review and a code review. And I have to remember to do the unit tests. Oh yes, and save time for bug fixes), the total comes out a big bigger.

As I see it, the big advantage of estimating in days is that it’s quicker and simpler. If you team is sitting there arguing whether a task is 3 hours or 4 hours, then you’re wasting time — after all, development estimates are never THAT accurate anyway: we always need to allow for the unexpected.

Considering these, I could be persuaded to do it either way. What is NOT useful is to think how many days it will take, multiply by the number of hours per day, then spend time arguing about whether it is one more or one less than this number.

costs.jsp

<% Cost[] costs = CostHelper.calculateCosts(loanData, currentDecision); %>
<h3>Current Costs:</h3>
<table border="0">
  <tr>
    <th>Cost</th>
    <th>Amount</th>
  </tr>
  <% for(int i=0; i<costs.length; i++) { %>
    <tr>
      <td><b><%costs[i].getName()%></b></td>
      <td>$ <%costs[i].getValue()%></td>
    </tr>
  <% } %>
</table>
<p style="text-align: center; font-style: italic">
  <a href="costHelp.html" 
      onclick="
        window.open(this.href, 'Help');
        return false;
      "
  >Explain these</a>
</p>

And it was confusing. But after a time we became enlightened and we realized that we should separate out the business logic from the rest of the presentation. We used tag libraries or velocity macros or any of a host of other technologies, and the code looked something like this:

CostController.java

Cost[] costs = CostHelper.calculateCosts(loanData, currentDecision);
modelAndView.addObject("costs", costs);

costs.vm

<h3>Current Costs:</h3>
<table border="0">
  <tr>
    <th>Cost</th>
    <th>Amount</th>
  </tr>
  #foreach( $cost in $costs )
    <tr>
      <td><b>$cost.name</b></td>
      <td>$ $cost.value</td>
    </tr>
  #end
</table>
<p style="text-align: center; font-style: italic">
  <a href="costHelp.html" 
      onclick="
        window.open(this.href, 'Help');
        return false;
      "
  >Explain these</a>
</p>

And it was confusing. But after a time we became enlightened and we realized that we should separate out the styling from the rest of the presentation. We used CSS markup in separate files, and the code looked something like this:

CostController.java

Cost[] costs = CostHelper.calculateCosts(loanData, currentDecision);
modelAndView.addObject("costs", costs);

costs.css

.costName {
  font-weight: bold;
}
.helpLink {
  text-align: center;
  font-style: italic;
}

costs.vm

<h3>Current Costs:</h3>
<table border="0">
  <tr>
    <th>Cost</th>
    <th>Amount</th>
  </tr>
  #foreach( $cost in $costs )
    <tr>
      <td class="costName">$cost.name</td>
      <td>$ $cost.value</td>
    </tr>
  #end
</table>
<p class="helpLink">
  <a href="costHelp.html" 
      onclick="
        window.open(this.href, 'Help');
        return false;
      "
  >Explain these</a>
</p>

And it was confusing. But after a time we became enlightened and we realized that we should separate out the dynamic JavaScript from the rest of the presentation. We called this unobtrusive JavaScript, and the code looked something like this:

CostController.java

Cost[] costs = CostHelper.calculateCosts(loanData, currentDecision);
modelAndView.addObject("costs", costs);

costs.css

.costName {
  font-weight: bold;
}
.helpLink {
  text-align: center;
  font-style: italic;
}

costs.js

$(function() {
  $('.popup').click(function() {
    window.open( $(this).attr('href'), 'Help');
    return false;
  });
})

costs.vm

<h3>Current Costs:</h3>
<table border="0">
  <tr>
    <th>Cost</th>
    <th>Amount</th>
  </tr>
  #foreach( $cost in $costs )
    <tr>
      <td class="costName">$cost.name</td>
      <td>$ $cost.value</td>
    </tr>
  #end
</table>
<p class="helpLink">
  <a href="costHelp.html" class="popup">Explain these</a>
</p>

And it was better. Things get a little bit confusing because each screen is rendered from several different files instead of from a single one, but separating out the concerns of business logic, styling, and dynamic behavior made each piece easier to understand and work with.

Recently, though, I became enlightened, and realized that we still have too many things going on in the same file. The HTML file still contains two independent concerns: the definition of the structure and the actual text. These facets are not really related, and they tend to be edited by different people. (HTML designers don’t necessarily produce the marketing copy and other text.) So this is a perfect candidate for another separation of concerns. That’s why my current project is organized like this instead:

CostController.java

Cost[] costs = CostHelper.calculateCosts(loanData, currentDecision);
modelAndView.addObject("costs", costs);

costs.css

.costName {
  font-weight: bold;
}
.helpLink {
  text-align: center;
  font-style: italic;
}

costs.js

$(function() {
  $('.popup').click(function() {
    window.open( $(this).attr('href'), 'Help');
    return false;
  });
})

cmstext.properties

cost-title: Current Costs:
cost-col1-label: Cost
cost-col2-label: Amount
cost-help: Explain these

costs.vm

<h3>#cms("cost-title")</h3>
<table border="0">
  <tr>
    <th>#cms("cost-col1-label")</th>
    <th>#cms("cost-col2-label")</th>
  </tr>
  #foreach( $cost in $costs )
    <tr>
      <td class="costName">$cost.name</td>
      <td>$ $cost.value</td>
    </tr>
  #end
</table>
<p class="helpLink">
  <a href="costHelp.html" class="popup">#cms("cost-help")</a>
</p>

The code that processes this allows no markup in the cmstext.properties file — any metacharacters will be properly escaped. So in principle it would be easy to give this file directly to the business to edit without IT help — or even to allow them to edit it on the live website without needing a QA cycle. The down-side is that we have yet another file to deal with: time will tell whether the benefits outweigh that cost.

SQL Injection

The most venerable is the SQL-injection attack (and related attacks for things other than databases). This is the danger that data entered by users will be treated as meta-characters not just as text, and will allow a visitor to your site to execute arbitrary code in your database (or some other system). Nothing illustrates this kind of attack better than this XKCD cartoon:

XSS

There are XSS attacks – which stands for “Cross Site Scripting” (the acronym “CSS” was already taken). XSS is the danger that your site might serve up some content which was entered by a user (or perhaps loaded from a 3rd party site). Normally, this makes sense: whole empires have been built on the concept of allowing users to customize their own pages on your site. But browsers assume that any code served up by your site is trusted, and will allow it to do things like accessing you site’s cookies. Particularly if users can get others to view their content, this can be quite dangerous. The first line of defense is to carefully sanitize and properly escape all content from users that it intended as plain text. As for content from users that is intended to contain markup, some sites just prohibit this, while others can use a solution like Google caja, which provides a secure sandbox for untrusted code to run in.

CSRF

XSS attacks exploited the fact that the user (or the user’s browser) trusted content coming from your website (even if that content wasn’t written by the site owner). The converse is a CSRF attack which exploits the fact that the site trusts content coming from it’s user’s browser. One of your users visits a malicious site which contains some sort of link to your site — preferably a link which causes some action to be taken. The request comes from the user’s browser, it has the user’s cookies and everything, so your site thinks the user intended to send the request. The idea is for a site containing pictures of cute cats to trick your banking site into transferring money to someone’s account. There are several defenses such as always using POST (helps, but doesn’t fix the problem), a per-session unique ID, and double-submit cookies.

Clickjacking

The latest hot issue in the web security community is “clickjacking“. This is a variant of CSRF in that it occurs when your site’s user navigates through a malicious website. But instead of the foreign site trying to generate a “click” on your website by embedding an image or executing some JavaScript, the foreign site embeds your site, perhaps within an IFrame. They then cover it up so you can’t SEE it, but when you click you perform an action on the attacked site. You can see an innocuous example at http://noscript.net/getit, where the “install now” button installs from Firefox’s site, not from noscript’s site. The most evil versions will continually move the attacked site so it stays under the cursor… no matter where you click, the button you don’t see will be under your cursor. Some of the details of this exploit have not been released yet, so it is too early to make clear statements about how a website (or a user) can protect themselves.

Plenty of others

I am sure that there are other fundamental forms of attack against web applications. Go ahead and add your own items in the comments below!

I am sending email copies of the technical postings to this blog to an email list of people at my work (I suppose I’d accept others also) who expressed interest. It started as a bonus goal project. Here is the process I came up with for sending the email from Outlook:

1. First, publish the blog entry to the web normally.
2. Launch Outlook
3. Go to “Tools > Options > Mail Format” and set the email format to HTML while disabling the use of Microsoft Word for editing emails.
4. Go to “View > Toolbars > Web” to enable the web toolbar.
5. Navitage to the website (http://mcherm.com/) and follow the link to the story to be emailed. Manually add “?stylesheet=outlook” to the end of the URL and hit return to view it.
6. Go to “Actions > Send Web Page by E-mail”. This will launch Outlook’s terrible, almost unusable HTML editor. The page will look entirely wrong – for instance, it will ignore the stylesheet. Fortunately, only one thing really has to be fixed (everything else will look OK when viewed. That one thing is to delete the first three characters at the beginning of the page. (Sometimes is messes up quotes too… you can check the body of the story.)
7. Now copy the mailing list and add it to the BCC field. Be sure it’s the BCC field, not the CC or TO field, to protect the privacy of the message recipients.
8. Fix the email subject. It should read “Technical Essay Series: <title-of-article>”.
9. Hit send.

You probably have antivirus software protecting your computer from malicious computer viruses — in fact, you’d be a bit nervous about connecting to the internet if you didn’t have it. After all, viruses can make everything run more slowly, take over your computer and use it to send spam, or worst of all they can even delete files and destroy data. There are even a few viruses that take your data hostage — encrypting it, deleting the original, then offering to decrypt it if an anonymous payment is made to a certain account.

It is less common these days (hard drives have gotten much better), but occasionally you’ll have a catastrophic failure — your hard drive will crash with no way to recover it. This can be a real disaster: the cost of a new hard drive is nothing compared to the loss of all your precious data.

Or it’s even POSSIBLE (unlikely, to be sure), that while you were cooking dinner your five-year-old got onto the computer you had left on (despite KNOWING he wasn’t supposed to) and sort of accidentally dragged into the trash a folder containing some files — files you had been working on for quite a long time. And, unlikely though it sounds, it’s possible that you didn’t notice this until AFTER you had already emptied the trash can and deleted the files forever.

Of course there is one solution to ALL of these problems — backups. Everyone knows that they ought to back up their files. But not everyone gets around to doing it; in fact nearly everyone doesn’t – until right after they have an accident and lose data. In fact, I am guessing that you, yes YOU have data someplace (probably at home) that isn’t backed up properly.

There is an obvious “right” way to do backups; just follow a few simple rules. First, the backups have to be frequent — a 9-month old backup isn’t going to be very useful; every few days is about right for most people. Second, you MUST test your backup. A backup process, even one designed with the best of intentions, can’t be considered safe unless you have tried doing a restore from it. And succeeded. It doesn’t matter whether the backup is to CDs, tape, a spare hard drive, or over the internet, but you should store the backups someplace separate from the computer (otherwise they won’t be much help in case of a housefire).

And doing all this is a pain… particularly the part about doing it regularly. Murphy’s law says that you won’t lose data until sometime after you finally become lax and start skipping your backups. It is FAR better if the backups can be done automatically… so if you are lazy like the rest of us it will still get backed up. But for most of us, setting up some automated system is even harder than doing backups.

So what you really need is some product which will automatically back up your files for you every few days. It should be trivially easy to use, should take almost no time to install, and should work even if your computer isn’t on or net-connected 24/7. It should make restoring files a breeze (and not something you do only in case of catastrophe) and should automatically warn you if backups are NOT working for some reason. It should not require purchasing any special tape drives or other hardware. Oh yeah, and it should be free, or at least cheap.

Fortunately, such a product exists! Actually, several such products exist, but I’m just going to talk about my favorite which is called “Mozy” (it’s for Windows and Mac… you Linux users will need to find a different alternative). Mozy’s software installs quite easily. Once installed, it asks you to create an account: home use is FREE if you need to back up less than 2GB, and costs $5/month for unlimited amounts. (They have small business backup solutions also.) It will ask you to specify which directories need to be backed up and will send the data over the internet to Mozy’s servers. The documentation says that it is encrypted, so Mozy themselves cannot access your data without the password. (In case you were wondering, that’s a good thing.)

After the initial setup, Mozy runs without any supervision. It waits for a time when your internet connection is up and running but you are not currently working on your computer, and then it sends the changes (only the changes) from the backed-up directories over to Mozy’s servers. If it does not manage to do a backup within 7 days, then it begins to warn you that the backups are not current. And any time that you want to get files back, you can review the files stored on the server and restore any amount from a single file to the entire backup set.

There are a couple of features that I would want in an ideal backup system which Mozy lacks. I would want the entire disk backed up, including the OS so that I could completely swap in a new disk if needed. And I would want all versions of a backed-up file to be saved, not just the latest, so I could return to previous states of things rather like a version control system. But Mozy provides the most important features: it backs up my data, and it is very easy to use (good, because I’m too lazy to use it otherwise).

So now we come to the wager. At this point, as I predicted, you are thinking to yourself “Gee, this Mozy sounds like a good idea. I should really install it on my own computer.” But it’s one thing to THINK about it, and quite another to actually go DO it. There is a certain mental inertia to overcome. So I’m placing a small bet – shall we say $5? – that you won’t actually step up and do it. If I lose… if because of reading this article, you actually DO go install Mozy or another automated backup tool, then post a reply using the reply box below, and I’ll pay up on my bet. (For the record: Yes, Mozy does have an affiliate program but No, I am not participating. I get no kickbacks with this deal – just the warm fuzzy feeling of having saved someone else from losing their data. (Offer good only for the first 12 days after this is posted. Offer may be rescinded if this gets posted to the front page of Slashdot, Digg, reddit, or the like.))  You could have your data safe AND win $5 at the same time… what are you waiting for? Or are you too lazy?

Following a hint I learned from the web, here is the procedure I came up with.

1. Launch Outlook.
2. Go to “Tools > Options > Mail Format” and set the email format to HTML while disabling the use of Microsoft Word for editing emails.
3. Go to “View > Toolbars > Web” to enable the web toolbar.
4. In the web toolbar, navigate to the page containing the HTML you want to send. If the page you are sending is not an existing blog post, you can write it by hand in the HTML editor of your choice then navigate there.
5. Go to “Actions > Send Web Page by E-mail”.
6. This will launch Outlook’s terrible, almost unusable HTML editor. Fortunately, it will prefill it with the content of your web page, so if the web page was correct then you should be able to fill in the headers (subject and To) and just hit “send”. I found some character set problems (e.g.: EM dashes replaced with patches of line noise) so you may want to watch out for these.

I still recommend creating a sample and sending it to an email address beforehand. Test every link and image to make sure that they’re working before you send out the mass mailing.

No one at Google decides how to rank the sites, they rely on how other sites throughout the net link to them. The administrators of MySpace and facebook don’t produce any of its content. Wikipedia doesn’t employ experts to write its articles. The owners of YouTube don’t film videos themselves. Craigslist itself doesn’t buy or sell anything. Digg doesn’t write articles, select them or even comment on them. These (and many others) rely on user generated content.

It’s pretty clear that there are huge profits to be made off of content that other people produce for you. But there is a trick to doing it properly, and some of these sites “get it” while others don’t. The secret is nothing new: businessmen have said for years that “The Customer Is Always Right“. With user generated content, it is more important than ever to treat your customers well—even if you don’t “have to”. Here are some examples of companies that understand this and some that don’t.

Google says they make 40% of their revenue from AdSense, and I’m surprised it’s not higher. Everywhere you go on the web, you see small Google ads. To the owners of the small websites they run on it’s great: a way to make money with very little effort. To the advertisers, it’s great because they don’t have to pay unless people click on the ads. Even the viewers (not normally someone that advertisers cater to) are mostly pleased to have small inobtrusive text ads rather than huge flash animations that cover up the page they’re trying to view. Everyone is so happy that they don’t mind Google making off with a huge chunk of the revenue.

Wikipedia seeks to resolve conflicts by consensus and by maintaining a neutral point of view. Yet the people who run the site often silence any criticism of themselves or the way they run it.

YouTube doesn’t plaster the screen with ads. But they recently introduced the practice of putting ads next to a video—but only if the poster permits it. And they share part of the revenue with the poster. Somehow, the YouTube community does not seem to be up in arms about this new advertising: it’s almost as if they think it is fair.

Facebook recently decided to partner with numerous businesses including Blockbuster, and tell all of one’s “friends” every time a user rented a video (or whatever the business was selling. Although it was probably illegal, this was a great business deal for the companies involved: Blockbuster gets more “shelfspace” in their customer’s minds, Facebook gets a useful business partnership. The only ones hurt are the customers who just might not want everyone knowing what video they rented last night (at least Blockbuster doesn’t rent porn).

Amazon recently started highlighting one good and one poor review for each product. This may hurt the sales of certain products, but it won’t hurt Amazon.

The message is clear. If all you have is your customer’s content, you’d better make certain that you treat them well.

What is Miro?

Miro is a video player. You launch the program and it displays a list of “channels”. Each channel contains a series of videos. Some could be short (a couple minutes, like YouTube videos), others longer (like whole 1-hr TV episodes). Click on a video and it downloads (unless it pre-loaded) then you can watch it.

The VLC video player

Video today is a mess of different formats. Wikipedia currently lists 22 different commonly used video codecs, and that’s not to mention the other levels of formatting aside from the codec itself. There are numerous sorts of DRM applied and different delivery mechanisms. Aside from the passionate videophiles who invent these encoding schemes, no one really cares what format they have — they just want their video to play properly (and look decent!). Instead of having to remember what to view using Real Player and what to use Windows Media Player or iTunes for, we’d rather have something that just knew how to play everything. VLC is an open-source (GPL’ed) video player that handles lots of different formats. It plays small or full-screen and the controls are intuitive.

Intuitive UI Design

The UI for VLC is really quite easy to use. “Channels” (really just RSS feeds with video content) appear on the left-hand side of the screen. They can be organized into folders if desired. Viewing a channel shows the content of that channel — in blue if not yet downloaded, orange if currently downloading, and green if already downloaded. Click on a blue item to start downloading it, or click on a green item to play it. You can also set a channel to automatically download new videos when they appear — use this option in combination with running Miro in the background and you’ll find your video already downloaded and waiting for you when you get there! And there is a search and ratings system for discovering new channels.

Miro also allows one to download video from several major video sites, like YouTube, Google Video, Revver, and so forth. For these it provides a search capability for finding content. You can even save a search as a “channel” in the main Miro interface. Moving a step beyond what YouTube and all the abilities to pre-load video, save it on disk, and so forth are all available even for this content.

BitTorrent Distribution

Video files can be fairly large. Posting such a file on your server to be downloaded by a couple of friends is quite manageable, but if tens of thousands of people want to download you’ve suddenly got a problem. If your video becomes popular, then your bandwidth bill might suddenly become enormous — or it might not even matter because your server might fail while trying to handle the load.

It would be great if all of those tens of thousands of people who want to view your video could help. If everyone who downloaded it were to pass on a copy to someone else, that would be enough bandwidth to transmit it everywhere. BitTorrent is a protocol for doing exactly this — with a few nuances, like breaking the file into many small bits and distributing them so no one has to stand in line waiting for someone upstream, and like passing around hashes of the pieces to protect against transmission errors or intentional modification of data. Miro uses BitTorrent to distribute the files, and it automatically “pays back” by helping to upload files that you have downloaded as long as they are available on your drive.

A Thoughtful Approach to Copyright Law

BitTorrent is a wonderful tool, but it gets a bad name because its most common use is to distribute pirated copies of video and audio. For that matter, many video and audio distribution sites have been shut down due to legal issues with copyright law (remember Napster?). Many people (myself included) feel that the current laws in this area ought to be revised, but it is difficult to make a case for this when one is viewed as a “pirate” or lawbreaker.

Miro is owned by a non-profit called the Participatory Culture Foundation. This group’s board is staffed by well-known luminaries such as Cory Doctorow (“the guy from BoingBoing”) and John Lilly (“the guy who runs Mozilla”). Even more significantly, the PCF has taken the “right” approach to copyright law.

Feeds can come from anywhere (including searches on sites like YouTube which are famously full of unauthorized copyrighted material. And if you get your feed from someone who distributes pirated material, it will work fine with Miro. But the feeds which Miro advertises, and the ones which it lists on their website, are all reviewed by the PCF in an attempt to ensure that they only contain material with an open (Creative Commons) license. Of course, at some point someone will slip in some inappropriate material, but the folks who run things at Miro are doing their best to prevent it.

And this is exactly the right approach to take. There are plenty of people who want to share their work. Encourage them to do so legally. Encourage the Miro viewers to view these fully-legal videos (by featuring them in the Channel Guide). If some Miro users also choose to take in some pirated works, well… that’s their own choice.

Recently, Comcast has been interfering with BitTorrent. They get away with this because… well, they get away with it because they are a monopoly, but they feel like it is acceptable because they figure that most people using BitTorrent are pirates. Miro’s role as an “upstanding citizen” here is an important one.

It Will Change Television Forever

Television will be changing quite dramatically in the very near future. Already one wave of changes has replaced advertising-supported free-over-the-air broadcast of several channels with pay-to-subscribe to a few hundred channels delivered over cable. [PS: I don't have cable. But everyone else does.] Tivo and it’s ilk are currently changing us from shows-scheduled-by-broadcaster to your-choice-of-shows-available-when-YOU-want and breaking the old advertising model at the same time. An even bigger change will come when “anyone” can produce shows, instead of the select few networks acting as gatekeepers.

Imagine a future where shows are distributed by BitTorrent on the net (almost no cost to distribute!). Where many shows are produced on a shoestring by dedicated fans, and those with bigger production budgets are supported by a combination of advertising, product placement, and donations. Where viewers can see what they want whenever they are ready — even catch up on old shows after they discover a new series. It’s a world where Wonderfalls and Firefly would never have been canceled. Where open licensing would allow fans to create follow-on materials that would boost the value of the core shows enormously. And the only technological problem keeping us from this world today is that not every household has a copy of Miro installed.

But you could solve solve that one (for your household anyway) right now.