If you ever see “Host error number XXX”, it means that this was the XXX’th error of the day that this Profile instance wrote to the logs. Get someone to look it up in the Profile logs.

Also, Calling mrpc ZWRAP with [925, 8864, ""44758220"", |!|] will fail if 8864 is not a valid profile userid (which is the case for me).

InputStream does not represent a valid SOAP 1.1 Message

check the namespace of the SOAP envelope

SOAP 1.1: http://schemas.xmlsoap.org/soap/envelope/

SOAP 1.2: http://www.w3.org/2003/05/soap-envelope/

What I found most interesting was something that David Pollak (the post’s author) alluded to but did not emphasize: an example of a language that has solved this problem. Surprisingly, it is the much-maligned Java. (And perhaps this feature is one of the reasons for Java’s success in “the enterprise”, where backward compatibility to old or unmaintained libraries is often a very big deal.) The Java solution is to provide an incredibly strong amount of backward compatibility at the binary level (not just the source). As far as I know, essentially all code written under Java 1.0 (16 years ago) will still compile under the most recent Java release, and code compiled by that Java 1.0 compiler will still run under the most recent JVM. The price paid is some real ugliness in the name of backward compatibility like old APIs that still return Hashtable or ArrayList instead of Map or List, and type erasure that makes typed collection less powerful than they could be). But however much you may scoff at Java for poor language design, this feat of backward compatibility is something quite impressive.

There really isn’t much benefit to putting in lots of hours developing a detailed estimate if the project isn’t going to proceed according to plan anyway (and it rarely does). This is why most agile development approaches — including Scrum — use a less-precise but also less time-consuming approach. By going with rough requirements, and a simple imprecise estimation process a team can produce rough estimates in a surprisingly short amount of time. The time saved writing requirement documents and producing estimates can be used to build something useful instead.

The process that I have found to be most useful starts out with requirements that are simple: just a paragraph or two written down for a feature and a few minutes discussion to make sure everyone understands it. The team meets, making sure to include someone from the “business side” who can answer questions about what is needed, the developers, QA, DBAs, and whatever other specialists are needed. The business person explains what is needed; the team talks through how they will code and how it will be tested. Then we’re ready to estimate.

Everyone just says how long they think it will take. To avoid “groupthink” where everyone just agrees with the first person to speak, it’s good to have each person come up with their idea independently before comparing: selecting cards and all revealing at the same time is one way to do this. Everyone estimates: yes, that means the DBA may estimate a Java coding task, but that’s OK. To avoid long useless debates over whether it’s 23.2 or 23.4 we usually limit the estimated sizes to some discrete values: 1, 2, 3, 5, 8, 13, 20, and “more” are a widely used set of values (the values chosen to make it easy to split a task). If, after hearing what was said we all agree on the size then we’re done (this is where we all discount the DBA’s estimate of the Java task); if not then we discuss for a few more minutes: maybe someone realized an extra step the others missed or knows where to find test data without having to enter it. If we still disagree after that, just take the larger estimate.

That’s it! It takes only a few minutes to produce estimates this way. Of course, the estimates are worth what you put into them: the business MUST realize that these are only rough numbers. A common way to do that is to estimate in “Story Points” instead of “hours” or “days”. Speaking in terms of a unit that is less concrete seems to help remind everyone that this is only a rough value. But it is a rough value that did NOT take weeks of preparation, and thus well worth it.

You really only have to do it for data that is not read-only. I would for example say that it’s pretty rare for classes to change after they have been set up for the first time (presumably before any threads are even started), making the class basically read-only, which could be safely shared across threads.

I wanted to give a detailed response with why this approach is nieve. Actually, it has been tried before and failed. It may work OK with certain kinds of languages (mostly “functional” languages), but fails with other kinds of languages, and Python is an extreme example of the kind of language where it won’t work.

For an example, consider Java. The JVM (Java Virtual Machine) has special features that were added to support exactly this behavior, but in practice few programmers use them. Let’s take a really simple example: suppose you create some data structure and a function to initialize it. In thread A you create the object, then initialize it, then pass it off to existing threads B and C. Threads B and C simultaneously read stuff from the data structure in ways that WOULD be dangerous except that the data structure is immutable after initialization.

The problem is that the guarantees provided in threading are MUCH weaker than you think. It’s not just that there are different threads all working at the same time and reading and writing from the same memory locations, the architecture of modern CPUs makes that impossible. You see, it takes hundreds of times longer to read something from memory or write it to memory as it takes to process something in the registers. So to execute “X = Y + 1″, the computer COULD spend 100 cycles reading Y, then 1 cycle adding 1 then 100 cycles writing X for a total of 201 cycles to execute. But that would be unbearably slow. Instead, it takes 100 cycles to do a bulk read of the whole memory area around where Y is stored into high-speed caches. It takes another 100 cycles to do a bulk read of the whole memory area aroudn where X is stored. It takes 1 cycle to add, then takes 100 cycles to do a bulk write of the memory area containing X. That’s 301 cycles… which sounds even worse.

But it’s NOT worse if the compiler cheats. Instead, it spends 100 cycles reading Y and 100 cycles reading X. Then it executes the +1 for one cycle. Then, BEFORE writing out X it does some OTHER calculations on the chunks of memory that have been read in. If the program has good cache locality (active objects are near each other in memory) it may get 75 cycles of useful work done before it needs to spend 100 cycles to “flush the cache out” (write X and the other things that were updated. That would be a total of 375 cycles to do 75 bits of work, or just 5 cycles per line — a LOT better than 201!

But in order to do this, the compiler has the “cheat”. It has to execute bits of work out of order, although it can take special precautions to make sure that it gets the same answer as if it executed them in the order written. As seen by THIS thread. But as seen by a DIFFERENT thread, the steps may appear to happen in a very different order. The other thread won’t see the effects until they get flushed to main memory, and that won’t happen after every computation (unless it is running 100x too slow!!!).

WHEW!! Big wall of text there, but the story should explain why one thread in a program may see the computations by another thread happen in a different order. So imagine this:

“In thread you A create the object, then initialize it, then pass it off to existing threads B and C.”

But imagine that from thread C’s point of view, A created it, then passed it off, and only initializes it LATER. In fact, perhaps C will start using it at the same time that A is initializing it — so it’s not really immutable, and terrible errors result. This is NOT just a theoretical risk: I have written real code that exhibited this behavior when running on a multi-core machine.

In order to help protect against this, the Java langage added a special exception to the Java threading model. Despite all other threading rules, if a class is declared “final” (immutable) and then all code executed within the constructor is guaranteed to be occur before the constructor ends EVEN AS SEEN BY OTHER THREADS. In theory, this is a great tool for creating data structures ahead of time and then reading them after initialization from other threads.

But in practice it isn’t so good. Initializing everything within the constructor of an immutable object turns out to be a real pain. Often you really want to use a hashtable (not immutable), or use Spring injection to populate your objects after the constructor, or slew of other choices that make it hard to stuff all your setup code inside of constructors. Some languages support this better: Scala and Closure are examples of languages on the JVM that use this feature well, but in Java it is awkward because there’s no little special support for working with immutable objects. Python, as a language, is even worse: there are NO immutable objects in Python! So while it might be possible to do as you suggest (create special locks and use them around every __init__ method, then carefully make sure nothing is modified outside of __init__), the resulting language wouldn’t really read like Python.

There are a few really good reasons for wrapping a library. The most important of these, is in order to add functionality or simplify use of the library. For instance, in a recent project we used Spring’s library for web service calls in order to make calls to our own company’s collection of web services. But when calling our web services, there are a bunch of things that would be nice to do. We always want the same value for the address to connect to, the timeout for the calls, and the set of headers to provide. We want additional special handling for errors wrapped around every call. Adding these features in a wrapper makes the wrapper less powerful (now it’s good only for calling our services whereas Spring’s original library could call any web service), but at the same time makes it much more useful for that one specific purpose.

I have also seen cases where the existing library had a terrible interface (API), and the wrapper attempts to make it palatable. The “Slick” library is a Python[ed] Java wrapper around LWJGL adding no real functionality but making it decent enough to use. This is a rare use case: most libraries that have a terrible interface also have lousy features and you’re better off finding a different library instead.

The most common argument that I hear is neither of these cases: the most common argument that I hear is that we should wrap the library so we can easily switch to a different library. In fact, I most often hear this from people developing in a language with strong typing, such as Java. I find this argument completely unpersuasive, for two reasons. First of all, when you switch libraries, the new library typically does not have exactly the same API. For example, when we switched to SLF4J for logging one of the reasons for doing so was that it offered a better API that allowed functionality not possible with the previous API. Secondly, if you DO switch to a library with an API that is equivalent, in a strongly-typed language you can use standard refactoring tools to perform the switch without any risk of introducing bugs. (If the APIs are close enough a simple search-and-replace for an import statement may do it.)

There are advantages to using a library directly. Developers who have encountered the library elsewhere may already be familiar with it. The documentation for the library is likely to be far more extensive than the documentation for your wrapper. It is often safe to assume that the designers of the library are better at designing an API for this feature than you are. As the library is upgraded, newer features will automatically be available. Most of all, having one fewer layers means there is simply less to learn to understand the system.

There are still a few advantages to wrapping without extra features. It gives you a place to add some logging code, or timers around an external call (for profiling), or validation checks. And there are some cases where you want to be able to use different libraries with the same codebase — then a wrapper is indispensable. Commons Logging is an example of this: it allows a library to use different logging frameworks depending on what application it has been embedded in.

So my approach to the “wrap or not to wrap” question goes like this. First of all, will I add functionality with my wrappers or will removing functionality but thereby simplify the interface? If so, then wrapping makes sense. Secondly, if I haven’t yet chosen which library to use or if I want to switch back and forth between libraries, then a wrapper will be required. If neither of these applies, then I begin with a strong presumption that I should use the library on its own, and only a real need to add logging, monitoring, or other wrapped behavior will persuade me otherwise.

In spring-ws, in the class org.springframework.ws.soap.saaj.SaajSoapMessage, in the method getImplementation(), it uses SaajUtils.getSaajVersion(SOAPMessage) to determine the SAAJ version of this message. Unfortunately, that has a bug in it (or at LEAST a poor design) which can be quite confusing.

getSaajVersion() calls SOAPMessage.getSOAPPart().getEnvelope(). If the message was badly formed (in my case, it declared an xsi:schemaLocation, but failed to declare xmlns:xsi) then this is where the exception will be thrown. The code in getSaajVersion() then catches any generic SOAPException and swallows it (assuming the message must be SAAJ_11). The “parse was invalid” SOAPException will be ignored.

That wouldn’t be so bad if it weren’t for the fact that the SOAPMessage is mutated. The getSOAPPart() and getEnvelope() start with “get” so it suggests that they do not mutate the SOAPMessage, but in fact they DO! In the normal flow of processing, this just happens to be the first time that the content of the message gets looked at. During the first time, the InputStream is read, but after that first time it won’t be read again. So the NEXT time these are called, they will behave differently (returning null instead of throwing an exception). This confused me for a couple of days, thinking I had 2 different errors (it was returning null, and my SAAJ version was messed up), and producing the truly bewildering behavior (which should have tipped me off) that I could fix things by looking at something in the debugger.

#
# Programming challenge from
#  http://blogs.msdn.com/b/ericlippert/archive/2010/09/09/old-school-tree-display.aspx
#
# Done in Python, by Michael Chermside
#

import unittest
import itertools

# ============== Provided Problem ==============
class Node:
    """This class is, by gentleman's agreement, immutable.
    Do not modify it after creation."""
    def __init__(self, text, *children):
        self.text = text
        self.children = children

# ============== SOLUTION ==============
NEWLINE = '\n'

def dumper(root):
    """Passed top node of the tree. Returns string for the tree."""
    return ''.join( dumper_helper(root, ()) )

def sequence_and_item(sequence, item):
    """Returns an iterable (one which can be iterated multiple times)
    whose items are taken from the given sequence, followed by the
    given item."""
    return [x for x  in itertools.chain(sequence, (item,))]

def dumper_helper(node, prefix):
    """Generator which is passed a node and a sequence of prefix
    strings to be applied for each line. (Note: it must be possible
    to iterate the prefix sequence multiple times.)

    Yields a series of strings that, when assembled, will contain
      (1) the node name,
      (2) a newline,
      (3) line contents for all its children, each preceeded by the specified prefix."""
    # -- the node name --
    yield node.text
    # -- newline --
    yield NEWLINE
    children = node.children
    if len(children) >= 1:
        # -- All but the last child --
        for child in children[:-1]:
            for x in prefix:
                yield x
            yield '├─'
            for x in dumper_helper(child, sequence_and_item(prefix, '│ ')):
                yield x
        # -- Last child --
        child = children[-1]
        for x in prefix:
            yield x
        yield '└─'
        for x in dumper_helper(child, sequence_and_item(prefix, '  ')):
            yield x

# ============== Unit Tests ==============

class TestDumper(unittest.TestCase):
    def test_min_tree(self):
        tree = Node('a')
        self.assertEqual(dumper(tree), 'a\n')
    def test_one_top_child(self):
        tree = Node('a', Node('b'))
        self.assertEqual(dumper(tree),
                         'a\n'
                         '└─b\n')
    def test_some_top_childs(self):
        tree = Node('a', Node('b'), Node('c'), Node('d'))
        self.assertEqual(dumper(tree),
                         'a\n'
                         '├─b\n'
                         '├─c\n'
                         '└─d\n')
    def test_three_levels(self):
        tree = Node('a', Node('b', Node('c')))
        self.assertEqual(dumper(tree),
                         'a\n'
                         '└─b\n'
                         '  └─c\n')
    def test_full_to_three_levels(self):
        tree = Node('a',
                    Node('aa',
                         Node('aaa'),
                         Node('aab'),
                         Node('aaz')
                    ),
                    Node('ab',
                         Node('aba'),
                         Node('abb'),
                         Node('abz')
                    ),
                    Node('az',
                         Node('aza'),
                         Node('azb'),
                         Node('azz')
                    )
                )
        self.assertEqual(dumper(tree),
                         'a\n'
                         '├─aa\n'
                         '│ ├─aaa\n'
                         '│ ├─aab\n'
                         '│ └─aaz\n'
                         '├─ab\n'
                         '│ ├─aba\n'
                         '│ ├─abb\n'
                         '│ └─abz\n'
                         '└─az\n'
                         '  ├─aza\n'
                         '  ├─azb\n'
                         '  └─azz\n')
    def test_erics_example(self):
        tree = Node("a",
                   Node("b",
                       Node("c",
                           Node("d")),
                       Node("e",
                           Node("f"))),
                   Node("g",
                       Node("h",
                           Node("i")),
                       Node("j")))
        self.assertEqual(dumper(tree),
                         'a\n'
                         '├─b\n'
                         '│ ├─c\n'
                         '│ │ └─d\n'
                         '│ └─e\n'
                         '│   └─f\n'
                         '└─g\n'
                         '  ├─h\n'
                         '  │ └─i\n'
                         '  └─j\n')

if __name__ == '__main__':
    unittest.main()

DEVELOPMENT AND DESIGN NOTES:

(1) I chose to write this in Python (Python 3). I don’t know .Net that well and didn’t have a compiler handy. Python seemed a good, readable choice. Its performance characteristics are different, but I don’t care about performance.

(2) I chose to write unit tests. Unusually for me, I wrote this using “Test Driven Design”: writing a test, then afterward modifying the code to make it work. TDD is perfect for this kind of small, contained problem when I have a fairly good idea of how to proceed but I need to be careful of subtle errors (always a danger with recursion).

(3) I chose a functional, not imperative, approach. This was for two reasons: because I wanted to practice that style of coding, and because the nature of the problem (tree processing) is well-suited for such an approach.

(4) I was NOT smart enough to write perfectly clean code on the first try. There was a final pass where I renamed functions and variables and removed unnecessary layers.

(5) Nor was I smart enough to see the elegant recursive subroutine on my first glance at the problem. At first, I passed the list of children (and a prefix) to my recursive subroutine and returned complete lines. Partway through, I looked at the structure of my code and realized that it was far cleaner if I passed in nodes (and a prefix) and returned “the node name, a newline, and the lines underneath it”. Perhaps I have figured that out from the start by examining the sample output better… but the cool thing was that making the change was a simple, easy transformation, and realizing there was a cleaner approach came from looking at the structure of my code. So I conclude that just starting in without a perfect understanding of the ideal recursive unit was *just fine* because I could clean it up later with refactoring. This was, perhaps, the most interesting thing I learned from the exercise.

(6) I chose to write my recursive function as a “pure function” (no side effects, immutable arguments). So a call returned the output for that sub-section of the tree. The other alternative would have been to pass some sort of “output object” to the function that it would write to as it went. I chose this approach mostly to practice functional programming. I feel it still came out very readable.

(7) At first, I was returning strings. But I felt bad about constantly building strings. After all, strings are immutable (in so many modern languages, including Python, .Net, and Java) and I worried that all the string manipulation would be a performance problem. Even though I didn’t care about performance, I chose to return lists of lines instead. Then I realized I could return lists of bits of string and never have to concatinate any strings until the final step. This approach would look perfectly normal in a language like Haskell. By the way, I never did actual performance testing so my belief that the string manipulation would be slow might be completely wrong. I know that intuition is unreliable for performance questions, but one often still must make a decision without complete information.

(8) Python has this nifty feature called a “generator”. It’s a function where you use the keyword “yield”. The compiler turns the function into one returning an iterator which will give out the values you would get if you ran the function and each “yield” put an item onto a list. Except that it does not actually run any bit of code until needed, and it does not materialize the list in memory. Another way to think about it is that when you pull a value from the iterator, the function executes until the first “yield” statement, then it exits. The NEXT time you pull a value from the iterator the function picks up where it left off with all instance variable state intact and continues until the next “yield”. It is a VERY cool feature, and I used it because of the amazing way it made my function be very readable. Other languages should adopt this idea!

(9) There was one place where I wanted to do something simple, but the most readable syntax I could find for it was unbearably ugly and complex. So I made it a subroutine, just for the sake of readability and of having a function comment explaining what the heck it was doing. That is why the function “sequence_and_item” exists.

dg.DirectGatewayException: Error on call to mrpc ZMRPCSETRATE with parameters [PRIME, 19/08/2010, 3.75], error message was '3'.

The mysterious “error message was ’3′.” is the key here. The cause is an error in the way that the exit is coded from the MRPC. When it generates this error it is because the author attempted to return a value via the QUIT statement:

..do stuff..
Q 1

Instead, they should have set the return (or ret) parameter and then returned nothing via the QUIT statement:

...do stuff..
S RET 1
Q ""

With this change, the error message goes away.

1. Severity – mandatory: no logging system should be without this
2. Tree of Log Topics – mandatory: no logging system should be without this
3. Configurable – vital: configuring log levels at runtime is something we use often
4. Rotating Log Files – vital: our log files would be too big for the OS without this
5. Configurable Log Line Format – vital: it is unlikely that the off-the-shelf fields would be the ones we want to use
6. Logging of Exceptions – vital: getting stack traces from logs is one of our most productive debugging techniques
7. Delayed String Construction – vital: I consider this to be an very undervalued feature. Without it, software will be slowed significantly and also will be less readable.
8. Log to Multiple Locations – desirable: sometimes this is handy. We used to use it, but at the moment we don’t.
9. Configure where Logs are Directed – desirable: this, too, we have used in the past but are not using right now.
10. Standard or Widely Used – desirable: in the Java world, only Log4J (the most widely used library) and java.util logging (which is in the standard library).
11. Unique Messages – desirable: as suggested in the comments, an ability to identify each log usage uniquely (probably by source file and line number) would be handy.
12. Sensible Fallbacks – desirable: it’s nice that the library works OK when your config fails, because it helps in debugging the config problem.
13. Logging Queued to Avoid Delays – nice extra: although this seems like it would be a very useful feature, I do not know of any serious production logging tools that implement it.
14. Threadlocal Context Data – nice extra: theoretically, this is extremely useful. In practice, people usually live without it and find the data by reading back through the log.
15. Log Filtering – nice extra: if this were easy, we would use it to filter out SSNs and passwords from our logs, but we can live without it.
16. Internationalization – undesirable: I recommend against ever using this. Use logging ONLY for developers, NOT for end users; all developers should speak the same language.

Next, I will assemble a list of different logging libraries in Java to be evaluated.

• Log4J: Log4J by Apache is the most widely used logging framework in Java.
• Java util logging: Rather than adopting Log4J as the standard for logging in Java, Sun chose to clone it, creating something almost-but-not-quite the same as a standard Java library.
• Commons Logging/Log4J: Commons Logging is a wrapper from Apache which is designed for use in libraries. It simply delegates to an underlying logging framework. The purpose is so a library can be configured to use the same logging system as the rest of the application. I will consider Commons Logging backed by Log4J.
• SLF4J/Log4J: SLF4J is a project begun by the original author of Log4J. It is intended to provide a better API for calling into a logging framework, and it can connect to different logging back ends. I will consider SLF4J backed by Log4J.

There are others (logback, jLo, and many others), but I am fairly confident that one of these 4 will be the final choice, so at this point I am going to perform a full analysis just on these 4.

So, after considering all of these options, I have concluded that my preference is to use SLF4J as an interface, with the implementation from Log4J. Most of the options I have considered have more or less the same features. The deciding factors are (1) Threadlocal storage (MDC) is useful and not present in java.util.logging, and (2) The API for SLF4J provides an elegant solution to delay string construction, which is rather important for performance.

Conveniently, SLF4J publishes a tool for automatically converting existing java.util.logging and Log4J code to SLF4J, so the conversion should be relatively painless.