The only thing that really matters in software development.

Catchy title? I mean it. After 25 years of programming, the only thing that really matters in software development is if a company understands the project triangle. That from the three variables scope, time and resources you can only define two. There are two kinds of companies. Those that understand this and those that don’t. Everything they do flows from there. This is what really matters about software development.

Update: There are a lot of other factors, probably people being one of the most important (think “Peopleware”). Without understanding the Scope-Time-Resources triangle your development won’t work. It just can’t. You can screw up by chosing the wrong architecture, or methodology, or screw up during requirement engineering or with the wrong language for the problem.

But from my point of view this triangle is like gravity. If you ignore gravity, all else will fail.

How to do TDD with Flex?

Well, how do you?

JUnit recipes: Work around static attributes in classes

Beware of the class loader. I recently had a major testing problem with some code because I didn’t take the (sometimes ugly) Java class loading mechanism into account. There was a small class where I wanted to unit test the calculateAge method.

class AgeCalculator {
    public int calculateAge(Person person) {
      ....
    }
}

So I dropped EasyMock into my project and mocked the person class. I was using jMock in the past and could live with methods as strings. With the new JMock 2 syntax this has changed. I still struggle with {{ in JMock expectations and so does my IDE. When IDEA learns to format JMock expectations, I’m back. For now I’m using EasyMock though, sometimes with JMock assertions from the excellent Hamcrest assertion library. Back to the problem at hand . The mock was passed into the AgeCalculator to calculate the age of the person. The problem was the Person class.

class Person {
  private static Configration conf =
    new DatabaseConfiguration("Person");
    ...
}

Even though I didn’t use the Person class, just a mock, the static attributes where initialized and the DatabaseConfiguration tried to load data from the backend. This happens whenever your classloader sees the Person type, even in an interface.

interface Calculator {
    public int calculateAge(Person person);
}

Using Calculator in mocks will initialize the static attributes in Person. Which contradicted the isolation principle of unit tests and also led to long startups of the database backend. This stopped my efforts to test the AgeCalculator. Time passed.

Then I stumbled upon JMockit. Others have too. The friendly and supportive developers of JMockit have helped (thanks Rogerio) and with a simple line I was saved:

Mockit.redefineMethods(
  DatabaseConfiguration.class,
  MockDatabaseConfiguration.class
);

Constructs like the above with static attributes should never be created in the first place (hello all those people who propagate static inialized attributes for singletons to avoid synchronized). But if you have such code in your legay base, JMockit to the rescue.

Thanks for listening.

IconFinder pure genius

The IconFinder website is pure genius. Highly appreciated. With icon finder you can search with keywords through free (CC and LGPL) icon libraries. I often find myself searching through the clear or famfamfam icons which are in different directories or on different computers. I hope IconFinder will grow to other collections. I’ll stay tuned..

Grails more productive than Rails

Catchy title. But the guys from ALTERthought have some - much needed - numbers. Their numbers show Grails to be more productive than Rails, which is in turn more productive than Spring and pure JEE. As I’ve argued for years, there is no science in computer science. We need to put facts back into computer science. Hard numbers instead of faery tales. Hacknot thinks so too. Thanks for the real numbers. Two thoughts:

1. This is development effort, not maintenance. Glass shows that 40 to 80 % of effort go into maintenance.
2. We compared Grails to Seam and think they have much in common. Seam productivity should be nearer to Grails than to the standard JEE stack

Thanks for listening.

Black Swans II

Update to my black swan post. I found an excellent quote in a risk book about hedge funds, A demon of our own design (as always no partner link) which reflects my black swan post. Richard Bookstaber writes about risk management for financial instruments: “The types of risk that could be readily measured were better controlled, but those were not the risks that mattered. The real risk is the on you can’t see.” Exactly what I said about software development. Risk people can’t see or have trained themselves blind spots for.

Rails, Scrum, CMMI and Survivor Bias

The biggest problem with tales about software success is that they only show the survivors. This problem is independent of the domain, be it Rails, Scrum or CMMI success stories. Survivor bias is when you only look at the survivors of a process and then attribute the success of the survivors to some attributes like cleverness or strongness. Survivor bias is most easily shown with financial funds. Say you want to examine the success of some financial funds. You look at funds which are investing in high tech stocks. When looking at nine funds with the growth rates of 7.3, 9.2, 12.1, 5.4, 6.7, 7.2, 10.1, 8.5, 8.8 we get an average rate of 8.4 percent. This finding would suggest that funds which invest in high tech stocks have an average growth rate of 8.4. But assume there was one fund which defaulted. People looking at average growth usually only look backwards, which means they only select surviving fonds. When we go back in time and look toward the future, we see ten funds we want to measure. After some years, nine have survived and one has defaulted. So from a point back in time we selected ten funds, not nine. And with the tenth fund, with a growth rate of -100%, included in the average we get an average growth rate of -2.5 percent for the ten funds compared to 8.4 percent for the nine surving funds. That’s survivor bias.

The same can be said about other processes, most notably progessing of CEOs and companies, especially start ups. When looking at successfull startups people are mislead to attribute the success of a startup to some genius founder, good products or other attributes when in fact, an equally good explaination is that the startups had only luck (for that we need to get back in time and look into the future with all the startups, not look back from the point of successful startups). The same goes for CEOs: most could be successfull by luck, the unlucky ones just get kicked out faster and don’t show up as often as lucky ones.

Coming back to software development. We hear lots of success stories from companies and developers about tools, languages, frameworks, plattforms and processes. Those people praise Rails, Scrum or CMMI. But looking at funds and transfering our knowledge to those praise stories, we could assume that those involved were just lucky. Companies which failed with Rails, or Scrum, or CMMI and the countless other solutions for software development seldom talk about their experiences. Companies which fail with Scrum just drop it. Companies who try Rails in a pilot just don’t use it. Companies who are not able to get their CMMI certification just don’t start it. And some of the companies even didn’t survive to tell their tale. Most people publish success stories on their blogs, in magazines or on websites because people want to read success stories and success stories make the authors look good. Failures make them look bad. This leads to survivor and publish bias.

Next time you hear a success story, think about all the people who don’t or couldn’t tell tales about their failures. The success might just be plain luck and survivor bias.

Ruby: Complexity cannot be reduced

There is a lot of discussion about concise code. Comparing Java to Ruby, obviously Ruby has more concise code (This article contains code examples, which could be written better and which experienced Java and Ruby developers would write in differently. But the examples were chosen to illustrate points).

Compare this Ruby code taken from the Pragmatic book

songType = if song.mp3Type == MP3::Jazz
            if song.written < Date.new(1935, 1, 1)
              Song::TradJazz
            else
              Song::Jazz
            end
          else
            Song::Other
          end

with the corresponding Java code:

public SongType getType(Song song)  {
       if (song.isMp3Type(Mp3Jazz.class) {
               if (song.isWrittenBefore(new Date(1,1,1935)) {
                       return SongType.TRAD_JAZZ;
               } else {
                       return SongType.JAZZ
               }
       } else {
               return SongType.OTHER;
       }
}

Java has more code than Ruby in this example, but the cyclomatic complexity is the same for both examples. Although the size of the Ruby example could be reduced, in this version the LOC is also the same. You can’t reduce the CC with choosing another language, only by choosing another algorithm or language paradigm.

When adding comments to the code examples like this

/**
 * Gets the song type from a song. This does not
 * return the mp3 type information from the song
 * but tries to determine the song type from the mp3 type,
 * time period and other enviroment variables
 *
 * @param song the song which should be checked
 * @return the song type for the song
 */
public SongType getType(Song song);

or like this for a dynamic-reference-typed language:

/**
 * Gets the song type from a song. This does not
 * return the mp3 type information from the song
 * but tries to determine the song type from the mp3 type,
 * time period and other enviroment variables
 *
 * @param {Song} song the song which should be checked
 * @return {SongType} the song type for the song
 */
 def type(song) ...

The difference between the two language styles dimish and the CC [1] to understand the code for a developer which has never seen the method is the same. In his head he has to order and solve the nested if statements to come to an understanding of what the method does. He might parse the Ruby example faster, because it’s more concise, but the understanding part after the mental parsing is the same. Current studies show though that people don’t read and parse text word by word but detect meaning by looking at patterns of word beginnings and endings. This would suggest that less code doesn’t increase parsing speed a lot.

If conciseness of code is the deciding factor for faster understanding written code, then languages like K should be instant (some K programmers claim to instantly understand code by seeing it). A Hilbert matrix [2] in K [3] looks like this:

 {1%(!x)+\:1+!y}

The Java example from before can be written more understandable, for example with the Hamcrest library. This code is even less concise, but easier to understand

public SongType getSongType(Song song)  {
       if (is(song.getMp3Type, equalTo(Mp3Jazz.class)) {
               if (is(song.written(), before(1,1,1935)) {
                  return SongType.TRAD_JAZZ;
               } else {
                  return SongType.JAZZ
               }
       } else {
               return SongType.OTHER;
       }
}

It would be very interesting to see studies which compare code styles, conciseness and cyclomatic complexity and how they releate tounderstanding code which you haven’t written. Because as Robert Glass writes in the excellent facts and fallacies of software engineering: “Fact: Maintenance is the most important life cycle phase with 40-80% of the cost.” I have currently the feeling that most language innovation happens with rapid development, not low maintainance.

[1] http://www.fromthetrench.com/category/cyclomatic-complexity/
[2] http://en.wikipedia.org/wiki/Hilbert_matrix
[3] http://www.langreiter.com/space/k-notes

European Open Source Society?

I thought about open source and had lots of discussion in the past few years about open source in general, business models, different and superiour licenses, pragmatic open source and stuff. As I’ve been also developing open source software for the last twenty years, I’m interested in the state of the open source developers.

Some studies have shown, that half or two thirds of open source developers come from Europe (with South America and Asia growing strong). Compare that with the image that open source is a North American phenomenon[*]. In the mind of Europeans and especially European politicians open source has not the importance it should have based on the influence Europe has on open source through it’s developers. What about founding an European Open Source Society? For exchanging ideas, moving open source to the future and lobbying politicians to pay more attention to open source, the economic impact, the social impact and the chances there are for Europe to play an important role in open source. Sounds good to me.

What about founding one?

[*] Perhaps because more loud open source advocate live or move to the US and more companies in the US monetize open source projects

Cleaning up

I have to do some cleaning of my open source projects (cintoo, Gabriel, Radeox). For there are other rights holders and trademark owners, for some there are old repositories I have no access to. With me leaving my last employer some things have changed and now I have to clean up the mess ;-) Sorry for keeping you waiting, I’ll do my best to consolidate all stuff into Reposita when the rights have been cleared.

cintoo Messages for Java 1.4?

I’ve been thinking about 1.4 support for some time now. With TestNG as an example, it should be possible to both support the new 1.5 features like autoboxing and varargs and supply a 1.4 jar, any ideas and suggestions?

Moved cintoo Messages from jMock to EasyMock

Several reasons convinced me to move cintoo Messages from jMock to EasyMock 2. The main reason is that jMock has no static importable methods for mock creation and verification. The way to do this in jMock is to extend a jMock TestCase class. Unfortunately this class depends on JUnit, so when moving to TestNG I still had to keep the JUnit jar around. Also TestNG promotes a no-inheritance test structure where jMock just doesn’t fit in. The second reason is that I was bitten by method name refactorings several times. As some methods were equal in different classes, IDEA wrongfully replaced the methods strings in jMock which broke lots of tests. Another minor reason is that jMock doesn’t seem to move very fast.

The transition has been made easier by EasyMock as that framework adopted a lot of ideas from jMock in version 2.x (I think :-). I still hope to use the jMock spinoff assertion DSL though with assertThat( eq(5), …).

In the last years since the first jMock release I’ve been convincing clients to use jMock for their unit testing. I still think jMock is great for JUnit, but it just doesn’t work with TestNG in the current version.

cintoo Messages 1.0 released

Due to the help by Alex with TestNG I finalized the Messages 1.0 release.

Messages is a free framework to make internationalization easier for Java
applications. It adds several features compared to default Java internation-
alization and support locales for threads and different bundles for different
packages. This allows the usage of different bundles for different parts of
the application like plugins, the installer or logging.

Key features:
- a lot less code than standard internationalization in Java
- easier to use for developers
- supports bundle refactorings
- bundles associated with Java packages
- hierarchies of bundles,
e.g. one bundle for com.corp and one for com.corp.accounting
- Low intrusive code, e.g. $(this, “FILE_NOT_FOUND”, fileName)
- Locales per thread or global
- Apache license

http://cintoo.org

Downloads http://cintoo.org/dist/messages/
Documentation http://docs.cintoo.org/messages/

Successfully moved Messages to TestNG

Thanks to the help from Alex (from the TestNG project, great support) I managed to move Messages from JUnit 3 to TestNG. TestNG is more flexible and seems to move faster. JUnit 4 was released after I started moving projects to TestNG, so my default now is TestNG and I only use JUnit 4 on demand from customers. Also works great with pulse integration server, just point pulse to the xml file generated by TestNG and use the JUnit task in pulse.

Why, oh why, doesn't TestNG work for me?

I switched several projects from JUnit 3 to TestNG. After some minor problems with the fact that TestNG doesn’t seperate tests as JUnit does per default, everything works in IDEA and Maven. But whatever I tried, I can’t get TestNG to work with Ant. The TestNG Ant task just doesn’t find any tests. I used testjar, dir, includes and it also doesn’t help that the TestNG documentation is sparse and misleading. I moved Messages to TestNG too, but without Ant support this will not be released. Everything works fine, just Ant doesn’t. If I can’t get it working, I’ll give Junit 4 a try.

cintoo Messages 1.0 RC1

Just made the last version the RC1 version without changes. Everythings seems to be working, no complaints. If everything moves along as planned, this will become 1.0 final. After that I’ll have some ideas on 1.1. Thanks to all users for their feedback.

http://messages.cintoo.org

Method names in Java can be named _ and why static import is a good thing

When formatting messages with Messages.format code becomes quite noisy. So I thought why not use a special char for the method name? I used “_” which seems to be unintrusive.

then becomes

combining this with the new static import feature from Java 1.5 (great timing :-) this becomes

which looks shorter and less noisy than the first and as the default Sun solution for i18n.

I think about using “$” for the method name,

looks perhaps more familiar with Prototype.js using $ and this also looks like templating engines as Velocity/ JSP.

100% test coverage for Swing and database apps

A recent post about in-memory database unit testing by Mike Bosch reminded me of a presentation I wrote but never published about developer testing and how to achive 100% test coverage for nasty applications. High test coverage for most parts of an application is easy, for some parts it’s difficult to do. The hard parts are Swing parts (which Jemmy can help with) and database parts (which hsql in-memory can help with). With the tips and tools in the presentation it’s possible to reach 100% test coverage. But I conclude that most of the time 100% test coverage not needed.

cintoo Messages moved to Java 5

I moved cintoo Messages to JDK 1.5 because of several reasons. I know this will reduce the potential user base, but with Mustang on the horizon, a lot of projects are moving to 1.5 now or in the near feature. Beside I really like generics (being a strong opposer for some years, though IDEA could make code less noisy and hide generics completely) and Java 5 has varargs and outoboxing. This makes dealing with String formatting much easier.

Instead of:

(there was a helper in Messages which took int[] )

now you can write

which is mapped to

The Object… is syntactic sugar and is automatically mapped from “3, 7″ to Object[] with
autoboxing 3 and 7 to objects.

Ever more complex software stacks

In the last 30 years the software stack has grown from year to year. When I started coding in Z80 machine code, there was no software stack. Just the programmer and some hardware registers. No other software involved. Direct access to the machine. From there software stacks grew, to compilers and libraries. With Java the stack grew to JIT compilers, virtual machines, garbage collectors, byte code compilers and plattforms. To close the gap between the machine and the customers requirements one solution for many seems to be to grow the software stack even further. With MDA and containers the software stack grows to unprecented complexity. From PIM with transformations to PSM to a Java EE container, with AOP and compilers to Java, with byte code compilers to the VM with GCs. Then with JIT compilers to machine code (and from mc to microcode). Where will the stack move next?