June 9, 2009
For some years I’ve been interested in lean software development and how to reduce waste. While introducing lean practices, I’ve needed a small, simple Kanban Board application. Thought I’d write one.
You can download a very first alpha here or see it in action here.
Suprised? The application is just one HTML file, no installation needed, no Java, Ruby or PHP. One HTML file, no other dependencies.
How to use the application?
Edit the source of the HTML file with an editor of your choice, preferably one which knows HTML. You will find a list of stories. The example contains those:
T_Q,S18,Checkout optimize
DE,S2,Build old shop
DE,S4,Rebuild with SOAP
DE_Q,S10,Rebuild with REST
P,S17,Do something with OpenID
D,S3,Make application faster
D_Q,S7,Credit Card Payment
DE,S13,Build something astonishing
P_Q,S17,Fix YSlow
R,S39,Google Page Speed fix
There are three columns for stories. The first column contains the state the story is in, the second contains an identifier for your story and the last column the name of the story. You can edit the columns, change states, change names, remove and add stories. You can also export form Excel to csv, then cut&paste into the application source.
The available states are described next in the file:
D,Design
DE,Development
T,Test
R,Release
They need to be available for the stories and match the states of the stories. You can add, remove or change states. Every state has a “sub-state” as a ready queue. For example the ready queue state for Design “D” is “D_Q”, for Test “T” it is “T_Q”. You do not need to describe the ready states, they are automatically created and a ready queue is shown in front of every state. For example “Test Ready” is shown left to “Test”, if there are stories in that particular state.
Customize the colors
The colors of each state are defined in a CSS block.
.box_P { background-color: #FFFF00 ; color: #000000;}
.box_P_Q { background-color: #F0F0F0; color: #606060;}
Feel free to change them to your taste. “.box_P” is for the “P” state box, “.box_P_Q” for the corresponding ready queue.
The main use case for this application is to inform a company about the stories which are in development and in which state they are. It’s an ideal information radiator. I plan to use it on a huge screen.
Future
I’m thinking about a CouchDB storage implementation for storing data and application logic. Or storing data in the file, with drag and drop, inlining Jquery, editing and storing like TiddlyWiki does.
Future features? Add WIP limits, add “From here” signs to display cycle time until live.
Have fun with this One-File-HTML-application. Tip: you can easily mail it around, no install needed.
April 22, 2009
I’ve been toying with Scala for some months now, one thing I’ve struggled with coming from Java are constructors in Scala. They are comparable to Java, but the syntax is different.
To help get you going faster in Scala, the top 5 things to know about constructors. Here we go:
- How to do constructors with a parameter
public class Foo() {
public Bar bar;
public Foo(Bar bar) {
this.bar = bar;
}
}
Looks in Scala like this:
class Foo(val bar:Bar)
In this case val creates an immutable final public field, using var would create a mutable public field.
- How to have private fields
public class Foo() {
private final Bar bar;
public Foo(Bar bar) {
this.bar = bar;
}
}
Looks in Scala like this:
class Foo(private val bar: Bar)
Update: Changed due to comments. Thanks for the commentors to point this out
Private fields are not as necessary as in Java, you can have public fields for attributes and change them to a method (def) later - without changing your clients.
- How to use super() ?
public class Foo() extends SuperFoo {
public Foo(Bar bar) {
super(bar);
}
}
Looks in Scala like this:
class Foo(bar:Bar) extends SuperFoo(bar)
- How to have more than one constructor?
public class Foo {
public Bar bar;
public Foo() {
this(new Bar());
}
public Foo(Bar bar) {
this. bar = bar;
}
}
Looks in Scala like this:
class Foo(val bar:Bar) {
def this() = this(new Bar)
}
Secondary constructors like this() need to delegate to another constructor to work (Thanks @Synesso).
- How to get bean style setters and getters?
public class Foo() {
private Bar bar;
public Foo(Bar bar) {
this.bar = bar;
}
public Bar getBar() {
return bar;
}
public void setBar(Bar bar) {
this.bar = bar;
}
}
Looks in Scala like this:
class Foo(@BeanProperty var bar:Bar)
The attribute bar will still be public, which is not a big issue (see above) in Scala. But @BeanProperty helps when working with Java libraries and you need Bean conventions for the libraries to work.
To add getBar and setBar but not a public field you need to:
class Foo(aBar:Bar) {
@BeanProperty
private var bar = aBar
}
Update: Changed to var, thanks to @eivindw.
Hope this helps you, if you have something to add, leave a comment. Should you struggle with the limited this() syntax (only one expression), then perhaps your constructors are doing too much. Consider the factory or better builder pattern instead.
Additional tip: use @Serializable to make your Scala classes serializable.
Nice coding in #Scala.
January 28, 2009
Title: The Definitive Guide to Terracotta: Cluster the JVM for Spring, Hibernate and POJO Scalability
Author: Ari Zilka (Terracotta CTO) and his team
Pages: 368
What the book is about
The books is an introduction about Terracotta which helps you distribute -transparently- the Java Virtual Machine memory over several JVMs. The main part of “The Definitive Guide to Terracotta” focuses on use cases. Those are quite good motivated, explained and described with many examples and working code.
What I’ve learned from the book
- What Terracotta and virtual heaps are
- How to use TC with ehCache, Hibernate and for session clustering
- Dropping in ready-to use functionality with TC integration modules (TIM) is easy
What I didn’t like
- Chapter about optimizations but not extensive enough and not enough information about deployments and deployment scenarios.
Should you buy this book?
Yes, highly recommended, it’s written by the Terracotta guys, you can’t get better and more accurate information.
Who should buy the book
- Every developer or architect who wants to use or evaluate Terracotta
Notes
Book kindly supplied by the publisher. This is a short version of my former review
I’ve chosen the micro review format because it lends itself to be used as a future micor format and I like short reviews myself. You can read the table of contents elsewhere, I don’t like it when reviews iterate the content.
What do you think about this short review style?
January 23, 2009
Many people object to ScrumMaster certifications:
- It’s a money making machine
- Scrum Masters do not learn anything during classes
- The certification is nothing worth - because nothing is certified
I have been a Certified ScrumMaster (CSM) and a Scrum practioner for some years. People who object to the certification do see it from the wrong angle. You need to understand Zen to understand the goodness in CSMs.
photo credit: darkpatator
Certification is a Zen joke, because the role of a ScrumMaster cannot be certified. It’s not about knowing some technical questions. What should a trainer certify in such a class? That you can lead an agile Scrum team as a ScrumMaster? No one can certify the fact that you’re a leader, catalyst and enabler. You either are or you aren’t. Zen masters (ha, another master without a master!) would laugh at the fun in the ScrumMaster certification. They laugh about the idea of certifying enlightenment.
Scrum without ScrumMasters
As another parallel, both in Scrum and in Zen, masters are only enablers. They are not needed after the act of enabling Zen/Scrum. My Scrum trainer told me, the goal of a ScrumMaster is to make himself obsolete. There is a Zen koan which goes like this:
If you meet the Buddha, kill him.
— Linji
If you see a ScrumMaster, kill him. Zen tells you:
If you are thinking about Buddha, this is thinking and delusion, not awakening. One must destroy preconceptions of the Buddha. Zen master Shunryu Suzuki wrote in Zen Mind, Beginner’s Mind during an introduction to Zazen, “Kill the Buddha if the Buddha exists somewhere else. Kill the Buddha, because you should resume your own Buddha nature.”
If you think the ScrumMaster is Scrum, you’re delusioning yourself. In Scrum the product owner and the scrum team can, and should from my view, act by themselves, without the need of a ScrumMaster. The ScrumMaster helps them achieve their Scrum. Helps them overcoming initial obstacles in their productivity.
Kick your ScrumMaster
If the ScrumMaster is not good enough for them, certification and coaching inside the company hasn’t helped, the Scrum team has always the right to kick their SM if he isn’t good enough for them. And they should do so. If in Zen a master isn’t good, pupils will just leave him. This might lead to problems within the organization, especially if the ScrumMaster is their boss, but that should be the problem of the organization, not a team problem.
Practitioner certification
There are many more certifications from the Scrum alliance. If you dig deeper, the real fun part is that CSM doesn’t mean anything, practitioner means much more:
The practitioner level of certification (CSP) is only offered to those CSMs who have hands-on experience using Scrum. Applicants must complete an extensive questionnaire with probing questions that focus on applicants’ real-world experience using Scrum on software development projects. Their application is reviewed for answers demonstrating competence and comprehension of principles that can only result from hands-on work. The applicant may be questioned to determine eligibility. To maintain CSP status, you must submit a new application every two years.
Is the certification any use?
Yes. The Certified Scrum Master training has several merits:
- Calling the Scrum training “Certified” guaranties the quality of the trainer
- It motivates the Scrum master to think in Scrum
- If managers take part, it helps the organisation adopt a “we can do it” view about Scrum
- Certification (CSM) seems to be one of the main reasons for Scrum success in the enterprise. The certification makes Scrum compatible for managment.
The view about acceptance is shared by Peter Stevens:
It is also about branding, and has been quite successful. The acceptance of the CSM program is high (especially from corporate customers, and this is where the money is). I believe the CSM program is an important reason why Scrum is better accepted than say, XP, in corporate management circles.
Scrum is successful. I’ve seen it help development departments gain productivity. If you do not scrum yet, go for it.
January 9, 2009
If you have or plan an application with synchronous communications over an external API, it will sooner or later break. Why do we need asynchronous communications? Matt Tucker is clear about that:
Take, for example, Twitter. High Scalability recently covered the load stats on Twitter reporting that they average 200-300 connections per second with spikes that climb to 800 connections per second. Their MySQL server handles 2,400 requests per second! Recently, the [2008] Macworld keynote became the most recent culprit for causing Twitter to cut off its API, which has 10x the load of their website.
When one of my web pet projects needed a messaging backbone which extends to the browser. Whenever a resource did change on the server, all users watching the resource should get a notification without need to reload their browser. Two candidates are Javascript for ActiveMQ, which uses Comet
ActiveMQ supports Ajax which is an Asychronous Javascript And Xml mechanism for real time web applications. This means you can create highly real time web applications taking full advantage of the publish/subscribe nature of ActiveMQ.
ActiveMQ is a messaging bus, often used as an Enterprise Service bus as mentioned in my recent concurrency rant. Components can send messages to the bus and subscribe to topics.
photo credit: mudpig
The other unsuspected contender is Javascript for Jabber. Jabber with the XMPP protocol is usally used for sending chat messages. Comparing these two and my thoughts:
ActiveMQ
- Standard solution, JMS based
- Routing solutions like Camel available
- Easy access for different languages via Stomp
- Attach Jabber as a service
- Notification easily over topics
Jabber
- Free OpenFire server
- Messaging with only one user with UUID for resource which did change
- Messaging with many users, who join one chat room
- Chat rooms as topics
- Server side filtering? How to make it secure, that people only get their own messages?
In the end I decided to go with Jabber/XMPP. The main points for me have been:
- Server does scale to connections
- Chat client can be used for debugging
- Very easy to use with different programming languages
- Presence protocol to detect services
- Easy to implement additional chat solution
This worked quite well as a spike. I followed a similar mode as Adrian Sutton, who had good experiences with Jabber/XMPP too when spiking a cache solution:
We grabbed the Smack API and started playing with it and quickly discovered that sending and receiving messages was ridiculously easy. It turns out that the absolute simplest way you can minimize stale data in your caches is to simply have all the servers join a preconfigured chat room. Whenever they save a change to a resource they send a message to the room with the unique ID of that resource and whenever they receive a message from the room they assume it’s a unique ID and remove any cached versions of that resource.
Though I had some major problems accessing Jabber consistently from Javascript. With more on messaging in the backend, I would have went with ActiveMQ as a message bus. And perhaps I might move to ActiveMQ in the backend and then I’m still free to attach Jabber on top of that and keep the frontend code. Best from two worlds.
Think innovative, use technologies in a way to help you. Jabber/XMPP is more than a chat protocol.
January 5, 2009
You’ve by now have read a lot about Erlang style concurrency. In Erlang actors are sending messages to inboxes of other actors and react to messages in their own inbox. The advantage of this approach with immutable messages is that you can’t get as easily in a deadlock as with basic Java concurrency with synchronized and threads. A simple ping pong example looks like this:
ping(N, Pong_PID) ->
Pong_PID ! {ping, self()},
receive
pong ->
io:format("Ping received pong~n", [])
end,
ping(N - 1, Pong_PID).
pong() ->
receive
finished ->
io:format("Pong finished~n", []);
{ping, Ping_PID} ->
io:format("Pong received ping~n", []),
Ping_PID ! pong,
pong()
end.
Two actors sending each other ping and pong messages with the ! symbol and acting on those with receive. Although there is more to concurrency than Erlang style message passing, as I’ve written lately in a post, for some problems it’s best and easiest to use message passing.
Is Erlang the next Java, but you are stuck with Java and can’t use Erlang style concurrency? Wrong.
There are concurrency solutions for Java which mimic Erlang style concurrency:
Update: “One clarification: ActorFoundry just uses Kilim’s weaver and therefore is not built on top of Kilim.” (see comments, thanks)
Update 2: Another option sugested by Diego Martinelli is to use Functional Java
Sujit Pal has written a comprehensive post comparing the performance of those frameworks with lots of example code:
Over the past few weeks, I’ve been looking at various (Java and Scala based) Actor frameworks. In an attempt to understand the API of these frameworks, I’ve been porting the same toy example consisting of three pipelined Actors responding to a bunch of requests shoved down one end of the pipe. To get a feel for how they compare, performance-wise, to one another, I’ve also been computing wallclock times for various request batch sizes.
And as Kilim has been shown to be as fast as Erlang - I’ve written about the fact here - these Java solutions do indeed look comparable to Erlang in the concurrency space.
What does this mean to your Java Enterprise?
Don’t worry about the multi-core future, Java has plenty to give for your multi-core platforms. And if you’re only stuck to the Java VM but not the Java language, you could go for Scala and gain some functional programming capabilities on top.
Thanks for listening. Hope you’ve learned something. As ever, please do share your thoughts and additional tips in the comments below, or on your own blog (I have trackbacks enabled).
December 31, 2008
The pound is rapidly going down and will go down even more next year (some say to 0.90 EUR). Comparing a Nikon D700 on Amazon.co.uk vs. Amazon.de results in
1571 EUR vs 2054 EUR
That’s a 25% difference for electronics (p&p etc not taken into account).
December 22, 2008
People talk a lot about concurrency. With the rise of multi-core processors, concurrency becomes more important. It’s sad that developers don’t know much about concurrency - and most of them just parrot what they have read in other blogs. I wanted to write this post for quite some time to shed more light into concurrency.
There are mainly two different types of concurrency
- One Task, many workers: For example parallel Fibonacci Numbers
- Many Tasks, many workers: For example web requests
They have two different characteristics:
- First: Need to share data
- Second: No need to share data (or “not to share in real time”)
Most people think of the first kind, when they discuss concurrency. Although most applications are of the second kind. I wish people would not confuse those two and try to fix the second problem with their solutions, because the second problem is solved sufficiently (think FaceBook). So this post will only discuss the first type of concurrency.
The breakthrough for concurrency of the first kind came with threads and the synchronizd keyword in Java 15 years ago. Before that concurrency was an esoteric topic for niches, with Java it became a topic for every developer. Today most people recognize that threads and synchronized are too low level though and create quite some problems if you don’t know what you do. One half of the blogosphere damns Java for this and favors Erlang style concurrency: Message passing between objects, each object has an inbox (a queue) of messages which it works through and objects send messages to the inbox of other objects, where messages are immutable. The benefits are clear: No shared state, no need to regulate the access to shared state and the freedom to implement the scheduler of the objects the way it works best (not being limited to thread libraries).
This half proposing Erlang over everything usually doesn’t know what the other half does and think they still use synchronized. But this is only the most basic technique (and not even the best basic one with the advent of atomics). Surprise, enterprise Java developers don’t use threads (directly) and synchronized. I’ll tell you what they do.
There are lots of better methods for concurrency nowadays like Futures, Executor Services and especially Doug Leas Fork/Join framework in Java JSR 166y. The FJ framework splits a task into subtasks, distributes them, solves them and joins the result (in a very clever way with queues which are written on one side and read on the other and tasks which can steal work from others). The algorithm for forking and joining looks like this:
Result solve(Problem problem) {
if (problem is small)
directly solve problem
else {
split problem into independent parts
fork new subtasks to solve each part
join all subtasks
compose result from subresults
}
}
As an example to calculate Fibonacci numbers:
class Fib extends FJTask {
static final int threshold = 13;
volatile int number; // arg/result
Fib(int n) { number = n; }
int getAnswer() {
if (!isDone())
throw new IllegalStateException();
return number;
}
public void run() {
int n = number;
if (n <= threshold) // granularity ctl
number = seqFib(n);
else {
Fib f1 = new Fib(n − 1);
Fib f2 = new Fib(n − 2);
coInvoke(f1, f2);
number = f1.number + f2.number;
}
}
}
(you can use threads or a different scheduler for this to work)
A similar approach to concurrent work is MapReduce as implemented by Hadoop. It also splits work into sub tasks, does a mapping step for transforming data and then reduces the result. It works best for data crunching and reducing input data to output data.
Other techniques developers often use instead of concurrency primitives are communication abstractions, like communicating over concurrent access queues, for example java.util.concurrent.ArrayBlockingQueue (ah queues again! or call them inboxes).
Threads talk to each other by adding (hopefully immutable) messages to a queue. Sounds familiar? Those can even be distributed (also see Hazelcast for distributed queues). This is very similar to Erlang concurrency, just imagine input queues for all your workers, aka actors.
Scala has an Erlang like message passing actor concurrency implementation. When looking into the Scala code, Scala uses the Fork/Join Framework of Java, the cirlce closes. And Scala uses different schedulers for it’s implementation with one thread per actor only being one option.
We can abstract concurrency even more. Jonas writes an excellent piece about abstractions on top of actors in a piece about fault tolerant, Asynchronous concurrency but misses one crucial point:
Actors can simplify concurrent programming and reasoning immensely and I believe that Scala Actors is a key piece in the future Java concurrency puzzle. However, programming with actors and with explicit message passing and message dispatch loops can feel a bit unnatural and unnecessary verbose for Java developers that are used to regular OO method invocations and synchronous control flow.
As pointed out before, people use queues and are used to work with asynchronous flows. Java enterprise developers in particular as they use Enterprise Service Buses (ESBs). There are many implementations like ServiceMix, OpenESB, Camel, OpenMQ, ActiveMQ, Mule and others. They range from pure message buses to routing and integration solutions. Because ESBs are fault-tolerant, asynchronous concurrency, the developers who use them know about asynchronous flows. Companies like LinkedIn uses ESBs to distribute tasks in a fault tolerant and parallel way. Nothing new there. Java developers think in asynchronous messaging already.
Stand back a thousand feet and ESBs look like the Erlang model. Worker cling to the bus and wait for work. Each is listening to different messages, the waiting messages for each worker are a virtual input queue similar to Erlang. There isn’t as much difference between concurrency in different languages as people want you to believe there is!
As a final word: It’s interesting that the first and second kind of concurrency start to merge. With applications like Twitter or identi.ca, the second type is sometimes becoming the first type of concurrency because different requests need to share data with each other (hopefully you don’t use the database for this as Twitter did in the beginning). One can argue that more and more applications need to share data between sessions. You can use an actor model for this (Lift does this). You could use ESBs. Or you could go a very different way with distributed method calls and objects - Terracotta to the rescue.
Thanks for listening. Hope you’ve learned something. I did by writing this post. As ever, please do share your thoughts and additional tips in the comments below, or on your own blog (I have trackbacks enabled).
Update: Actors Guild for Java looks also interesting
Update 2: If you find this post offending, go back read it again without you favorite programming language in mind which you need to defend. There is nothing to defend. You’ve chosen the programming language to the best of your knowledge, as have others (and if they haven’t but just chose the language because of some hype or others told them to, well, not a good idea). Spread your knowledge, don’t feel offended. Be happy if others find solutions to their problems. This is not some kind of football game which you win if you gain enough points against a different language. If the other one is better, use it. If not then don’t. Sounds awfully common sense, but we sometimes seem to forget this. Merry xmas.
December 18, 2008
Thanks to all of you for following. I do my best to write interesting - somtimes provoking - blog posts to keep you interested and amazed how wonderful software development is :-)
The Feedburner stats which describe your rising following:
(as people asked: The jump was on October 1st, from 481 to 674, there was no jump in the Google analytics visits for that time frame, the second jump was on October the 27, I’ve looked into it and the jump came from the time where the website was slashdotted by my 6 reasons why my startup failed story on Reddit)
December 10, 2008
The best take-away pizza in Berlin you can get is the one at Dirty Harrys, Schivelbeinerstr (right across the street where I live). 23cm, fresh, very good taste, full of toppings, very friendly, 2 EUR.
Go there if you’re visiting Prenzlauer Berg.
Only bad thing about Dirty Harrys: Closed on Sundays :-(
December 4, 2008
I’m always on the search on good interview questions as I’m doing a lot of IT recruiting. Sometimes in interviews I talk about books the candidate has read recently, to gain some insight into his interrest and enthusiasm - Amazon looks into enthusiasm as an indicator for good candidates.
Another approach to gain some insights into candidates is to ask them about the 5 best software development books, which he considers must-read-books, and why those made it into their Top 5 list. I’ve written about books I consider good,
My personal list would be something along the lines of:
- Refactoring
- Patterns of Enterprise Application Architecture
- Clean Code and other books by Robert C. Martin
- Domain Driven Design
- All McConnel Books (Rapid development, Code Complete, Software Estimation,…)
- The Pragmatic Programmer
- Implementing Lean Software Development
- The GoF book
- The Practical Guide to Defect Prevention
(Not on the list are psycholocial books like Influence, design books like About Face, compiler or programming language books, I concentrate this post on software development books in the stricter sense.)
Getting people talking about the books they like, or the recent books they’ve read and don’t like, gets most of the candidate emotional and involved. They show enthusiasm (or not). Don’t hire those who do not read books (my opinion) or don’t show enthusiasm for topics of books (positive or negative).
Bonus Point: And most of the time I learn something about books I haven’t known yet.
Thanks for listening. As ever, please do share your thoughts and additional tips in the comments below, or on your own blog (I have trackbacks enabled).
December 2, 2008
I have an Iphone with an edge data flat and a 300mb limited HSDPA plan. As a curious guy I want to know how much I have used of those 300mb HSDPA. Calling T-Mobile for 3 times results in three different opinions:
- Callcenter person 1: I should wait for the next month, then I will see the usage in my T-Mobile account (One month later I see nothing)
- Callcenter person 2: I can’t look online but I should call the call center to ask, they will tell me (and the employee told me my usage, 41mb of 300mb back then)
- Callcenter person 3: No one knows how much I’ve used. They can’t see it in their computer (I wonder how the second one could know) and it’s not possible for me to see my usage
Isn’t that funny? I have a limited data plan, but they can’t tell me and I can’t see my usage? They are able to monitor all my calls and monitor all my usage and tell (on demand ;-) the police, but they can’t tell me how much I’ve used? They can limit my flat rate to EDGE after 300mb, but they can’t tell me how much I’ve used already? And I can’t see?
I’ll call tomorrow for a fourth opinion ;-)
November 27, 2008
I wish Scala would have native support for types. Currently in Scala you need to write classOf[Person] for the Java Person.class expression.
It would be nice to have something easier than the noisy classOf e.g. when using Google Guice. Because using Guice with Scala looks ugly. I know I could do DI without Guice/Spring in Scala, but for varios reasons I don’t want to follow Jonas approach for now.
I think Groovy can just use Person for referencing types, but something like #Person would be ok too.
November 23, 2008
In January this year I’ve written about the future of functional programming in 2008, specifically about Scala and F#. Now that the end of the year nears, I’m going to look at my predictions and see how I fare.
In 2007 lots of people claimed that this year would be the year of functional programming languages. One quote I used in my post was from the website lambda the ultimate:
“My prediction is a little bit more conservative, but I predict Scala will gain momentum, and at least one high visibility project will use Scala”.
I was more than sceptical
A safe bet, really? [...] Don’t kid yourself. There will be no rise for application and especially enterprise development in functional programming.
mostly because I’ve seen the shift for languages taking a very long time. It took decades for people to move from C-style procedural code to object orientation. And though functional languages have been around forever, they haven’t started their ascend into the mainstream yet. But there was also a curious and hopeful tone in my post nevertheless:
They both are the most promising functional candidates when it gets to momentum. The transit for C# and Java developers seems to be easiest compared to other languages.
After nearly 12 months have passed, now let’s see what 2008 brought us concerning functional programming, especially Scala - as I specifically mentioned Scala in my post and because I know most about Scala from the bunch of functional languages.
A lot has happend to Scala in 2008 for sure. Even CIO.com mentions Scala in a Scripting Language comparison:
Scala is particularly attractive to Java developers.
(not sure why they call Scala a scripting language ;-)
Some real examples have shown up. Those include the real world Scala examples by Jonas Boner and Twitter. This matches with the “a high visibility project will use Scala” prediction. Some droped Scala in their development though. What hasn’t happen in 2008 is that greater numbers of developers moved from Java to Scala or bigger numbers of developers moved to Scala from other languages. As The Disco Blog writes in a comparison between Scala and Clojure:
What remains to be seen is if they can compellingly convince the Java market to learn their way of programming rather than leveraging what’s home (i.e. the Java language) for the majority of their respective target audiences.
Books
Often books are cited as one indicator for momentum in programming languages, which sometimes is problematic as I’ve written before. Scala has been quite successful 2008 with books. There are three, one released and two announced for 2009:
Growing pains - syntax debates
Guido van Rossum, the Python inventor, had some troubling words to say about Scala:
Perhaps my biggest disappointment in Scala is that they have so many rules to make it possible to write code that looks straightforward, while being anything but — these rules all seem to have innumerable exceptions, making it hard to know when writing simple code will work and when not.
Following one debate about Guidos post on the Scala debate list, there are many counter points raised. Sadly the bigger number of commentators want the language to stay the way it is. It seems only David R. MacIver wants to change the language to make it easier. I strongly agree with him and Guido that there are parts which are inconsistent, which need to be changed to get Scala into the mainstream.
What bothers me personally with Scala? It’s not that well integrated with Java as it could be. In my post on Clojure vs Scala I mentioned that already:
But the idea of Clojure - tight integration with Java through Iterable and Iterator, implementing Java interfaces, but keeping immutable structures, compared to Scala which creates it’s own incompatible versions, should prove much more successful. I like that definitely way better, Scala should adopt that approach.
And they should drop .NET support. It looks like nobody uses it and it hampers Java integration. Drop it.
But Guidos post has some hope in it too. Alex Paynes comment to Guidos post says:
I’m seeing more and more programmers getting real work with Scala, not musing about its theoretical possibilities. I find that pragmatism encouraging.
Competitors
The success of functional programming - in this case Scala - depends on competitors. How did they do in 2008?
I see two competitors to Scala. They mainly arise from the notion of Scala as a Java successor - which is the main market for Scala I think. Not much people will come from the dynamic language camp (some who might be fed up with large Rails code bases), not much from functional programming. Most are current or former Java developers.
The first competitor is Clojure. Why? It has a big momentum and a lot of buzz. Rich Hickey seems to be an approachable, knowledgable guy who can give good speeches. He does a lot of things right. But Clojure is mainly a Lisp style language. And lots of people don’t like S-Expressions because they find them unreadable in quantities. It’s also not focusing on OO style development, when most people today agree that OO is a good way to model business problems. Scala will win over Clojure I guess.
The second, and more important competitor, is Groovy. It’s been in this space for quite some time, but with the recent aquisition of G2One by SpringSource I guess it will make a big jump into the enterprise. SpringSource has a lot of good speakers at conferences who are also quite good sales people. Groovy had some problems, but mostly adapted, made it’s syntax easier, added powerful features and got faster. The Groovy vs. Scala debate might be more about dynamic vs. static typing (see last comments by Paul and James) and might be decided by the type issue.
All in all the competitors seem to have done better in 2008, with Clojure getting momentum and mentioned pratically everywhere and Groovy with the G2One takeover.
Killer applications
There is still no killer application in 2008 for Scala. Ruby took a decade to get momentum. There was some momentum when I developed in Ruby at the end of the 90s (mostly by Java people who were agile and fed up with Java), but this momentum got Ruby nowhere. Rails as a killer application for Ruby let the number of Ruby developers explode. In a way that today Rails and Ruby are synonymous, very few people use Ruby without Rails.
What could a killer application for Scala look like? A Rails clone? David Pollaks Lift didn’t make it. Perhaps it’s actors or a application server build on actors (comet friendly, scalable) is the killer application for Scala. Other ideas? Mainly the lack of such an application explains the slow growth of Scala in 2008.
Conclusion
And in the end? I’m more optimistic about functional programming than at the beginning of 2008, although the TIOBE index for 2008 doesn’t look that promising for functional languages.
Object-Oriented Languages 57.9% +1.6%
Functional Languages 2.6% +0.4%
I mostly think now OO and Functional Programming will work together in the future to write good software, and OO languages will borrow more things from functional languages. I’ll write a deeper post about how OO and Functional Programming fit together for sure.
This was a post mostly about the future and state of Scala. Wasn’t it supposed to be about functional programming? Following blogs and debates it seemed to me that other functional languages didn’t move in 2008. Erlang had some hype during the summer, which ebbed too soon. Haskell didn’t move, it’s still the promising functional language it had been for years. F# did move, but beside the promises by Microsoft I didn’t see any breaktthroughs in 2008 (perhaps I haven’t looked in the right places).
Thanks for listening. As ever, please do share your thoughts and additional tips in the comments below, or on your own blog (I have trackbacks enabled).
Update: See the impressive comment by Don Stewart about the growth in Haskell this year. Looks like real momentum, contrary to my (obviously wrong) perception ” Haskell didn’t move, it’s still the promising functional language it had been for years.”
Update 2: Good points made in comments: IDE support got much better 2008 in Eclipse, Netbeans, Intellij. Bad points: High profile googlers seem not to like Scala.
November 10, 2008
My hoster needed a second downtime, to remove the old harddisk (160gb), not sure why he needs that and why it couldn’t stay in the server. Sorry again. And there will be a 3rd downtime, because my hoster found a RAID controller in my server which shouldn’t be there. And he wants to remove the controller and get me a 3rd downtime. How unprofessional.
Update: The support was very supportive, the issues are resolved and no 3rd downtime needed.
November 9, 2008
Sorry for the downtime, the hardware upgrade and Debian to Ubuntu migration didn’t work as expected due to mistakes by my hoster. Thanks for coming back.
November 4, 2008
Currently attending the Java/Enterprise/Agile/Osgi/SOA W-Jax 2008 conference in Munich. Really nice up to now.
October 31, 2008
Good news:
“Google Guice Plugin 1.0 M1. Supports discovering new Google Guice (implicit) components, adding/removing implicit setter and field dependencies and reconfiguring @Singleton’s.”
October 27, 2008
Sorry for the inconvenience, got slashdotted by reddit. Never thought so many were
interested. And no, scaling was not one of the reasons the startup failed ;-) No ssh from my work so it took some time to fix it. Thanks for coming (back)
During the dot com boom I founded a software startup with some friends - with me as the CTO. We developed a software for knowledge management. It was a combination of blogs, wikis, a document management system, link managment, skill managment and more. We started in 1999 which was quite early for wikis and blogs (Moveable Type was announced 2001). The link management system was essentialy the same as Delicious later. Beside all those new ideas (for 1999 at least) there were three great features:
- Everything could be tagged. Skills, people, links, documents, blog posts, wikis, something which is today called folksonomies. Tags could refer to other tags to form onthologies. Tags could link to other documents, blog posts, persons.
- Everything could be rated from 1 to 5
- We had a clever fuzzy search based on tags and ratings. Searching for “people with oracle knowledge” would also reveal experts for SQL Server - for example to stuff your project if an Oracle guru wasn’t available
We’ve got quite some money as a seed investment from a VC and were quite happy and successfully developing our application. We did show it to many users and received very favorable feedback from big companies. Why did the startup fail and I’m no millionaire? There are a myriad of reasons, but as I wrote in “Rules for a successful business”, the rules for a sucessful business are easy:
- The customer is the most important thing in your business
- The best business plan is to sell people the things they want
- Your business is successful if your earnings are higher than your spendings
So the most important thing is to sell - a fact lots of startups forget. And we did too. After much thought it comes down to these six reasons why we failed (beside the obvious one that the VC market imploded when we needed money and noone was able to get any funding):
- We didn’t sell anything
- We didn’t sell anything
- We didn’t sell anything
- The market window was not yet open
- We focused too much on technology
- We had the wrong business model
In more detail:
We didn’t sell anything, Part 1
We didn’t sell anything because we didn’t have a product to sell. As good engineers we wanted to wait until the product is finished and then start selling. Midway we started selling nevertheless with a nearly finished 1.0. This led to too much focus on development and not enough on sales. Without a finished product we thought we couldn’t go to customers and try to sell it. We’ve slowly learned two things:
- You don’t need a product to start selling if it’s software. The first sales meetings with managment were perfectly possible with screenshots, mockups and slides. As the topic was new to our customers, we first needed to convince them on the concepts (wikis, blogs, tagging) and that could be done without a finished product.
- Start selling before you found the company. Start selling now! You do not need to have a company to sell new ideas to customers. Start selling now! When people really want to buy your product, start the company.
We didn’t sell anything, Part 2
We didn’t sell anything because we had no sales person. Bummer. Of course we were looking for one and the business plan said: High priority is finding a sales person for the founding team. This took time and resources and we didn’t have that. If you want to sell something, get a sales founder or hire someone from the start.
We didn’t sell anything, Part 3
We didn’t sell something because customers wouldn’t buy. The product was great, the customers favorable, but they took too long to decide. We wanted to sell a knowledge managment tool bottom up to project managers and through them to companies. But everytime a superior heard of knowledge managment he decided it should be on his agenda. So the topic of knowledge managment moved up the chain of command and we had no real decision maker. We talked to irrelevant people (because the topic became strategic fast for our customers) and lost lots of time. Ask people if they are allowed to buy your product. Get to the one who says “Yes” fast. We had several big companies in our sales queue and I’m sure they would have bought in the end, but as a startup we couldn’t wait. SAP for example compared to us could have waited and sold the product after 12 months. Selling enterprise software takes a lot of time.
The market window was not yet open
The market window was not yet open. Noone heard of blogs, wikis and tagging. We had to educate our customers on the benefits of wikis (Everyone can edit! Everyone! How dare they!) and blogs (Everyone can have an opinion and write about it! Everyone!) and tagging (They can build ontologies! We need a comittee to define an ontology for everyone! Otherwise chaos will eat us!) Several years later it would have been much easier to sell a blog, wiki and tagging plattform.
We focused too much on technology
All the founders were interested in technology. We’ve worked with EJBs (not mature back then), we made everything spit out XML and rendered that to HTML with XSLT (not fast enough), wrote our own OR-Mapper - what a stupid idea (Hibernate not available), tried CSS driven websites (not enough knowledge available back then). This lead to rewrites and took lots of our time. Discussions about technology - we should have talked about customers - took time too and led to frustration.
We had the wrong business model
Plain and simple: we had the wrong business model. Selling software can eventually reap lots of money, but it takes time. We had upfront costs, making sales deals took a lot of time and we burned money without income.
The better model would have been: Do consulting on knowledge management and start with an open source product.
We did consulting to companies on how to do knowledge managment, use wikis etc. But we didn’t take any money for it, because it was part of our sales process. Focusing on cosulting and billing people would have created a steady income.
I did get into open source later with SnipSnap. SnipSnap took (a small part of) the ideas from the startup (wiki and blog) and was relased as an open source tool. Lots of people downloaded the software and installed it on their desktops. We really made installing SnipSnap easy, so it spread fast. I’ve talked to a boss of a very big software and consulting shop and he told me, wikis would never work for them: too chaotic and not structured enough. Well - in fact I knew that there were several SnipSnap installations in his company :-) As others are practicing now we could have gotten our foot in the door with an open source project, then sell support and enterprise features on top. Companies later paid us money to put features into SnipSnap, make it scale better and for other enterprise thingies. But in 1999 we didn’t know as much about software business models as we know now.
What can you learn from my mistakes? Not sure, but start selling. I’ve learned a lot though about software managment, products, business models, money and being a CTO.
Thanks for listening.
October 22, 2008
In my last post about “50k lines of code considered large?” I’ve wondered about large code bases and the different perceptions on what a large code base is. I came to the topic because of a blog post: “The Maintenance myth” by Ola Bini. One minor point he makes about maintanence is lines of code in dynamic languages. I know maintanence is mainly about technical debt. But I’m interested in how lines of code factor into maintanence problems. Ola says
“(very large code bases is 50K-100K lines of code in these languages)”
pointing to Ruby and Python. In a reply to a comment from me Ola writes “I would consider 50k-100k in Ruby to be very large, yes, definitely. I know of Python code bases between 100k and 200k, but that’s about the largest I’ve heard of.” With my Java background - and some Ruby and Python background mainly from the 90s - I consider very large applications to be much bigger, perhaps 500k to 1M for very large - not 50k lines as for example SnipSnap has ;-) The Linux kernel contains between 6.4M and 10M lines of code depending on the way you count. There seems to be a huge difference in what people consider very large. There could be several reasons:
- Python and Ruby are very difficult so smaller code bases are considered very large
- Python and Ruby are more dense so the same amount of logic can be expressed in lesser lines of code
Considering the second hypothesis the factor should be between 10x (50k compared to 500k) and 20x (50k to 1M) for things people consider very large - taking Ola and his coworkes and me (I didn’t ask my team ;-) as a very small sample set.
Therefor I’ve expressed an example in code. The example is an application fragment for managing songs - the idea coming from the common Ruby introduction. I’ve chosen to compare Python and Java because Python is considered by some people a more mature language and used by larger projects than Ruby and because I did more and bigger projects in Python (my Ruby experience is only some years of coding web applications in Rails and writing an OR mapper and web component framework in Ruby by myself: “Convention over Configuration Framework in Ruby from 2002″). Someone could do a Ruby comparison :-)
People may be surprised, but Java development and style - at least avantgarde - has changed over the last 13 years. So the example might not look like you think that Java should look. It reflects the style I would right now write green field Java code. It is inspired by Domain Driven Design and functional principles (for more about DDD and composite oriented programming in Java see Qi4J and real world Qi4j). In true DDD style I would prefer more objects like Name and Duration - see “Never, never, never use String in Java (or at least less often :-)” - but I’ve cut the example for brevity. Some people would not use a SongList domain object but a List directly. From my point of view, if SongList is a Domain Object and not an implementation detail, you should create a class and not use a List. So I’ve used a list.
A note on formatting: Lately parts of the wave front surfing Java developers switched to one line formatting of small methods, something which helps code readability and understanding a lot (you’ll see). IntelliJ IDEA does support this as a formatting option. It’s a very good feature in IDEA but to my shame I only detected it very late, but glad I did as it’s so much better this way.
For manipulating, filtering and transforming lists I currently use Google collections. For an introduction see here. Google collections make working with lists much easier.
The REST part of the application is missing in Python as I have not enough knowledge to write the code with a state of the art REST framework. Perhaps someone could fill me in. In the Java example I’ve chosen to create the JSON and XML on my own without an automatic mapper. Automatic mappers do exist though, one could use JAXB. The code for the builder is explained in a previous post.
One cautionary note: The only Python books I have are “Internet Programming With Python” and “Programming Python”, the first edition, both from 1996. Sorry that my Python is rusty, all correcting comments or comments on how to do it better are welcome. Please focus on better, not on shorter.
On to the examples:
Java
public class Song extends Entity<SongId, Song> {
public Property<String> name;
public Property<Integer> duration;
public One<Artist> artist;
public Song(String name, int duration, Artist artist ) {
this.name = read(name);
this.duration = read(duration);
this.artist = read(artist);
}
}
public class Artist extends Entity<ArtistId, Artist> {
public Property<String> name;
public Artist(String name) {
this.name = read(name);
}
public String toString() { return name.get() }
public Artist artist(String name) { return new Artist(name); }
}
public class SongList implements Iterable<Song≶{
private List<Song> songs = newArrayList()
public SongList addSong(Song song) { songs.add(song); return this}
public Iterator<Song> iterator() { return songs(); }
public Iterator<Song> filter(Predicate<Song> p) { return filter(iterator(), p); }
}
Some example usage:
// Not counted, as usually the pattern is
// SongList list = ... from Database ... or
// SongList list = ... from UI ...
SongList list = new SongList()
.add(new Song("S1", 5, artist("A1"))
.add(new Song("S2", 8, artist("A2"))
.add(new Song("S3", 13, artist("A3"))
// Print all songs
for (Song song: list) {
System.out.println( song.name() + " by " + song.artist() + "(" + song.duration() ")" );
}
// Print all songs with duration smaller than 10 (minutes)
final Predicate<Song> durationLowerThan10 = new Predicate<Song>() {
public boolean apply(Song song) { return song.duration.get() < 10; }
}
for (Song song: list.filter(durationLowerThan10) {
System.out.println( song.name() + " by " + song.artist() + "(" + song.duration() + ")" );
}
and a simple REST service which returns a JSON or XML (depending on the request) representation of the song list.
// Example REST Service
// Returning JSON and XML to a REST call, without automatic mappers like JAXB
public class SongListResource {
@Inject ListService service;
@GET @Path("/songs/{listId}")
@Produces("text/xml", "application/json")
public Node getList(@PathParam("listId") String listId) {
SongList list = service.listForId( listId );
return $("songs", new List>Song<(list) {
protected Node item(Song song) { return $("name", song.name() );}
});
}
}
The example in Python
class Song:
def __init__(self, name, duration, artist):
self.name = name
self.duration = duration
self.artist = artist
class Artist:
def __init__(self, name):
self.name = name
def __str__(self):
return self.name
class SongList
def __init__(self):
self.songs = []
def add(self, song):
self.append(song)
some example usage
# Not counted, as usually the pattern is
# songList = ... from Database ... or
# songList = ... from UI ...
# Not using SongList, the examples should be the same though
# or not?
songList = [ Song("S1", 5, Artist("A1")),
Song("S2", 8, Artist("A2")),
Song("S3", 13, Artist("A3")) ]
for song in songList:
print "%s by %s (%d)" % (song.name, song.artist, song.duration)
# could provide print method to list
for song in (song for song in songList if song.duration < 10):
print "%s by %s (%d)" % (song.name, song.artist, song.duration)
A very preliminary conclusion
The example is very short and perhaps not very meaningful. One would need to do more empiric research (e.g. comparing FP to LOC in different languages). And perhaps some readers will provide addtional information. So the conclusion is preliminary and will be updated. Counting the lines of code there are 33 NCSS in Java and 19 NCSS in Python. Java has around 1.7 times the LOC of Python from my example. Taking the hypothesis above this could mean several things:
- I’ve written sub-par code and most applications differ significantly in style and are much shorter in Python
- Code complexity and lines of code arise from frameworks not the language
- Java is really only 1.7x more verbose than Python, not 10x to 20x
I can’t comment on the first conclusion. The second conclusion means, someone would need to compare two framework examples, say the song list in Seam and Django. The third conclusion is very interesting. It would mean that people consider applications written in Python very large although they (relatively) contain a lot less lines of code. Ola considers 50k to 100k very large, with a factor of 2x this would make 100k to 200k of Java lines. I can’t speak for most Java enterprise/startup developers, but as I consider 500k to 1M very large, Ola and I differ by a factor of 5x of what very large is. I only can speculate what’s the reason for this.
- This is a personal thing, and different developers have hugely different views on “very large” (perhaps depending on what they have seen)
- Developers only write small applications in Python and consider everything else “very large”
- Python is not maintanable above 50k to 100k lines of code and because of that people consider this code bases very large
- Developers have trouble understanding and refactoring bigger code bases than 50k to 100k lines of code (perhaps because it’s a dynamically reference type language)
The first conclusion somehow fits with another quote from Olas post: “And it’s interesting, the number one question everyone from the static “camp” has, the one thing that worries them the most is maintenance.”. They may have seen “very large” applications contrary to the “dynamic camp”.
“Of course, this is totally anecdotal, and maybe these guys are above your average developer.”
I’m glad to provide a step (small one) from the anecdotal to the empiric and from the empiric of this post I don’t think people considering 100k of lines “very large” are “above your average developer”.
Another side note: “But in that case, shouldn’t we hear these rumblings from all those Java developers who switched to Ruby? I haven’t heard anyone say they wish they had static typing in Ruby.” Perhaps because they do green field (not brown field) development? And you need to develop for several years in one application to make it a brown field? And it takes several years to accumulate enough technical dept? Because most of them just started and don’t do “very large” applications?
Other interesting stuff:
- A paper (PDF) from 2000 about Scripting, C and Java comes to the conclusion: “Designing and writing the program in Perl, Python, Rexx, or Tcl takes no more than half as much time as writing it in C, C++, or Java and the resulting program is only half as long.” matching the 1.7x factor of my short example
- Dhananjay Nene wrote a performance post about Python and Java (and some other languages) and the LOC for Java is 86 and for Python 41, a factor of 2.1x
- Dave rewrote a Java programm to Python from 4700 lines of code to 700 (factor of 6.7x). This would fit more with Olas impression. Not sure how this fits in, the developer can’t show the source and it was a rewrite by a different developer. Also counting comments and empty lines, the styles between the developers could differ significantly.
- Daveh did a comparison, with Python having 214 LOC (not NCSS) and Java 282 LOC (not NCSS). A factor of 1.3x
Lots of open questions and I would be very interested in other opinions and other examples - and to explore the topic further.
Thanks for listening to this very long post.
Update: Ryan (see comments) supplied a version of a function in C and Python and after removing the hand memory allocation code and the Python interface code of the C version, the factor is 2.2x (38 to 17 NCSS). Thanks.
Update 2: Looking at Oloh (see comments) the factor of Java and Python is 4x. Very large base of examples. One would need to check the types of programs.
Update 3: An old article I’ve found again “7 reasons I switched back to PHP after 2 years on Rails”. An interesting info: After going to Rails and coming back, with the Rails knowledge the PHP app was reduced in size “- … and much more. In only 12,000 lines of code, including HTML templates. (Down from 90,000, before.)”. Looks like rewrites or prior experience in the domain reduces code size. Could explain Olas experience with Java developers who switched to Ruby. Came to this article again through a comment by Harry Pynn “Point number 7 is that programming languages are like girlfriends: The new on is better because you are better. Could it be that people moving to dynamic languages from static languages find it easier to write maintainable code having honed their skills with a static language?” on Frank Carvers blog.
