Blog of Adam Warski

Frameworks vs Libraries as Inheritance vs Composition?

For quite some time inheritance was the dominant model of structuring programs in OO languages like Java. Very often it was used as a mechanism for reusing code – “common” functions where placed in an abstract class, so that subclasses can use them.

However, this often proves to be very limiting, as you can only inherit from a single class. The code becomes constrained and tied to one particular framework-specific class. Not to mention testing – such base classes often depend on outside state, making tests hard to setup.

That’s why nowadays time and again you can hear that you should prefer composition over inheritance (see for example this StackOverflow question). When using composition, you can leverage multiple re-usable chunks of code, and combine them in an arbitrary way. Also using IoC/Dependency Injection strongly favors composition.

I think the above inheritance-composition opposition strongly resembles the framework-library distinction. When using a framework, you are forced into a specific structure, where you must model your code in one specific way. Quite obviously it’s often hard or impossible to use two frameworks in one layer/module. That’s how hacks, ugly workarounds, reflection madness, etc. is born.

Libraries on the other hand (unless they are deliberately wrongly written), can be freely combined. Just like composition of classes, you can compose usage of many libraries in one module. Your code can be kept clean and only use the library functionality it really requires.

I’m not saying that frameworks are bad – just like inheritance, they may be very useful if used in the correct places. However, next time you put your code into a framework, maybe it’s better to think twice: can this functionality be implemented using composition, with the help of a library? Won’t this make my code cleaner and more maintainable?

Adam

ElasticMQ 0.4: message replication

ElasticMQ is a message queue system, with Java, Scala and an Amazon SQS-compatible REST interface.

The 0.4 release brings a new major feature: data replication. You can now setup a cluster of ElasticMQ nodes, and have each queue and message replicated across this cluster. Each node can use either the in-memory storage, or the DB-backed storage. This enables a wide range of deployment scenarios, e.g.:

• in-memory storage, single node: ideal for testing (ElasticMQ is easily embeddable and can be used to test applications which use Amazon SQS in production)
• db storage, multiple nodes, single shared database
• in-memory storage, multiple nodes, replication: good if at least one node is always alive
• local db storage, multiple nodes, replication: each node has its own persistance. Data safety without the need to setup a clustered database

Depending on how safe you want to be from loosing a message, there’s a variety of replication modes to choose from:

• fire-and-forget: data replication is done in the background
• wait for at least one cluster member to acknowledge that it received the message
• wait for a majority of members
• wait for all members

See the README for more details. Starting to use ElasticMQ is really simple!

Clustering is implemented using JGroups (thanks to Bela for the help on the forums!). The cluster can be configured using a standard JGroups configuration file, which means you can either use the UDP stack with multicast discovery, TCP stack with a list of initial node IPs, AWS discovery, and so on. ElasticMQ doesn’t require any particular JGroups configuration so you are free to use whatever suites your deployment best.

Please note that the replication feature is still at an experimental stage, so bugs (as always) may happen :).

Feedback welcome!

Adam

ElasticMQ 0.3: new API, new in-memory storage

ElasticMQ 0.3 just got released! ElasticMQ is a simple message queue system, which exposes both a native and an Amazon SQS-compatible interface. There are two major changes in this release.

Firstly, there’s a new all-Scala/Java in-memory message storage, significantly faster than the previous one, which simply used an in-memory H2 instance; currently the implementation is based on Java concurrent queues. The usage is still very simple, to create an ElasticMQ instance with in-memory storage just execute:

  val node = NodeBuilder.withInMemoryStorage().build()
  val server = SQSRestServerFactory.start(node.nativeClient,
    8888, "http://localhost:8888")

To show some numbers, here are a couple of runs of the MultiThreadPerformanceTest test on my MBP (2.4GHz Core2Duo, 8GB ram; all operations executed using the native API, without the http overhead):

    Storage: InMemory, number of threads: 5, number of messages: 50000
    Send took: 3 (3140), ops: 250000, ops per second: 83333
    Receive took: 5 (5057), ops: 250000, ops per second: 50000

    Storage: MySQL, number of threads: 5, number of messages: 2000
    Send took: 5 (5500), ops: 10000, ops per second: 2000
    Receive took: 74 (74269), ops: 10000, ops per second: 135

    Storage: H2, number of threads: 5, number of messages: 2000
    Send took: 0 (841), ops: 10000, ops per second: 10000
    Receive took: 30 (30388), ops: 10000, ops per second: 333

I think the difference is pretty clear.

Secondly, the native API has seen a major rewrite. It has a much nicer, object-oriented feel, for example (all classes are thread-safe):

  val node = NodeBuilder.withInMemoryStorage().build()
  val client = node.nativeClient
  val queue = client.createOrLookupQueue("queue1");
  queue.sendMessage("msg1")
  queue.sendMessage(MessageBuilder("msg2").withNextDelivery(tomorrow))
  queue.receiveMessage().map(message => {
    println(message.content)
    message.delete()
  })

See the Client, Queue, QueueOperations, Message and MessageOperations traits for details.

I also created a Google Group in case you have any questions, ideas or problems with ElasticMQ (feedback is very welcome!).

See the README for details on how to download and use ElasticMQ in your SBT/Maven project.

Adam

ElasticMQ 0.2 – support for delayed queues and messages

Time to start blogging in 2012 :)

I just released version 0.2 of ElasticMQ – a simple message queue system, which implements the Amazon SQS interface. Ideal if you want to test a service which uses SQS, or create a system which can leverage either the AWS infrastructure or run stand-alone.

The new version brings support for some new Amazon SQS features: delayed messages and queues. See the Amazon Blog for details.

Also, I changed the testsuite to use Amazon’s Java SDK. So you can be pretty sure that ElasticMQ works with SQS clients :). As an example, here’s the required code (in Scala, Java equivalent would be obviously a bit longer ;) ) to start an ElasticMQ node, send & receive a message using Amazon Java SDK and shutdown the server.

// First we need to create a Node
val node = NodeBuilder.withInMemoryStorage().build()
// Then we can expose the native client using the SQS REST interface
val server = SQSRestServerFactory.start(node.nativeClient, 8888,
      "http://localhost:8888")

// Now we need to create the sqs client
client = new AmazonSQSClient(new BasicAWSCredentials("x", "x"))
client.setEndpoint("http://localhost:8888")

// Using the client is quite straightforward
val queueUrl = client.createQueue(new CreateQueueRequest("queue1"))
      .getQueueUrl
client.sendMessage(new SendMessageRequest(queueUrl, "message1"))

client.shutdown()

// Finally we need to stop the server and the node
server.stop()
node.shutdown()

For more information about ElasticMQ, information on SBT and Maven dependencies, see the webpage.

Stay tuned,
Adam

ElasticMQ 0.1 released!

ElasticMQ is a simple messaging system, exposing an SQS-compatible REST interface. It can run using an in-memory H2 database (ideal for testing), or backed by a normal database (e.g. MySQL).

Using ElasticMQ can’t get much simpler, just include the dependency and: (the code below is Scala, but it is just as easy to use from Java)

// First we need to create a Node
val node = NodeBuilder.withInMemoryStorage().build()
// Then we can expose the native client using the SQS REST interface
val server = SQSRestServerFactory.start(node.nativeClient,
                   8888,
                   "http://localhost:8888")

// ... use: http://localhost:8888 exposes the full SQS interface ...

// Finally we need to stop the server and the node
server.stop()
node.shutdown()

ElasticMQ is written entirely in Scala, leveraging the Squeryl library for ORM and Netty for implementing the REST server.

The 0.1 release contains support for all SQS actions, providing a fully-functional SQS replacement. A tiny fraction message attributes is still not supported (e.g. SenderId).

Moreover, ElasticMQ will soon see its first production usage, serving as the messaging system behind one of the services we develop. The service used to be deployed on US-based servers, and it then successfully used the US-East SQS service. However, after moving the servers to Australia, as there’s no local AWS center, the communication delays with US and Singapore SQS servers became unacceptable; hence the need for a local SQS installation.

Adam

Envers bundled with JBoss AS 7.0.2!

Using Envers is now even easier! Since version 7.0.2, Envers comes bundled with JBoss Application Server.

To showcase how easy the integration is, I created a small JSF/CDI application, which uses Envers and can be deployed straight to AS7. To build the application I decided to try out JBoss Forge, which turned out to be very easy to use and provided me with a simple one-entity CRUD in no time. Just a few changes made it possible to track changes made to the entity, and view its history.

So if you are using AS 7.0.2, to add Envers to your app you just need to:

1. Add hibernate-envers to the pom.xml using the provided scope
2. Add @Audited to the entities that you want to audit

To view all the changes necessary in detail, just take a look at this commit (note that the MANIFEST.MF file is not required, as I initially mistakenly thought).

Apart from storing the changes, you may also want to view the history of an entity. That’s also pretty straightforward. All we need is to create a new JSF view and a CDI bean with a method running a simple history query. Again, this commit shows all the changes needed.

You can deploy the application in two ways:

1. Run mvn clean install, and copy the resulting .war to jboss/standalone/deployments
2. If you are using Forge, run forge from the checkout directory, and invoke: build, then: as7 deploy

After deployment, you should be able to access the application using http://localhost:8080/envers-as7-demo. Click on the Person link on the left to add, edit, list and view history of the entity.

By default, the application uses an example datasource, but if you wish to explore the additional schema generated by Envers you can easily change it to another database by modifying persistence.xml and creating a new datasource in the AS administration console.

As always I invite you to submit feedback on the Envers forum. Have fun!

Adam

How to handle inheritance and @Audited?

Some Envers uses had problems because of the limited flexibility in specifying which fields from superclasses (especially those annotated with @MappedSuperclass) should be audited or not. We improved a bit in the latest release, by being able to explicitly enumerate the superclasses to audit.

But still this is not fully flexible. That’s why Łukasz Antoniak started a discussion on the Envers forum (http://community.jboss.org/message/624057#624057) to gather remarks and use-cases on how people use Envers with inheritance. Your opinion will be very valuable to us, so please share!

Adam

Blending Ruby (on Rails) and CDI on OpenBlend 2011

I will have the pleasure to speak on OpenBlend 2011, in exactly a week: 15th September 2011, which will take place in Ljubljana, Slovenia.

Apart from many other interesting presentations, I will be demoing Torquebox & CDI integration. Torquebox is an enhanced JBoss AS, which besides being the regular JEE container everybody knows lets you deploy Ruby applications and provides Ruby integration for various AS components (like clustering or caching). This also includes applications written in Ruby on Rails.

Moreover, you can deploy mixed Ruby and Java applications. And that’s exactly what we will do: deploy an application which has a Ruby-on-Rails frontend (which will provide us with instance code changes) and a CDI/Weld backend (with all the goodies coming from a DI framework, typesafety and Java).

Additionally, everything will run on the brand-new and lightning-fast JBoss AS 7, which is the base for Torquebox 2.0.

See you there!
Adam

Modules, modules, modules …

I think everybody will agree that writing modular applications and modularity in general is a good thing.

But how does support for modularity look like, both from the Java and Scala languages and various Java/Scala frameworks? There’s a lot of different approaches! Let’s look at some of them. Below “protection” means how well modules are separated either at compile-time or run-time.

Packages

First of all we’ve got Java (or Scala) packages. The concept is very useful when it comes to organizing the source code. However it doesn’t offer compile-time or run-time (except for the package-protected visibility, which isn’t used too widely) protection, so it’s rather hard to say that packages are any help in modularizing the code. Also, there’s a lot of naming problems and hence conventions: if for example we have two implementations of a data reader, one using a database, the second using the filesystem, should the associated classes go to database and filesystem subpackages, or stay in the same top-level package? Should the class names be prefixed with Database and Filesystem, even if they are inside the dedicated packages?

Build subprojects

Secondly, there’s Maven/SBT modules/subprojects. They offer compile-time protection, which is a very nice thing: we can now statically make sure that our code only references classes from the explicitly specified dependencies. However, then comes the obvious question: when is a functionality “big enough” to make it a separate Maven module? Is it ok to have many Maven modules, each containing only several classes (which can be a PITA, having to define a separate pom, directory structure for each), or should they be bigger? What about naming conventions, should the package name correspond to the module name? If so, we’re clearly violating DRY here ;)!

Imagine we decide to put our database and filesystem data readers into separate packages, maven modules and name them appropriately. So we have a DatabaseReader (plus 10 helper classes) in a foo.bar.database package in the myapp-database Maven module. Now a simple refactoring: renaming “database” to “db” becomes a really complex task, most certainly ending up in using various terms in various places.

OSGi

The next step of the evolution would be OSGi bundles (or, equivalently, a module in JBoss Modules or in Project Jigsaw). One such bundle is typically a product of one (or more) Maven module, but it also offers run-time protection by appropriately scoping class-loaders. All of the problems from Maven modules are inherited … with the additional need to name the bundle!

Nested classes/cake pattern

Another possibility of scoping your code is using nested classes, or even to go further and use the Cake pattern in Scala. Apart from the old problems: what package to use and how to name the module class, we’ve got a couple new ones. Firstly, the whole source code of the module is now in one file. This can mean really long files! However maybe that’s more of an IDE problem: nobody said that a file needs to be edited all at once; the editor could show only one module class/method (like in the Smalltalk IDE). Cake pattern is also not free of problems. And, what about nested modules?

DI frameworks

Finally, we’ve got various DI frameworks (like Guice, Spring or CDI), where, if we look from an appropriate perspective, we define small modules (classes) which can depend on other modules (injected by the container). Separating the module interface (java interface) from the module implementation (java class), and only injecting the interface is also very common. But again, the only way to partially statically prevent injecting the implementation we have to resort to creating separate -api and -impl Maven modules.

In fact, if we take the approach of nested classes, a DI framework may be very well suited for resolving inter-module dependencies and doing all of the wiring for us (which can be very useful if we want to be protected from situations where a module has a new dependency, and each use-site must be now amended).

Next?

To sum up, I think that the various approaches to supporting modularity could use some unification. And, unfortunately, this would probably mean totally departing from what we know today from Java/Scala: a new package system, a new build system, a new runtime system. Unfortunately the outcrop of new programming languages doesn’t offer much in that area.

Do you think there should be one module system, usable both in the small (single classes) and in the large (sets of classes)? After all, what a DI container does to resolve the dependencies and instantiate classes isn’t so much different from what Maven or an OSGi runtime does when booting!

Or maybe modules inherently have several types, small ones (class-level) and big ones (maven module/osgi bundle-level)?

Lastly, maybe there are some other non-JVM based languages, which solve the modularity problem (still being usable) in a nicer way?

Adam

Introducing ElasticMQ: Scala SQS alternative

Wanting to explore Scala and some new Scala/Java libraries I started writing ElasticMQ. It is a simple message queueing system, following closely Amazon SQS semantics and exposing a REST SQS-like interface. Currently only the basic operations are implemented. ElasticMQ can be useful as an SQS replacement, e.g. for for testing SQS applications.

My aim was to make the usage as simple as possible, that’s why to create a new node and expose the REST interface all you need to do is:

val node = NodeBuilder.createNode
val server = SQSRestServerFactory.start(node.nativeClient, 8888,
                 "http://localhost:8888")

Now simply point your SQS client to http://localhost:8888 and you should be able to create/delete queues, change the visibility timeout, send and receive messages.
When you are done using ElasticMQ, you can shutdown the node and the server:

server.stop()
node.shutdown()

All of the source code is available on GitHub. See the readme for instructions on how to add ElasticMQ as a dependency to your Maven or SBT project.

On the technical side, ElasticMQ uses a number of interesting things.

Firstly, there’s Netty, an asynchronous, event-driven java NIO framework. I used Netty to implement the REST server. All of the requests are processed in a non-blocking way.

The messages are currently stored in an in-memory H2 database (but this can be easily changed to point e.g. to a MySQL or PostgreSQL DB), and managed using the Squeryl Scala library. Squeryl lets you write SQL queries as typesafe Scala code, e.g.:

update(messages)(m =>
  where(m.id === message.id and m.lastDelivered === message.lastDelivered)
    set(m.lastDelivered := lastDelivered))

For integration testing I am using the Typica library.

On the Scala side, I created a simple internal DSL for defining rest handlers. The server (which uses Netty) is generic and can be provided with any number of “rest handlers”, to which requests are dispatched. Each rest handler specifies what is the path it will respond to, what are the required parameters and what should be run in response to a request. For example:

val handler1 = (createHandler
   forMethod GET
   forPath (root / "products" / %("name") / "price")
   requiringParameters List("currency", "quantity")
   requiringParameterValues Map("department"->"electronics")
   running electronicsPriceRequestLogic)

val server = RestServer.start(handler1 :: handler2 :: … :: Nil, 8888)

See the RequestHandlerLogic file for the implementation and RequestHandlerBuilderTestSuite for example usage.

In fact there’s another small DSL for defining paths. For example (root / "a" / "b" / "c") will only match a/b/c, but (root / "a" / %("p1") / "c") will match a/anything/c, with the middle component being assigned to the p1 parameter. You can also use regex, e.g. (root / "a" / """[a-z0-9]+""".r / "c") for matching path components. See the RestPath class and test for details.

Not sure where ElasticMQ will go in the future, but I can image lots of interesting possibilities. So stay tuned :).

Comments about the code, Scala style, Netty/Squeryl usage are very welcome! As well as any other ideas, improvement suggestions etc.

Adam