Blog of Adam Warski

ElasticMQ 0.2 – support for delayed queues and messages

Time to start blogging in 2012 :)

I just released version 0.2 of ElasticMQ – a simple message queue system, which implements the Amazon SQS interface. Ideal if you want to test a service which uses SQS, or create a system which can leverage either the AWS infrastructure or run stand-alone.

The new version brings support for some new Amazon SQS features: delayed messages and queues. See the Amazon Blog for details.

Also, I changed the testsuite to use Amazon’s Java SDK. So you can be pretty sure that ElasticMQ works with SQS clients :). As an example, here’s the required code (in Scala, Java equivalent would be obviously a bit longer ;) ) to start an ElasticMQ node, send & receive a message using Amazon Java SDK and shutdown the server.

// First we need to create a Node
val node = NodeBuilder.withInMemoryStorage().build()
// Then we can expose the native client using the SQS REST interface
val server = SQSRestServerFactory.start(node.nativeClient, 8888,
      "http://localhost:8888")

// Now we need to create the sqs client
client = new AmazonSQSClient(new BasicAWSCredentials("x", "x"))
client.setEndpoint("http://localhost:8888")

// Using the client is quite straightforward
val queueUrl = client.createQueue(new CreateQueueRequest("queue1"))
      .getQueueUrl
client.sendMessage(new SendMessageRequest(queueUrl, "message1"))

client.shutdown()

// Finally we need to stop the server and the node
server.stop()
node.shutdown()

For more information about ElasticMQ, information on SBT and Maven dependencies, see the webpage.

Stay tuned,
Adam

ElasticMQ 0.1 released!

ElasticMQ is a simple messaging system, exposing an SQS-compatible REST interface. It can run using an in-memory H2 database (ideal for testing), or backed by a normal database (e.g. MySQL).

Using ElasticMQ can’t get much simpler, just include the dependency and: (the code below is Scala, but it is just as easy to use from Java)

// First we need to create a Node
val node = NodeBuilder.withInMemoryStorage().build()
// Then we can expose the native client using the SQS REST interface
val server = SQSRestServerFactory.start(node.nativeClient,
                   8888,
                   "http://localhost:8888")

// ... use: http://localhost:8888 exposes the full SQS interface ...

// Finally we need to stop the server and the node
server.stop()
node.shutdown()

ElasticMQ is written entirely in Scala, leveraging the Squeryl library for ORM and Netty for implementing the REST server.

The 0.1 release contains support for all SQS actions, providing a fully-functional SQS replacement. A tiny fraction message attributes is still not supported (e.g. SenderId).

Moreover, ElasticMQ will soon see its first production usage, serving as the messaging system behind one of the services we develop. The service used to be deployed on US-based servers, and it then successfully used the US-East SQS service. However, after moving the servers to Australia, as there’s no local AWS center, the communication delays with US and Singapore SQS servers became unacceptable; hence the need for a local SQS installation.

Adam

Envers bundled with JBoss AS 7.0.2!

Using Envers is now even easier! Since version 7.0.2, Envers comes bundled with JBoss Application Server.

To showcase how easy the integration is, I created a small JSF/CDI application, which uses Envers and can be deployed straight to AS7. To build the application I decided to try out JBoss Forge, which turned out to be very easy to use and provided me with a simple one-entity CRUD in no time. Just a few changes made it possible to track changes made to the entity, and view its history.

So if you are using AS 7.0.2, to add Envers to your app you just need to:

1. Add hibernate-envers to the pom.xml using the provided scope
2. Add @Audited to the entities that you want to audit

To view all the changes necessary in detail, just take a look at this commit (note that the MANIFEST.MF file is not required, as I initially mistakenly thought).

Apart from storing the changes, you may also want to view the history of an entity. That’s also pretty straightforward. All we need is to create a new JSF view and a CDI bean with a method running a simple history query. Again, this commit shows all the changes needed.

You can deploy the application in two ways:

1. Run mvn clean install, and copy the resulting .war to jboss/standalone/deployments
2. If you are using Forge, run forge from the checkout directory, and invoke: build, then: as7 deploy

After deployment, you should be able to access the application using http://localhost:8080/envers-as7-demo. Click on the Person link on the left to add, edit, list and view history of the entity.

By default, the application uses an example datasource, but if you wish to explore the additional schema generated by Envers you can easily change it to another database by modifying persistence.xml and creating a new datasource in the AS administration console.

As always I invite you to submit feedback on the Envers forum. Have fun!

Adam

How to handle inheritance and @Audited?

Some Envers uses had problems because of the limited flexibility in specifying which fields from superclasses (especially those annotated with @MappedSuperclass) should be audited or not. We improved a bit in the latest release, by being able to explicitly enumerate the superclasses to audit.

But still this is not fully flexible. That’s why Łukasz Antoniak started a discussion on the Envers forum (http://community.jboss.org/message/624057#624057) to gather remarks and use-cases on how people use Envers with inheritance. Your opinion will be very valuable to us, so please share!

Adam

Blending Ruby (on Rails) and CDI on OpenBlend 2011

I will have the pleasure to speak on OpenBlend 2011, in exactly a week: 15th September 2011, which will take place in Ljubljana, Slovenia.

Apart from many other interesting presentations, I will be demoing Torquebox & CDI integration. Torquebox is an enhanced JBoss AS, which besides being the regular JEE container everybody knows lets you deploy Ruby applications and provides Ruby integration for various AS components (like clustering or caching). This also includes applications written in Ruby on Rails.

Moreover, you can deploy mixed Ruby and Java applications. And that’s exactly what we will do: deploy an application which has a Ruby-on-Rails frontend (which will provide us with instance code changes) and a CDI/Weld backend (with all the goodies coming from a DI framework, typesafety and Java).

Additionally, everything will run on the brand-new and lightning-fast JBoss AS 7, which is the base for Torquebox 2.0.

See you there!
Adam

Modules, modules, modules …

I think everybody will agree that writing modular applications and modularity in general is a good thing.

But how does support for modularity look like, both from the Java and Scala languages and various Java/Scala frameworks? There’s a lot of different approaches! Let’s look at some of them. Below “protection” means how well modules are separated either at compile-time or run-time.

Packages

First of all we’ve got Java (or Scala) packages. The concept is very useful when it comes to organizing the source code. However it doesn’t offer compile-time or run-time (except for the package-protected visibility, which isn’t used too widely) protection, so it’s rather hard to say that packages are any help in modularizing the code. Also, there’s a lot of naming problems and hence conventions: if for example we have two implementations of a data reader, one using a database, the second using the filesystem, should the associated classes go to database and filesystem subpackages, or stay in the same top-level package? Should the class names be prefixed with Database and Filesystem, even if they are inside the dedicated packages?

Build subprojects

Secondly, there’s Maven/SBT modules/subprojects. They offer compile-time protection, which is a very nice thing: we can now statically make sure that our code only references classes from the explicitly specified dependencies. However, then comes the obvious question: when is a functionality “big enough” to make it a separate Maven module? Is it ok to have many Maven modules, each containing only several classes (which can be a PITA, having to define a separate pom, directory structure for each), or should they be bigger? What about naming conventions, should the package name correspond to the module name? If so, we’re clearly violating DRY here ;)!

Imagine we decide to put our database and filesystem data readers into separate packages, maven modules and name them appropriately. So we have a DatabaseReader (plus 10 helper classes) in a foo.bar.database package in the myapp-database Maven module. Now a simple refactoring: renaming “database” to “db” becomes a really complex task, most certainly ending up in using various terms in various places.

OSGi

The next step of the evolution would be OSGi bundles (or, equivalently, a module in JBoss Modules or in Project Jigsaw). One such bundle is typically a product of one (or more) Maven module, but it also offers run-time protection by appropriately scoping class-loaders. All of the problems from Maven modules are inherited … with the additional need to name the bundle!

Nested classes/cake pattern

Another possibility of scoping your code is using nested classes, or even to go further and use the Cake pattern in Scala. Apart from the old problems: what package to use and how to name the module class, we’ve got a couple new ones. Firstly, the whole source code of the module is now in one file. This can mean really long files! However maybe that’s more of an IDE problem: nobody said that a file needs to be edited all at once; the editor could show only one module class/method (like in the Smalltalk IDE). Cake pattern is also not free of problems. And, what about nested modules?

DI frameworks

Finally, we’ve got various DI frameworks (like Guice, Spring or CDI), where, if we look from an appropriate perspective, we define small modules (classes) which can depend on other modules (injected by the container). Separating the module interface (java interface) from the module implementation (java class), and only injecting the interface is also very common. But again, the only way to partially statically prevent injecting the implementation we have to resort to creating separate -api and -impl Maven modules.

In fact, if we take the approach of nested classes, a DI framework may be very well suited for resolving inter-module dependencies and doing all of the wiring for us (which can be very useful if we want to be protected from situations where a module has a new dependency, and each use-site must be now amended).

Next?

To sum up, I think that the various approaches to supporting modularity could use some unification. And, unfortunately, this would probably mean totally departing from what we know today from Java/Scala: a new package system, a new build system, a new runtime system. Unfortunately the outcrop of new programming languages doesn’t offer much in that area.

Do you think there should be one module system, usable both in the small (single classes) and in the large (sets of classes)? After all, what a DI container does to resolve the dependencies and instantiate classes isn’t so much different from what Maven or an OSGi runtime does when booting!

Or maybe modules inherently have several types, small ones (class-level) and big ones (maven module/osgi bundle-level)?

Lastly, maybe there are some other non-JVM based languages, which solve the modularity problem (still being usable) in a nicer way?

Adam

Introducing ElasticMQ: Scala SQS alternative

Wanting to explore Scala and some new Scala/Java libraries I started writing ElasticMQ. It is a simple message queueing system, following closely Amazon SQS semantics and exposing a REST SQS-like interface. Currently only the basic operations are implemented. ElasticMQ can be useful as an SQS replacement, e.g. for for testing SQS applications.

My aim was to make the usage as simple as possible, that’s why to create a new node and expose the REST interface all you need to do is:

val node = NodeBuilder.createNode
val server = SQSRestServerFactory.start(node.nativeClient, 8888,
                 "http://localhost:8888")

Now simply point your SQS client to http://localhost:8888 and you should be able to create/delete queues, change the visibility timeout, send and receive messages.
When you are done using ElasticMQ, you can shutdown the node and the server:

server.stop()
node.shutdown()

All of the source code is available on GitHub. See the readme for instructions on how to add ElasticMQ as a dependency to your Maven or SBT project.

On the technical side, ElasticMQ uses a number of interesting things.

Firstly, there’s Netty, an asynchronous, event-driven java NIO framework. I used Netty to implement the REST server. All of the requests are processed in a non-blocking way.

The messages are currently stored in an in-memory H2 database (but this can be easily changed to point e.g. to a MySQL or PostgreSQL DB), and managed using the Squeryl Scala library. Squeryl lets you write SQL queries as typesafe Scala code, e.g.:

update(messages)(m =>
  where(m.id === message.id and m.lastDelivered === message.lastDelivered)
    set(m.lastDelivered := lastDelivered))

For integration testing I am using the Typica library.

On the Scala side, I created a simple internal DSL for defining rest handlers. The server (which uses Netty) is generic and can be provided with any number of “rest handlers”, to which requests are dispatched. Each rest handler specifies what is the path it will respond to, what are the required parameters and what should be run in response to a request. For example:

val handler1 = (createHandler
   forMethod GET
   forPath (root / "products" / %("name") / "price")
   requiringParameters List("currency", "quantity")
   requiringParameterValues Map("department"->"electronics")
   running electronicsPriceRequestLogic)

val server = RestServer.start(handler1 :: handler2 :: … :: Nil, 8888)

See the RequestHandlerLogic file for the implementation and RequestHandlerBuilderTestSuite for example usage.

In fact there’s another small DSL for defining paths. For example (root / "a" / "b" / "c") will only match a/b/c, but (root / "a" / %("p1") / "c") will match a/anything/c, with the middle component being assigned to the p1 parameter. You can also use regex, e.g. (root / "a" / """[a-z0-9]+""".r / "c") for matching path components. See the RestPath class and test for details.

Not sure where ElasticMQ will go in the future, but I can image lots of interesting possibilities. So stay tuned :).

Comments about the code, Scala style, Netty/Squeryl usage are very welcome! As well as any other ideas, improvement suggestions etc.

Adam

Using Envers with AS7

Recently I tried deploying a web application which uses Envers into AS7, but unfortunately I encountered some problems (see the forum discussion). Luckily thanks to the helpful JBoss guys I’ve got it working now. Moreover, thanks to the work done by Strong Liu, you should see full Envers integration (included OOTB as a module) in AS7.1!

Until then, here’s how to use Envers in a webapp in AS7.

First, we need to create an Envers module. Create a directory: jboss-as-7.0.0.Final/modules/org/hibernate/envers/main and inside it, create a module.xml with the following content:

<?xml version="1.0" encoding="UTF-8"?>
<module xmlns="urn:jboss:module:1.0" name="org.hibernate.envers">
  <resources>
    <resource-root path="hibernate-envers-4.0.0.Beta1.jar"/>
  </resources>

  <dependencies>
    <module name="org.hibernate"/>
    <module name="org.jboss.logging"/>
    <module name="org.dom4j"/>
    <module name="javax.api"/>
    <module name="javax.persistence.api"/>
    <module name="javax.transaction.api"/>
    <module name="org.javassist"/>
  </dependencies>
</module>

Moreover, you’ll need to put the Envers jar inside that directory. You can download it straight from the JBoss Maven repository.

I’ve prepared a ready zip of the module, which can be downloaded here. Just unpack its contents to jboss-as-7.0.0.Final/modules/org/hibernate.

Secondly, you must add an entry in your manifest file (META-INF/MANIFEST.MF) to make the new module available to your application:

Dependencies: org.hibernate.envers services

And you’re done. Enjoy! :)

Adam

Trying to understand CAP

The CAP theorem, stated by Brewer and proved by Gilbert and Lynch specifies a property of distributed systems. It states that such a system cannot guarantee at the same time Consistency, Availability and Partition tolerance. It is also often said as a catchy phrase:

Consistency, Availability, Partition Tolerance – pick any two

used mostly when talking about NoSQL databases and suggesting that a distributed system can be characterized as either CA, AP or CP (see e.g. here).

For some time I’ve been trying to understand the different combinations and what do they mean in practice; having some time at various airports I caught up on reading, and here’s what I came up with.

What’s C, A and P?

First let’s define how I understand the three guarantees, basing on some of the articles I’ve read.

Consistency is the easiest one. It roughly means that the clients get the same view of data. By saying that a system is consistent we often mean strong consistency, but it also can come in different flavors, e.g. casual.

Availability is a property saying that every request to a non-failing node will return a (meaningful) response. The response may not contain all data (so the harvest will not be 100%, see the appropriate section in [3]), but it should be useful for the client.

Partition tolerance means that the system will continue working even if any number of messages sent between nodes is lost. This can be e.g. a network failure between two datacenters, where nodes in each datacenter form a partition. Also note that a failure of any number of nodes forms a partition (it is not possible to distinguish between a network failure and a node failing and stopping to respond to messages).

The hardest part for me is understanding the difference between Availability and Partition tolerance. Also, the various articles don’t specify what they mean by saying that a system is “working” after being e.g. partitioned – does it mean that every request gets a response with useful data, or are responses “Sorry, I can’t give you data right now” acceptable also?

P+C?

Let’s assume that a system is partition tolerant and that it has more than one node. If a partition is formed, splitting the system in two, the system should continue working. Hence both partitions allow clients to write. But then, how to guarantee consistency? If one client writes to partition 1, and another to partition 2? Hence: P => ~C.

A+~P?

Suppose now that a system is available and that we have more than one node. As the system is available, it should respond to requests even if some nodes die. As noted above, some nodes dying are equivalent to a partition. So if the system is still working, we have partition tolerance. Hence: A => P.

A+C?

Summarizing the two implications above (A => P and P => ~C), we get: A => ~C, so that an available system cannot be consistent (if it has more than one node). In practice however, there are of course AC systems, e.g. single-node RDBMS. Or even master-slave/master-master replicated RDBMS, provided there’s a central router knowing which nodes live and directing client appropriately. Such a router is then a single point of failure (SPoF).

Relax?

I suspect that in reality, when e.g. NoSQL/NewSQL systems are characterized with the CAP properties, they assume some relaxed form of C/A/P. Unfortunately, all of the definitions flying around seem to be pretty vague and are more of hand-waving than proper, precise statements. I think it would be much easier to explore the ever-growing ecosystem of new datastores if they could be more easily characterized; maybe the CAP vocabulary is just not enough?

Please correct me if I’m wrong somewhere, and I probably am! :)

Adam

Some articles I used for my research:

[1] http://www.julianbrowne.com/article/viewer/brewers-cap-theorem
[2] http://www.cloudera.com/blog/2010/04/cap-confusion-problems-with-partition-tolerance/
[3] http://codahale.com/you-cant-sacrifice-partition-tolerance/

Envers: track entity types changed in revisions

Łukasz Antoniak, who is a regular contributor to Envers (thanks!), recently finished working on HHH-5580, a feature which enables you to quickly retrieve the entity types that have been modified in a revision, by optionally storing them in a dedicated table.

Łukasz wrote an excellent tutorial on the subject, so if you are intersted, be sure to read it!

The feature described is available since the Hibernate 4.0.0.Beta2 release.

Adam