Integration testing with Ranger

range – /reɪn(d)ʒ/ (noun): the area of variation between upper and lower limits on a particular scale.

Intro

Filling the database with test data needed for integration testing can be a quite tiresome, repetitive, and error prone process, because you have to create a lot of objects with different properties and values and to make sure that all cases are covered. We wanted to remove the annoyance of creating test data manually (one object at a time), so we created Ranger – a free and open source Java library that allows for easy, declarative test data generation. You just have to describe your objects in regard to properties and values and Ranger will generate objects for you. For example, you can do something like this: create me a thousand instances of random Measurement objects such that exactly 10 are with username “charlie” and are from “heart_beat_monitor” sensor. Do you see how this can be useful for integration testing? Here’s a simple example.

Let’s say that you are working on IoT software and you have a query that fetches 50 newest measurements for the given owner and sensor, which probably looks something like this:

or in MongoDB query language:

You want to write an integration test for the query. The test data for the query should give the query a chance to be wrong. Which means that, before we run the query, we want to have a bunch or users with abunch of sensor data in the database. That way we can make sure that the user and the sensor selection logic of the query (WHERE owner = ? AND sensor = ?) works as expected. Also, we want to have more than 50 measurements to test our LIMIT 50 logic; and itwould be nice if data is not presorted in the database, so we can cover SORT BY created DESC part of the query.

Ranger allows you to do that programmatically in a declarative manner.

Good Software Engineering and Testing Practice

Why do I need this library? Why would I write complex database queries when I could just do the filtering and sorting in Java which are easily testable? Why would I pay so much attention to testing the database queries anyway? Can’t I assume that my database query just works, since it’s declarative after all?

I will probably address these questions in a separate blog post. But just to be on the same page with all the readers, this is what we consider good practice and what we have in mind while developing Ranger:

• Fetching, filtering, and sorting of the data should be done by the database language (SQL, cql, mongodb query language, n1ql etc.) which is the most natural and most performing way to do it.
• Since a large part of business logic is written in the database language, that logic should be covered with automatic (integration or functional) tests. Why?
• to make sure that complex queries do exactly what you want them to do
• to be able to do refactoring and make changes and be sure that the query still works as expected
• to be able to change/update the database driver (or even the database itself) to document what the business logic does and what it does not do(or what it must not do).

A brief intro to Ranger

For starters, let’s just see how Ranger is used and what is the idea behind it.

In order to test database queries, we need to create test data for each test case and put it in the database. This usually involves:

1. creating test data – Java objects (or entities if you like)
2. then inserting them into the database
3. then running the method/query that fetches the data
4. then validating that the actual result is also the expected result

The first step, creating test data for each test case, usually takes the most time, it is repetitive and error prone. You have to create each object separately and create several objects that are expected to be retrieved, then several objects that should not be retrieved and so on. This can be quite verbose, time consuming and hard to maintain afterwards.

Therefore, we created and open-sourced Ranger – a tool that allows us to declaratively generate any number of objects programmatically. You just have to describe how you want the objects of a given class to look like, what are the allowed values for each attribute, and Ranger will use reflection and set the randomly chosen value to the class’ field. Before proceeding, if you already haven’t, please take a look at Ranger’s github repository. I’ll wait here.

Okay, you’re back. So, as you’ve just seen, with Ranger, we can describe how our data looks like. As we said, we’ll use the IoT app example, where the core part of the system is Measurement class, which looks like this:

Ranger can help us with creating test Measurement objects, so we can populate the database before running an integration test. With Ranger, we can create recipes in which we declare possible values for each field. For example we can create a recipe like this: generate 50 objects of Measurement class such that owner is someone from this set (“alex”, “bob”, “charlie”), and created is a timestamp between [1496300000, 196400000), and sensor can be something from this set (“heart-rate-monitor”, “accelerometer”, “hygrometer”, “thermometer”) and so on. The recipe looks like this:

Simple, isn’t it? We can also combine the recipes so we can make data abide certain rules like: create a thousand Measurement objects for a bunch of users and with abunch of sensors, but only 100 of them will be from user “charlie”. In other words, we want 900 measurement objects for non-charlie users and 100 for user “charlie”:

And now we just pass nonCharlieMeasurementGenerator and charlieMeasurementGenerator to AggregatedObjectGenerator in order to generate objects from both generators:

Voilà!

Ranger – real world example

Now that we are familiar with the basic building blocks for Ranger library, let’s try it out with a real world example. I used to work on an IoT application that collects measurements from bunch of sensors and provides analytics and alerting based on the stored data. The application was querying MongoDB (which was used for storage), fetching n newest results for a particular user and sensor. So I had the mongodb query that looked something like this:

or translated to SQL (for people unfamiliar with Mongo):

And, of course, I wanted to create automated tests for that query. For simplicity, in this example, I will test all the cases in one test method. If you like to do it by the book you would probably create a separate, orthogonal test for each case.

As we can see, the test data is created in createTestMeasurements(). If we take a look at the query under test:

and its wrapper method:

We can see there are several components, or dimensions of the query. First, there is a username and a sensor type. Then, there is a timestamp (“created” field); and in the end there is a limit (we want only thelast n measurements, in this case: 50);

But, in order to test the correctness of the query, it is of critical importance to cover every dimension(property) of the query. Why? Because we want to give our query a chance to be incorrect. If we only had data for one user, we would not see if our user selection part of the query works. Or, if we only have one sensor we can’t know that the sensor selection part of the query works correctly. And so on.

How to create test data for the query?

Let’s take a look at the query once again:

When I need to write a test, I usually draw a table with expected/unexpected results to be sure that the test data is correctly created and that all the cases are covered:

Okay, so, we need:

1. expected result – 50 objects should have username : “charlie”, sensor : “thermometer” and be created between 1000 (inclusive) and 1100 (exclusive).
2. results with wrong username – abunch of objects (say 1000) will be the same as 1. but with different username
3. results with wrong sensor – abunch of objects (say 1000) will be the same as 1. but with different sensor
4. results with wrong timestamp – abunch of objects (say 1000) will be the same as 1. but created property’s values is below1000.

By increasing the number of test objects/records, we reduce the chances of passingthe test only because some strange coincidence occurred. For example, if I have only three objects in the database and I didn’t pay attention, there’s a high probability that the objectsare already sorted by timestamp in the database (because I saved them into thedatabase in that order). In that case, the test will be passedeven if the sorting part of the query is missing. But if the timestamp is randomly generated, and we have 50 objects in the database, the probability of having presorted records in the database is quite low (1 in 50!). That way we reduce the probability of making a mistake in thetest itself. I mean, we can even say that it’s better not to write a test at all than write the one that isalways passed.

With Ranger, once you declare how the data looks like, you can create any number of objects. I always prefer creating thousands of objects for integration testing. That way I can increase the probability of discovering some edge case that I havefailed to notice.

In addition to this, we want to give meaningfulvalues to all the attributes. It’s hard to debug and spot errors in a bunch of jibberish strings. For example, if I’m creating sensor data, I want them to have the names of real sensors. I want that in integration tests, because it is much easier to reason about them if the values are real; thus it’s easier to debug and maintain the tests later on.

But, back to the example. First, I’ll create a builder (or recipe, description) for 50 newest thermometer measurements for user ‘charlie’:

In this part I declared how the expected result looks like. Once I fetch the data in the test, I can check that the value of created property is in the correct range [1000, 1100). Since I’m in control of the test data, I just have to make sure that I don’t create newer data for this particular user.

Have you heard of those nasty off-by-one errors? Of course you have. Well, you can forget about those now, because Ranger will always create corner cases for you. What does that mean? That means that if you declare the range of values for a property, it will always include the range boundaries in the generated data. The range is defined with an inclusive beginning and exclusive end. In this case when I declare .withRanges(“created”, 15000L, 15050L) that means that at least one instance of the generated Measurement objects will have “created” with value 15000 (the beginning of the range) and at least one instance will have “created” with value 15049 (the end of the range).

Okay, so far I have declared a set of expected results. Now I need noise data – the overlapping data that mustn’t be part of the returned data set. First, I’ll create data for other users which will contain data from all the sensors, including the “thermometer” and will include newer and older measurements than in the expected result:

Now, let’s create measurements for “charlie” that are newer than what we defined in the expected result, but from the wrong sensor:

The final step is to create some Measurement objects for the correct user (owner: “charlie”) and correct sensor (sensor : “thermometer”) but the timestamp (created) is older than in the 50 results that are part of the expected result set:

Now, let’s take a look at what we have declared:

We can see that test data declared like this is:

• relatively easy to reason about
• the data set description fits into eye view
• each part of data set is (or at least could be) meaningfully named – newestMeasurementsForCharlieAndThermometer (as long as you love how things are named in Java :D)

Next Steps

Honestly, I was really surprised that something like this did not exist. On every project I worked on I had to create test data from the ground up, one object at the time. That lead to errors and frustration. With Ranger, you still have to pay attention when creating test data, but the creation is much easier to do it declaratively. What do you think about this? Would you find it useful? If you have any feature ideas or comments, please create an issue on Ranger’s github repo, or just ping any of us on Twitter (I’m @milannister).

If you’re interested and want to try Ranger out, you can take a look at the demo application which uses embedded MongoDB for integration testing.

P.S. We’re currently working on making Ranger run from the command line. That way we’ll be able to describe our data set in a yaml configuration file and just fill the database or API with generated objects/entities/requests. This will come in handy when you want to quickly populate the database with some meaningful values.