A simple tool for load testing stateful systems using Clojure

When it comes to understanding software application performance, software engineering often unfolds like a Chuck Norris action movie - fires, explosions, elevator fistfights. Often, these edge-of-your-seat gymnastics are triggered by the customer using your shiny new deployment doing ordinary things.

It may be that the first few times the elevator cables get cut, you have to jump, dodge, roll, fly through a window, and land on a moving military supply truck. And while this may be exciting at first, leaving you feeling just like Chuck Norris, at some point, we all figure that less exciting Friday afternoon matinees are probably best for our longevity.

Recently I've been lucky to work on a project that planned sufficient time for realistic performance testing before reaching the customer’s hands.

Naively firing off a barrage of HTTP requests in our case would be insufficient. Due to the nature of the product - key sharing - simulating user interaction was inherently stateful, requiring cryptographic signatures for each request combined with nonce updates. Because the essential operations require interactions between two parties, customer A shares a key with customer B, the construction of payloads requires knowledge of input and output from previous requests.

While we could have tried to reuse an existing loading testing or simulation testing library - given the project time frame, we decided to build a simple tool ourselves as Clojure is well suited to the rapid development of focused, simple, data-driven solutions.

We set the following expectations:

• Simple declarative way to define a test scenario
• Simple way to customize the steps

As we only needed to simulate a few hundred users at this time, the tool could be run on one machine. Given such modest needs, core.async seemed like a good, no-fuss foundation to build on.

In our load testing library, we have a single master that spawns executors that run the workflow.

(io.vouch.load-tests.master/start
    {:scenario scenario
        :reporter reporter})

Scenarios are just plain Clojure data, also known as EDN (Extensible Data Notation). Scenarios declaratively describe workflows and how many actors should participate in a given scenario.

(def scenario
    {
        :workflows {:listener [{:task :register-user}
            {:task :wait :duration 5}
            {:task :listen-to-friend-requests}]
            :inviter  [{:task :register-user}
            {:task :send-friend-request}]}
        :actor-pools [{:workflow :listener :actors 100}
            {:workflow :inviter :actors 10}]
    }
)

Workflows compose of tasks. Above are two workflows - :listener and :inviter. In each workflow there is a sequence of tasks. Each task definition map must have a :task key corresponding to a Clojure multimethod implementation. The task may contain additional configuration data that it also needs — the framework ships with a few common tasks.

Here's an example, waiting:

i.e. `{:task :wait :duration 5}` the `wait` task requires `duration` property.

(defmethod io.vouch.load-tests.executor/execute-task :wait
    [{:keys [id] :as executor} {:keys [duration] :as task}]
    (go
        (log/info id task)
        (<! (timeout (* 1000 duration)))))

Already, we can see that we’ve satisfied our first goal - a declarative way to define scenarios. But what about customizing the details? This also needs to pass the simplicity test.

Implementing a task<

To implement a new type of task, we define a defmethod io.vouch.load-tests.executor/execute-task. For example, :register-user task:

(defmethod io.vouch.load-tests.executor/execute-task :register-user
    [executor msg]
    (go
        (let [email (str "tester-" (rand) "@example.com")
            password (str (rand))]
        (http/post (str "http://example.com/api/user/register")
            {:body (json/encode {:email email :password password})
            :headers {:content-type "application/json"
            :accept "application/json"}}))))

The execute-task implementation must return a core.async channel.

Executor config

There are two issues with the above implementation of :register-user. The email should be unique, and with (rand), duplicates are possible. The second issue is the hardcoded URL – we may also want to reuse the same step against different environments.

Let’s address these issues by leveraging some features of the framework.

The executor inherits the config passed when starting the master. Let's pass the URL as configuration rather than hard coding it into the task:

(io.vouch.load-tests.master/start
    {...
    :api-url (System/getenv "API_URL")})

Now we can access the api-url inside the task:

(defmethod io.vouch.load-tests.executor/execute-task :register-user
    [{:keys [api-url]} msg]
    (go
        (let [email (str "tester-" (rand) "@example.com")
            password (str (rand))]
        (http/post (str api-url "/user/register")
        {:body (json/encode {:email email :password password})
        :headers {:content-type "application/json"
        :accept "application/json"}}))))

Next, we address the generation of random but unique emails.

(defn random-email
    []
    (str "tester-" (rand) "@example.com"))

(io.vouch.load-tests.master/start
    {...
        :api-url (System/getenv "API_URL")
        :unique-email (create-unique-generator random-email)})

(defmethod io.vouch.load-tests.executor/execute-task :register-user
    [{:keys [api-url unique-email]} msg]
    (go
        (let [email (unique-email)
            assword (str (rand))]
        (http/post (str api-url "/user/register")
        {:body (json/encode {:email email :password password})
        :headers {:content-type "application/json"
        :accept "application/json"}}))))

Now, it's much better!

Executor state

Our executors can register user accounts, but how do they authenticate subsequent requests? Our sample backend returns an auth token as a response to successful registration, but how can the executor access that token between tasks?

Each executor has its own state where our tasks can store data that should be accessible for subsequent tasks.

(defn- register-user
    [api-url email password]
    (let [response (http/post (str api-url "/user/register")
        {:body (json/encode {:email email :password password})
        :headers {:content-type "application/json"
        :accept "application/json"}})]
    (some-> response :body (json/decode true) :token)))

(defmethod io.vouch.load-tests.executor/execute-task :register-user
    [{:keys [api-url id unique-email state]} msg]
    (go
        (log/info id msg)
        (let [email (unique-email)
            password (str (rand))
            token (register-user api-url email password)]
        (swap! state assoc :auth-token token))))

We can see on the last line that we're updating the executor's state with auth-token. We've extracted the logic responsible for making HTTP requests and parsing responses into a separate function for readability.

Now we can access the token from the next task.

(defn- friend-requests
    [api-url auth-token]
    (http/get (str api-url "/user/friend-requests")
        {:headers {:authorization (str "Bearer " auth-token)
                   :content-type "application/json"
                   :accept "application/json"}}))

    (defmethod io.vouch.load-tests.executor/execute-task :listen-to-friend-requests
        [{:keys [api-url id state]} msg]
        (go
            (log/info id msg)
            (let [auth-token (-> state deref :auth-token)]
            (friend-requests api-url auth-token))))

Accessing other executors from within a task

We may need to access information about other executors. Consider a scenario where actors interact with each other through the backend. i.e., actor A sends a friend request to actor B, and actor B accepts or rejects the invitation – we can access other executors using the get-executors function.

(defmethod io.vouch.load-tests.executor/execute-task :send-friend-request
    [{:keys [api-url state get-executors]} msg]
        (go
            (let [auth-token (-> state deref :auth-token)
                email (-> (get-executors) shuffle first :state deref :email)]
            (send-friend-request api-url auth-token email))))

Filtering executors

In the example above, we shuffled the list of executors and picked the first one. But what if we want to define different pools of actors that behave differently? Each executor inherits three items from the actor-pool they belong to:

[:behavior :tags :workflow]

Let's consider this scenario:

{:workflows {:listeners [{:task :register-user}
    {:task :listen-to-friend-requests}]
    :inviter [{:task :register-user}
        {:task :wait :duration 1}
        {:task :send-friend-request
        :to {:behavior {:accept-friend-request true}}}
        {:task :send-friend-request
        :to {:accept-friend-request false}}
        {:task :send-friend-request
        :to {:workflow :listeners}}
        {:task :send-friend-request
        :to {:tags [:singleton]}}]}
:actor-pools [{:workflow :listeners :actors 10
    :behavior {:accept-friend-request true}}
    {:workflow :listeners :actors 10
    :behavior {:accept-friend-request false}}
    {:workflow :listeners :actors 1
    :tags [:singleton]}
    {:workflow :inviter :actors 10}]}

Some users are listening to friend requests (perhaps polling the backend for invitations), and others are sending friend requests. The :listen-to-friend-requests task polls the backend for invitations, and if any are found, it accepts or rejects them based on the :behavior of the :actor-pool the executor belongs to. This allows us to have only one task implementation but customize the behavior in a declarative way on the scenario level. That also saves us on the number of workflows we must define because multiple actor pools can use the same workflow but still have slightly different behavior.

Now the :inviter is supposed to send a friend request. We want the first request to be sent to somebody that will accept it. We do that by augmenting the task with {:to {:behavior true}}.

The second request should be sent to somebody that will reject it.

The third request should be sent to anybody that will act on it. We know that actors executing :listeners workflow will do something with friend requests; hence we use the {:to {:workflow :listeners}} selector.

The fourth request must go from all inviters to the same user. That's why we have an actor pool tagged as :singleton that has only one actor.

Here's our code for the task:

(defmethod io.vouch.load-tests.executor/execute-task :send-friend-request
    [{:keys [api-url state] :as executor} {:keys [to]}]
    (go
        (if-let [email (some-> executor
            (io.vouch.load-tests.executor/filter-executors to #(some-> % :state deref :email))
            shuffle first :state deref :email)]
        (let [auth-token (-> state deref :auth-token)]
            (send-friend-request api-url auth-token email))
        (log/warn "No executor matching following criteria" to))))

We’ve now satisfied our second goal - a simple way to customize the steps. Again, we lean on a data-driven approach over a code-centric approach to lower the potential for complexity.

An adaptable framework

While the scenarios above are similar to our present use cases, they're transferrable and can be applied to other problem domains where a simulation-based approach to load testing is called for.

So, now that our Friday afternoons are much more peaceful it leaves us to seek our excitement elsewhere, jumping, rolling, and flying our way into the next adventure.

Find the complete Github repository for the code shown here