Performance

Gatling load testing

In mail servers, performance do matters. Users expect good, and fast responding commands. Thus performance is a priority concern for Apache James developers.

As part of our work on the project, we need rock-solid tools to benchmark our software. And this across all protocols (JMAP, IMAP and SMTP). We used third-party Gatling load testing implementations for :

  • Testing main JMAP commands
  • Testing main IMAP commands
  • Testing basic SMTP scenarios

Load testing was conducted both on top of JPA and Cassandra + OpenSearch implementations. We succeeded to support 1.000 Thunderbird like users on top of both backends.

The following results were obtained on a server with an Intel Xeon CPU (E3–1231 3.40GHz 4 cores 8 threads) with 32 GB of RAM. We run Gatling on the same host, thus it has performance impact on the results. So far Gatling is run manually when we take time for it. This might eventually become part of our CI.

Metric collection

Additionally, we added metrics a bit everywhere in James using the brand new metrics API. We collect and export everything in OpenSearch using Dropwizard metrics. Then we graph it all using Grafana. This allows us to collect all statistics and percentiles. Boards can be downloaded here.

    We track so far:  
  • Protocols detailed time execution (and counts)
  • Percentile of mailet and matcher executions
    •  
  • Enqueue and Dequeue time
    All these solutions allow us to identify the components that need improvement. Here are some example of classical metrics James can reach with a mono server Guice + Cassandra + OpenSearch installation:  
  • About 100 users
    •  
  • So far a number of 1.500.000 e-mails, that we imported using IMAP-sync scripts
  • Around 15.000 incoming e-mails a day
    •  
  • 42.500 IMAP commands a day
    •  
  • 20.000 JMAP requests a day

More details on metrics are available on the metric page.

Measuring dequeueing speed

James can spool more than 65 mails per second (configured with Maildir mail storage and the FileMailQueue).

We ran some SMTP Gatling load testing during 5 minutes (https://github.com/linagora/james-gatling)

sbt
gatling:testOnly org.apache.james.gatling.smtp.scenari.NoAuthenticationNoEncryptionScenario

The result was:

  1. During these 5 minutes, 29.924 mails where enqueued (~100 email per second)
  2. The dequeueing of these emails took 7 minutes and 40 seconds (~65 emails per second)

We thus have a throughput of more than 65 mails/second.

These numbers are obtained with the out-of-the-box configuration (not optimized), on a HDD disk. Better performance is expected with better hardware and further JVM tuning (-XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods).