HTTP: Stress Testing
Sumber: https://www.blazemeter.com/blog/open-source-load-testing-tools-which-one-should-you-use
Performance Testing
Sep. 29th, 2018
日本語
Open Source Load Testing Tools: Which One Should You Use? Listen to this post: Open Source Load Testing Tools: Which One Should You Use? 00:00 14:36
Is your application, server, or service delivering the appropriate speed of need? How do you know? Are you 100-percent certain that your latest feature hasn’t triggered a performance degradation or memory leak? There's only one way to verify - and that's by regularly checking the performance of your app.
But which tool should you use for this? In this blog post, we'll review the pros and cons of the leading open-source solutions for load and performance testing.
If you're like many, chances are you've already seen this great list of 53 of the most commonly used open source performance testing tools. However, the truth is many of these tools aren’t necessarily suited to our needs. Some are limited to the HTTP protocol. Some haven’t been updated for years. Most aren’t flexible enough to provide parametrization, correlation, assertions and distributed testing capabilities. Given the challenges most of us face today, I would whittle this down to a short list of the following four options that we will review in this post:
The Grinder Gatling Tsung JMeter Locust
We’ll cover the main features of each tool, show a simple load-test scenario, and display sample reports. At the end, you'll find a comparison matrix to help you decide which tool is best for your project.
Just as a short note, if you are looking for a way to automate these open source tools, BlazeMeter created Taurus, our own open source test automation tool that exends and abstracts most of the above tools (as well as Selenium), and helps to overcome various challenges. Taurus provides a simple way to create, run and analyze performance tests. Make sure to check it out.
THE TEST SCENARIO AND INFRASTRUCTURE
For our comparisons we will use a simple a HTTP GET request from 20 threads with 100,000 iterations. Each tool will send requests as fast as it can.
The server side (application under test):
CPU: 4x Xeon L5520 @ 2.27 GHz
RAM: 8GB
OS: Microsoft Windows Server 2008 R2 x64
Application Server: IIS 7.5.7600.16385
The client side (load generator):
CPU: 4x Xeon L5520 @ 2.27 GHz
RAM: 4GB
OS: Ubuntu Server 12.04 64-bit
Load test tools:
The Grinder 3.11
Gatling 2.3.0
Tsung 1.7.0
JMeter 3.3
Locust 0.8
1. THE GRINDER
The Grinder is a free Java-based load-testing framework available under a BSD-style open-source license. It was developed by Paco Gomez and is maintained by Philip Aston. Over the years, the community has also contributed many improvements, fixes, and translations. The Grinder consists of:
The Grinder Console - This GUI application controls various Grinder agents and monitors results in real time. The console can be used as a basic interactive development environment (IDE) for editing or developing test suites. Grinder Agents - Each of these are headless load generators can have a number of workers to create the load
Key Features of The Grinder:
TCP proxy to record network activity into the Grinder test script Distributed testing that scales with an the increasing number of agent instances Power of Python or Closure, combined with any Java API, for test script creation or modification Flexible parameterization, which includes creating test data on the fly and the ability to use external data sources like files and databases Post-processing and assertion with full access to test results for correlation and content verification Support of multiple protocols
The Grinder Console Running a Sample Test
Increase imageopen source performance testing tools comparison
Grinder Test Results:
Increase imagetop load testing tools comparison
GATLING
The Gatling Project is another free and open source performance testing tool, primarily developed and maintained by Stephane Landelle. Gatling has a basic GUI that's limited to test recorder only. However, the tests can be developed in easily readable/writable domain-specific language (DSL).
Key Features of Gatling:
HTTP Recorder
An expressive self-explanatory DSL for test development
Scala-based
Production of higher load using an asynchronous non-blocking approach
Full support of HTTP(S) protocols and can also be used for JDBC and JMS load testing
Multiple input sources for data-driven tests
Powerful and flexible validation and assertions system
Comprehensive informative load reports
The Gatling Recorder Window:
Increase imagetop performance testing tools comparison
An Example of a Gatling Report for a Load Scenario
Increase imageA Comparison of Open Source Load Testing Tools
If you are interested in more information about Gatling, view our on-demand webcast Load Testing at Scale Using Gatling and Taurus.
TSUNG
Tsung (previously known as IDX-Tsunami) is the only non-Java-based open-source performance-testing tool in this review. Tsung relies on Erlang, so you’ll need to have it installed (for Debian/Ubuntu, it’s as simple as "apt-get install erlang”). Tsung was launched in 2001 by Nicolas Niclausse, who originally implemented a distributed-load-testing solution for Jabber (XMPP). Several months later, support for more protocols was added and, in 2003, Tsung was able to perform HTTP Protocol load testing. Today, it’s a fully functional performance-testing solution with the support of modern protocols like websockets, authentication systems, and databases.
Key Features of Tsung:
Inherently distributed design Underlying multithreaded-oriented Erlang architecture simulates thousands of virtual users on mid-range developer machines Support of multiple protocols A test recorder that supports HTTP and Postgres Metrics for operating systems for both the load generator and application under test can be collected via several protocols Dynamic scenarios and mixed behaviors. Flexible load scenarios let you define and combine any number of load patterns in a single test Post processing and correlation External data sources for data driven testing Embedded easily-readable load reports that can be collected and visualized during load
Tsung doesn’t provide a GUI for test development or execution. So you’lll have to live with shell scripts, which are:
Tsung-recorder, a bash script that records a utility capable of capturing HTTP and Postgres requests and that creates a Tsung config file from them
Tsung, a main bash control script to start/stop/debug and view test status
Tsung_stats.pl, a Perl script to generate HTML statistical and graphical reports. It requires the gnuplot and Perl Template library. For Debian/Ubuntu, the commands are:
apt-get install gnuplo
apt-get install libtemplate-perl
The main tsung script invocation produces the following output:
Increase imagehow to choose your load testing tools
Running the test:
Increase imagechoose the right load testing tool
Querying the current test status:
Increase imageopen source load testing tools a guide
Generating the statistics report with graphs can be done via the tsung_stats.pl script:
Increase imagewhich load testing tool to choose
Open report.html with your favorite browser to get the load report. A sample report for a demo scenario is provided below:
A Tsung Statistical Report
Increase imageOpen Source Load Testing Tool Review
A Tsung Graphical Report
Increase imageweb performance testing top tols to consider
APACHE JMETER
Apache JMeter™ is the only desktop application in this review. It has a user-friendly GUI, making test development and debugging much easier. The earliest version of JMeter available for download is dated March 9, 2001. Since then, JMeter has been widely adopted and is now a popular open-source alternative to proprietary solutions like Silk Performer and LoadRunner. JMeter has a modular structure, in which the core is extended by plugins. This means that all implemented protocols and features are plugins that have been developed by the Apache Software Foundation or online contributors.
Key Features of JMeter:
Cross-platform. JMeter can run on any operating system with Java
Scalable. When you need a higher load than a single machine can create, JMeter can execute in a distributed mode, meaning one master JMeter machine controls a number of remote hosts.
Multi-protocol support. The following protocols are all supported out-of-the-box: HTTP, SMTP, POP3, LDAP, JDBC, FTP, JMS, SOAP, TCP
Multiple implementations of pre- and post-processors around sampler. This provides advanced setup, teardown parametrization, and correlation capabilities
Various assertions to define criteria
Multiple built-in and external listeners to visualize and analyze performance test results
Integration with major build and continuous integration systems, making JMeter performance tests part of the full software development life cycle
The JMeter Application With an Aggregated Report on the Load Scenario:
Increase imageTop Open-Source Load Testing Tools - Load Testing
LOCUST
Locust in a Python-based open source framework, which enables writing performance scripts in pure Python language. The main uniqueness of this framework is that it was developed by developers and for developers. The main Locust targets are web applications and web-based services, however, if you are comfortable with Python scripting, you can test almost anything you want. In addition to that, it is worth mentioning that Locust has a completely different way to simulate users, which is fully based on the events approach and gevent coroutine as the backbone for this process. This process allows simulating thousands of users even on a regular laptop, and executing even very complex scenarios that have many steps.
Locust Key Features:
Cross-platform, because Python can be run on any OS High scalability on regular machines due to events based implementation Power assertion ability, limited only by your own Python knowledge (you can read this article to learn more about Locust assertions) Nice web-based load monitoring Code-based scripts implementation that is handy to use with version control (Git, SVN...) Scalability, because you can run Locust distributed with many agents The ability to test almost anything with the implementation of custom samplers based on pure Python code
Basic test script example:
from locust import HttpLocust, TaskSet, task
class SimpleLocustTest(TaskSet):
@task
def get_something(self):
self.client.get("/")
class LocustTests(HttpLocust):
task_set = SimpleLocustTest
You can run the script by using this command:
locust -f locustfile.py --host=http://192.168.1.170:8080
Increase imagecomparing open source load testing tools locust
After the script execution, you will find the detailed reporting on http://localhost:8089/:
Increase imageload testing tools comparison locust
HOW BLAZEMETER LOAD TESTING CLOUD COMPLEMENTS AND STRENGTHENS JMETER
While Apache JMeter represents a strong and compelling way to perform load testing, of course, we recommend supplementing that tool with BlazeMeter Load Testing Cloud, which lets you simulate up to 1 million users in a single developer-friendly, self-service platform. With BlazeMeter, you can test the performance of any mobile app, website, or API in under 10 minutes. Here’s why we think the BlazeMeter/JMeter combination is attractive to developers:
• Simple Scalability – It’s easy to create large-scale JMeter tests. You can run far larger loads far more easily with BlazeMeter than you could with an in-house lab. • Rapid-Start Deployment – BlazeMeter’s recorder helps you get started with JMeter right away, and BlazeMeter also provides complete tutorials and tips. • Web-Based Interactive Reports – You can easily share results across distributed teams and overcome the limitations of JMeter’s standalone UI. • Built-In Intelligence – The BlazeMeter Cloud provides on-demand geographic distribution of load generation, including built-in CDN-aware testing.
Increase imageblazemeter, load testing tools guide
THE GRINDER, GATLING, TSUNG, LOCUST AND JMETER PUT TO THE TEST
Let’s compare the load test results of these tools with the following metrics:
Average Response Time (ms)
Average Throughput (requests/second)
Total Test Execution Time (minutes)
First, let’s look at the average response and total test execution times:
Increase imageaverage response and total test execution times for open source load testing tools
Increase imagecomparing open source load testing tools to improve performance testing
As shown in the graphs, Locust has the fastest response times with the highest average throughout, followed by JMeter, Tsung and Gatling. The Grinder has the slowest times with the lowest average throughput.
FEATURES COMPARISON TABLE
And finally, here’s a comparison table of the key features offered by each testing tool:
Feature The Grinder Gatling Tsung JMeter Locust
OS Any Any Linux/Unix Any Any
GUI Console Only Recorder Only No Full No
Test Recorder TCP (including HTTP) HTTP HTTP, Postgres HTTP No
Test Language Python, Clojure Scala XML XML Python
Extension Language Python, Clojure Scala Erlang Java, Beanshell, Javascript, Jexl Python
Load Reports Console HTML HTML CSV, XML, Embedded Tables, Graphs, Plugins HTML Protocols
HTTP
SOAP
JDBC
POP3
SMTP
LDAP
JMS
HTTP
JDBC
JMS
HTTP
WebDAV
Postgres
MySQL
XMPP
WebSocket
AMQP
MQTT
LDAP
HTTP
FTP
JDBC
SOAP
LDAP
TCP
JMS
SMTP
POP3
IMAP HTTP Host monitoring No No Yes Yes with PerfMon plugin No Limitations
Python knowledge required for test development & editing.
Reports are very plain and brief.
Limited support of protocols.
Scala-based DSL language knowlegde required.
Does not scale. Tested and supported only on Linux systems. Bundled reporting isn’t easy to interpret. Python knowledge required for test development & editing.
RESOURCES
Want to find out more about these tools? Log onto the websites below - or post a comment here and I’ll do my best to answer!
The Grinder - http://grinder.sourceforge.net/
Gatling - http://gatling.io/
Tsung - http://tsung.erlang-projects.org/
Locust - https://locust.io
JMeter
Home Page: http://jmeter.apache.org/
JMeter Plugins: http://jmeter-plugins.org/
Blazemeter’s Plugin for JMeter: http://blazemeter.com/blazemeters-plug-jmeter
On a Final Note...
As I mentioned at the start, you might also want to read more about Taurus. When it comes to performance testing, a lot of these are really great...but not perfect. Automation and integration with other systems can be a pain, and the tool itself comes with a steep learning curve. Taurus is an open source test automation tool that provides a simple way to create, run and analyze performance tests.