How to test reports for the load. Load Meadme Test Service. How to determine a reasonable delay rate

Stress Testing

Stress Testing (eng. Load Testing.) - definition or collecting performance indicators and response time of a software and technical system or device in response to an external request to establish compliance with the requirements for this system (device).

To study the response time of the system in high or peak loads, stress testing is made, in which the load generated by the system exceeds the normal scenarios of its use. There is no clear boundary between load and stress testing, but these concepts are not worth mixing, as these types of testing respond to different business issues and use a different methodology.

Load Software Testing

Term stress Testing Can be used in various values \u200b\u200bin a professional software test environment. In general, it means the practice of modeling the expected use of the application using the emulation of several users at the same time. Thus, such testing is most suitable for multi-user systems, more often - using client-server architecture (for example, web servers). However, other types of software systems can be tested in a similar way. For example, a text or graphic editor can be made to read a very large document; And the financial package is to generate a report based on data for several years. The most adequately designed load test gives more accurate results.

The main goal of load testing is to, creating a load-defined load (for example, through virtual users) and, usually using identical software and hardware, monitor the performance of the system.

Example 1: The web service with the functionality of the buyer's basket is designed for 100 simultaneously working users who follow a certain scenario (given actions in the specified proportions): 25 users view the goods and come out of the system. 25 users add goods to the basket, draw it out and leave the system. 25 users use the return function and exit the system. 25 users are included in the system and do not show any activity. In this case, load testing should emulate the above-described typical work scenario with a web service in order to make sure that the system is ready for operation. At the same time, the performance of the system performance as a whole or each node of the system may be removed for analysis.

In the ideal case, as criteria for the success of load testing, the requirements for system performance, which are formulated and documented at the stage of developing functional system requirements prior to the start of programming the main architectural solutions. However, it often happens that such requirements were not clearly formulated or not formulated at all. In this case, the first load testing will be trial (exploratory Load Testing) And based on reasonable assumptions about the expected load and consumption of the hardware of resources.

One of the optimal approaches to using load testing for measuring the system performance is testing at the stage of early development. Load testing At the first stages of the preparedness of the architectural solution in order to determine its spength is called "Proof-of-Concept" testing.

Basic Principles of Load Testing

Below are some experimental facts summarized in the principles used in testing performance as a whole and applicable to any type of performance testing (in particular, to load testing).

1. Uniqueness of requests

Even having formed a realistic scenario with a system based on its use statistics, it is necessary to understand that there will always be exceptions to this scenario.

Illustration of various distribution dispersions for the time of execution of X and Y requests.

When Example 1. it can be a user adding to different from all the others unique pages Web service.

2. System response time

In general, the response time of the system is subject to the function of normal distribution.

In particular, this means that having a sufficient number of measurements, one can determine the likelihood with which the response of the system to the request will fall into one or another time interval.

3. The dependence of the response time of the system on the degree of distribution of this system.

Dispersion of the normal distribution of the response time of the system to the request is proportional to the ratio of the number of the system nodes, parallel to process such requests and the number of requests per node.

That is, the number of requests for each system number and the number of nodes, each of which adds some of which adds a random delay value to the scatter of the system response time.

4. System response time scatter

From statements 1, 2 and 3, it can also be concluded that with a sufficiently large number of measurements of the query processing time in any system, there will always be requests, the processing time of which exceeds the maximums defined in the requirements; Moreover, the more the total time of the experiment, the higher the new maxima will be.

This fact must be taken into account when forming the requirements for system performance, as well as during regular load testing.

5. Accuracy of reproduction of load profiles

The required accuracy of playing load profiles is more expensive than the more component contains the system.

It is often impossible to take into account all the aspects of the load profile for complex systems, as the more difficult the system, the longer it is spent on designing, programming and supporting the adequate load profile for it, which is not always a necessity. Optimal approach In this case, it is balancing between the value of the development of the test and the coating of the functionality of the system, as a result of which assumptions about the effect on the overall productivity of one or another part of the system being tested.

Toolkit for performance testing

It should be noted that for most types of performance testing uses the same toolkit that can perform typical tasks.

There is a common erroneous understanding that the tools for load testing system are the tools are the same on the principle of recording and playback as the tools for automation of regression testing. Load testing tools work at the protocol level, while instruments for automation of regression testing work at the level of graphical user interface objects.

There are various tools for detecting and researching problems in various system nodes. All system nodes can be classified as follows:

• Application,
• Database,
• Net,
• Treatment on the client side,
• Load balancing.

It should also be noted the appearance of network business-to-business (B2B) applications using the service level agreement (or SLA, SERVICE LEVEL AGREEMENT). The increasing popularity of B2B applications led to the fact that more and more applications go to a service-oriented architecture, in which case the exchange of information occurs without the participation of web browsers. An example of such an interaction can be a tourist services bureau that requests information about a certain flight between St. Petersburg and Omsk, while the airline is obliged to provide an answer within 5 seconds. Often, violation of the SLA treaty threatens a large fine.

The most popular tools for load testing are presented below.

BY	Producer name	Comments
OpenSta.	"Open System Testing Architecture"	Free Software for Load / Stress Testing, Licensed GNU GPL. Uses the distributed application architecture based on CORBA. Available version under Windows, although there are compatibility issues with Windows Vista.. Support terminated in 2007.
IBM Rational Performance Tester	IBM.	ECLIPSE-based software development environment that allows you to create a load of large volumes and measure the response time for applications with client-server architecture. Requires licensing.
Jmeter.	Open project Apache Jakarta Project	Based on Java cross-platform toolkit, which allows load tests using JDBC / FTP / LDAP / SOAP / JMS / POP3 / HTTP / TCP connections. Makes it possible to create a large number of requests from different computers and control the process from one of them.
Hp Loadrunner.	HP.	Tool for load testing, originally designed to emulate a large number of parallel users. It can also be used for Unit- or Integration Testing.
Silkperformer	Micro Focus.
Visual Studio Load Test	Microsoft.	Visual Studio provides a performance test tool including Load / Unit Testing
LoadComplete.	SmartBear.

Main indicators (metrics) performance

One of the results obtained with load testing and used in the future for analysis are the performance performance indicators. The main of them is disassembled below.

1. Consumption of the resources of the central processor (CPU,%)

The metric, showing how much time from the specified defined interval was spent by the processor on calculating for the selected process. IN modern systems An important factor is the ability of the process to work in several threads, in order for the processor to make calculations in parallel. An analysis of the processor resource consumption history may explain the impact on the overall performance of the process of processed data streams, application configurations and operating system, multipleness of calculations, and other factors.

2. Consumption random access memory (Memory usage, MB)

Metric, showing the amount of memory used by the application. Used memory can be divided into three categories:

• Virtual - the volume of the virtual address space that uses the processor. This volume does not necessarily imply, use the appropriate disk space or RAM. The virtual space of course and the process may be limited to the ability to download the necessary libraries.
• Private - the volume of the address space occupied by the processor and not shared with other processes.
• Working SET - a set of memory pages recently used by the process. In the case when free memory is enough, the pages remain in the set, even if they are not used. In the case when there is little free memory, the pages used are removed.

When the application is running, the memory is filled with references to objects that, in case of non-use, can be cleaned by a special automatic process called "garbage collector" (English. Garbage Collector). The time spent in a memory cleaning processor in such a way can be significant, in the case when the process took all available memory (in Java - the so-called "permanent Full GC") or when the process is highlighted large memory volumes that need cleaning. For a time required to clean the memory, process access to the allocated memory pages can be blocked, which can affect the final processing time by this data process.

3. Consumption of network resources

This metric is not directly related to the performance of the application, however, its indicators may indicate the performance of the system as a whole.

Example 3: The server application processing the user's request, returns to it a video stream using a 2 megabit network channel. The requirement states that the server must handle 5 user requests at the same time. Load testing has shown that the server effectively can provide data for only 4 users simultaneously, since multimedia stream has a bit rate of 500 kilobit. It is obvious that the provision of this thread 5 users are simultaneously impossible due to excess of the network channel bandwidth, which means that the system does not satisfy the specified performance requirements, although the consumption of processor and memory resources can be low.

4. Working with disk subsystem (I / O Wait)

Working with the disk subsystem can significantly influence the performance of the system, therefore the collection of disk working statistics can help identify bottlenecks in this area. A large number of readings or records can lead to a standing processor waiting for data processing from the disk and as a result, an increase in CPU consumption and increasing the response time.

5. Query time execution time (Request Response Time, MS)

The execution time of the application application remains one of the most important performance indicators of the system or application. This time can be measured on the server side, as an indicator of the time required by the server part for processing the request; And on the client, as an indicator of the full time, which is required for serialization / deserialization, shipping and processing of the request. It should be noted that not each application for performance testing can measure both of these time.

see also

Links

• Playground of services for testing sites and software (Rus.)
• The portal of testing specialists and quality assurance software (Rus) - the project is devoted to testing and improving the quality of software.
• Tester Knowledge Base (Rus.) - Bagtrakeners, Automated Testing, Load Testing, Usability Testing, Community, Printing, Books
• Automation of load testing (rus.)
• Notes on load testing (rus.)

Literature

• Lisa Crispin, Janet Gregory Flexible Testing: Practical guide for testers and flexible commands \u003d Agile Testing: A Practical Guide for Testers and Agile Teams. - m .: "Williams", 2010. - 464 p. - (Addison-Wesley Signature Series). - 1000 copies. - ISBN 978-5-8459-1625-9

Wikimedia Foundation. 2010.

Passing the web server in everyday operation, you need to be sure that he
will withstand the planned load. Only creating conditions approximate to combat,
It is possible to evaluate whether the power of the system is sufficient correctly
applications involved in creating web content and other factors affecting
Web server work. In this situation will come to the rescue special tools,
Which will help to give a qualitative and quantitative evaluation of the work. as
Web nodes in general and individual components.

Everything goes according to plan

Before rushing into battle, at first it should be sorted out what we want
Get as a result of testing. After all, check, like any other work,
requires pre-training. With incorrectly formulated task
may result in results that will not fully reflect the real
status. Based on the intended load of the web server, it is necessary
Determine the test criteria. Establish what will be considered as successful
And what is the unacceptable service of the service (for example, a response time, loading
Servers). Three test options distinguish:

• Load-Testing - system performance is determined
  With some strictly given in advance (planned, working) load.
• Sustainability (Stress) - Used to verify system parameters
  In the abnormal and extreme conditions, the main task during this test is
  Try to break the work of the system. Allows you to determine the minimum
  Required values \u200b\u200bof system resources for application operation, evaluate
  The limit features of the system and the factors limiting these opportunities.
  The ability of the system to preserve the integrity of data is also determined.
  The emergence of freelance emergency situations.
• Performance (Performance) - Comprehensive check, including
  Previous two tests are designed to evaluate all system indicators.

Test Result - Maximum Number of Users, which can
At the same time gain access to the website, the number of requests processed
Appendix, or server response time. Based on the result,
Web master and network administrator (in server work and other
Network components, routers, firewall, caching and proxy server, base
data, etc.) will be able to identify bottlenecks in advance arising from
unbalanced component work and correct the situation before
Include the system into real work.

During testing simultaneous work of several hundred
or thousands of visitors. For greater truthfulness, each of the virtual
users can "walk" on the site on the individual scenario and have personal
parameters. Also in the process of testing you can simulate short-term peaks
loads when the number of visitors is jumpingly increasing, which is very
Actual for sites with uneven audience. So to fully hold
Testing, you need to know:

• how many visitors are scheduled to take on average, in peak load,
  peak load time;
• can several users have the same IP address and / or
  Login: Password;
• average number of pages viewed by one user, is there
  Differences in behavior between registered and anonymous users,
  quantitative relationship between such users visited by pages and
  the time of finding the user on the node;
• availability of dynamic pages and pages changed within a certain
  period, and how often it happens;
• lit is involved email, for example, to confirm the authority
  user;
• what more additional information is used to check the status
  User (Cookies);
• whether the user's authority confirmation is required by a third-party organization
  or remote server (for example, credit card number), and whether
  presented information for testing;
• available channel bandwidth, its average width for one
  user;
• can the work of several users call the collision;
• whether protected HTTPS connection is used;
• is java applets, streaming media, special plugins that
  Required from the client side to support them;
• whether the caching of pages is used;
• planned technical events that may affect work
  Servers and time of them (synchronization, archiving, etc.).

Any of these parameters may affect the final result. Not necessary
All checks include in one test, you can break first task for subtasks.
For example, check base system (servers: web, applications, databases) and
Check individual modules (servlets, scripts, etc., for example, check
Authentication with a large number of users). As a result,
Testing is issued graphics of three types: linear, non-linear and saturation. IN
first case with an increase in load response time (i.e. processing) remains
constant. With a further increase in the load, the response time also increases
(almost linear), and finally there is a situation like a DOS attack when time
The response is infinitely increasing. Now that the action plan is ready, go to
Brief overview of utilities that will help to implement it. Let's start with free.

Open Systems Testing Architecture

OpenSta. (www.opensta.org)
- More than an application for tests, this is an open architecture, designed
around open standards. The project was established in 2001 by a group of companies Cyrano.,
which supported the commercial version of the product, but Cyrano collapsed, and now
OpenSta. applies as an application with open source under license
GNU GPL works in Windows NT 4.0SP5 / 2000 / XP. To work requires Microsoft Data
Access Components (MDAC), which can be downloaded to the corporation site.

Current toolkit allows you to carry out load test http / https
Services, although his architecture is capable of greater. OpenSta. Allows
Create test scripts on the SCL specialized language (Script Control
Language). To simplify creating and editing scenarios used
Special Script Modeler tool. Choose Tools - Canonicalize URL,
Web browser starts. Just walk on the site, collecting links that will
Saved to the script. All request parameters can be edited, possible
Variable substitution. The structure of the test and headlines will be displayed in tabs.
In the panel on the left. Tests are conveniently combined into sets. Proxy settings are set in
The script itself, so you can specify multiple servers. Implemented opportunity
Organization of distributed testing, which increases realistic, or when
From one computer, you can't load a powerful server. Each of the cars such
Systems can perform their task group, and the repository HOST collects
and storing results. After installation on each test system
The name server is launched, the work of which is mandatory. Supported
user authentication on the web resource and establishing connections for
SSL protocol. The operating parameters of the loaded system can be monitored with
using SNMP I. windows NT. Test results including time
Responses, number of passed bytes per second, response codes for each request
And the number of errors are displayed in the form of tables and graphs. Using big
Filter numbers Allows you to select the necessary results. The result is possible
Export to the CSV file. The conclusion of reports is somewhat limited,
But on the links on the site you can find scripts and plugins that are simplifying, including
Analysis of the information received.

Apache Jmeter.

Apache Jmeter. (jakarta.apache.org/jmeter)
is a Java open source application, designed for load
testing not only web applications and their separate components (scripts,
servlets, java objects, etc.), but also FTP servers, databases (with
using JDBC) and network. Functionality is expanding with plugins.
SSL is supported (through Java Secure Sockets Extension). Perhaps holding
tests both using the graphical interface and from command line.
Using Java implies cross-platform, so Jmeter.
Confidently works in various * Nix-systems, in Windows from 98 and some others
OS. Apply under Apache License.

IN Jmeter. Provides virtual authorization mechanisms
users, supported custom sessions, templates, caching and
Subsequent offline analysis of test results, functions allow you to form
The following request, based on server response to the previous one. I have an opportunity
carry out distributed tests. In this case, one of the computers is
server (bin / jmeter-server.bat) that manages clients and collects
Final information.

It is enough to run apachejmeter.jar or in the JMeter.bat console
(Windows) or jmeter.sh (* NIX).

Jmeter. has a built-in proxy server that is designed to record
sessions, but you can use external. Before testing you need
Create a test plan describing a series of tasks you want to perform
Jmeter.. It must contain one or more stream groups (Thread
Groups) and other elements:

• Logic controllers (Logic Controllers);
• Sample Controllers (Sample Generation Controllers);
• Listeners;
• Timers (Timers);
• Compliance (assertions);
• Configuration Elements (Configuration Elements).

First of all, add a stream group (Edit - Add - Thread Group). In her
settings indicate the name, the number of triggers, that is
Virtual Users (Number of Threads), delay time between launch
streams (RAMP-Up Period), the number of task execution cycles (Loop Count)
Here you can determine the execution of a schedule task (Sheduler). Further,
Slitted to the created group, you must add a sample of the query (SAMPLER) by selecting
Its from the list. For load testing or performance testing
Servers just select HTTP Request (Add -Sampler - HTTP Request). Here
Specify the name, IP address and port of the web server, protocol, data transfer method
(GET, POST), redirect parameters, file transfer to the server. Customize I.
Click on RUN. The result output is carried out using listeners, each
In its own way displays the result. For example, Aggregate Graph displays total
Test results in the form of a table and graphics.

Free products, alas, ended, now a couple of commercial solutions.

WAPT - Web Application Testing

WAPT (www.loadTestingTool.com)
Allows you to experience the stability of the website and other applications using
Web interface, to real loads. Developed by the Novosibirsk company
SoftLogica LLC. This is one of the easiest to use review programs. For
The simple test does not even need to look into the documentation, the interface
Simple, but not localized. Works under Windows from 98, supported
and Vista. For check WAPT can create many virtual
Users, each with individual parameters. Multiple supported
Types of authentication and cookies. The script allows you to change the delays between
requests and dynamically generate some test parameters,
Misses the behavior of real users as much as possible. In demand
Various options for HTTP header can be substituted, in the settings you can
Specify page encoding. User-Agent, X-Forwarded-for, IP parameters are indicated.
In the Script Settings. The values \u200b\u200bof the query parameters can be calculated
in several ways, including those defined by the server response to the previous
Request using variables and functions. Supported work on protected
HTTPS protocol (and all types of proxy servers). Created scenarios stored in
The XML format file can be reused. In addition to standard Performance and
Stress, there are several other tests that allow you to determine
Maximum number of users and test the server under load in
For a long period.

For testing, you must select New - Scenario, as a result
The test wizard will start. In the first step, the type of test is indicated and then in
Each window is filled by the parameters of the future test. Here you can specify
Fixed number of virtual users, or stepwise increase
indicating the minimum and maximum number and time interval, set
Timer testing. The next step sets the time between clicks (THINK
Time), the connection speed, indicates the range of IP addresses that will be
Used by virtual users. Pressing the IP ADRESS LIST will allow
Make a list of such addresses. The HTTP parameter User-Agent is also set.
The proxy emulation is included. If required to have virtual users
individual settings, in the next step of the wizard for each of them
You must create your profile by clicking New or downloadable saved. In the following
The program window needs to set the profile parameters.

After pressing the button is ready, the script is saved. Now to point to
Test object, create a NEW profile - Profile and fill all the parameters on
tabs. Here are also available for editing some parameters specified
Early with the master. Also indicate the loading of drawings by virtual
The user, authentication parameters, use cookies and others.
On the Recorder tab, specify the address of the site, the availability of which can be immediately
Check by clicking GO. At the same time will be a request for launching Recorder, which
will track the visited pages and write the URI (they will be displayed in
Panel on the left). When all the information is collected, click Run Test. Detailed reports
In the form of graphics are displayed in the course of the test, at the end it will be
HTML page formed. As a result, you can get information about the time.
server response with increasing load, by the number of errors transmitted and
accepted byte, etc.

NeoLoad.

NeoLoad. (www.neotys.com)
- Another system that allows you to carry out load testing
Web applications. Written on Java, works on computers working under
Manage Windows NT / 2000 / XP, Linux and Solaris. In the report you can get
Detailed information on each downloaded file. Neoload is very convenient for
Estimates of the work of individual components (AJAX, PHP, ASP, CGI, Flash, applets and
etc.). It is possible to set the delay time between requests (ThinkTime) globally
And individually for each page. Testing is carried out as with
using a very convenient graphic shell, and using the command
Rows (using a pre-prepared XML file). Supports work S.
HTTPS protocol, with http and https proxy, Basic web authentication and cookies,
Automatically defining the data during the script recording, and then loses in
Test time. To work with various profiles to register users
Variables can be used. When conducting the test you can use
Additional Monitors (SNMP, WebLogic, WebSphere, Rstat and Windows, Linux,
Solaris) to monitor and system parameters on which it works
Web server.

With help NeoLoad. Distributed tests can be carried out. One of
Computers is a controller, generators are installed on the rest.
Loads (LoadGenerator). The controller distributes the load between LoadGenerator and
Collects statistics.

Work with virtual users is very convenient. Users
have customized settings, then they are combined into populations (should
to be created at least one populations), in populations you can set a common
Behavior (for example, 40% of users of the population are visited by dynamic resources,
20% read news). Virtual users can have an individual
IP address, bandwidth and your test script.

The script of the future test is very simple. Run the application (when
The first launch will need to enter the registration key, the 30-day version after
registration will be shipped by mail), select New Project, enter the name
Project. After that, a small tip of further will be shown.
actions, pressing Start Recording will launch a web browser, all movements will be
recorded. After finishing, press Stop Recording or close the browser.
A wizard will be launched that will help create virtual users and
Produce automatic search dynamic parameters in recorded pages,
Exhibit the average value of ThinkTime. Page Components (HTML, Images, CSS)
Save separately. To get the result, you need to go through three steps:

• Design - setting up the project, here are three tabs. The repository is specified
  Web Pages and Query Settings, Virtual User is created in Virtual User
  Users specify the URL that they must "visit" and additional
  Conditions from the left tab of the Actions fields. In populations - tasks each of the groups
  users. The following steps may be selected in Actions: Delay
  (Setting the delay), loop (Repeat), While (cycle), if ... then ... else
  (Condition), Container and Random Container (Group Actions), Try ... Catch
  (error handling), Stop Virtual User (Virtual
  user).
• Runtime - The test parameters are indicated, the test is performed. Here B.
  Separate tabs in the course of the test statistics are displayed.
• Results - responsible for the withdrawal of various statistics in the form of tables and graphs.

And besides common values, using the filter system can be selected
Information for any parameter. If desired, the project is preserved for repeated
use. Among the presented products the ability to compare results
Test is only NeoLoad..

Using load testing utilities, you can get information about
Web service work, take the necessary measures to eliminate identified
disadvantages and guarantee the required performance.

Products from Microsoft.

Microsoft offers as many as two products that allow
Test web server under load. it Microsoft Application Stress
Tool. and WEB CAPACITY ANALYSIS TOOL. The first is distributed as
separate product and has graphic interface. The second is part of
Tool set Internet Information Services 6.0 Resource Kit Tools,
Works from the command line. Mast. more visual, in the creation of the test
A simple test wizard will help, work with cookies, adjustment
Loads on different URLs. Test script can be created manually or
Recorded using a web browser and, if necessary, edited. IN Wast.
The load level (Stress LEVEL) is adjusted by setting the amount of threads,
Requests to the server, and the number of virtual users
calculated as a product of the number of threads on the number of sockets, open each of
threads. At the end of the test, we get a simple report in the text form in which
Information is given in the number of processed requests per unit of time, average
Delay time, data transfer speeds to server and from server, quantity
Errors, etc. The report can be exported to the CSV file. No opportunities for
statistical processing is not provided, that is, it is possible only
Estimate work under certain conditions.

Stress Testing - Definition or collecting performance indicators and response time of a software and technical system or device in response to an external request to establish compliance with the requirements for this system (device). (Wikipedia)

Why load testing is performed:

• Checking and optimizing equipment configuration, virtual machines, server software;
• Evaluation maximum performancewhich is able to withstand a project with typical load scenarios on available resources;
• The impact of the project modules for performance, peak load processing scenarios;
• Estimation of stability at maximum loads during 24-hour tests, taking into account external factors (imports, backup, etc.);
• Detection of configuration restrictions, defining methods for further scaling and optimization.

In general, exist great amount Tools for load testing, both OpenSource and commercial. Let us dwell on the most commonly used and tell about their main opportunities.

Apache Http Server Benchmarking Tool

Free

Official site

The most commonly used, because Apache is part of Apache.

AB: //] HostName [: Port] / Path

where the basic necessary options:

C. concurrency. - the number of simultaneous queries to the server (default 1);
-N. requests. - Total queries (default 1).

As a result, we receive such a report:

Concurrency Level: 10 Time Taken for Tests: 0.984 Seconds Complete Requests: 100 Failed Requests: 0 Write Errors: 0 Total Transferred: 3725507 Bytes Html Transferred: 3664100 Bytes Requests Per Second: 101.60 [# / Sec] (Mean) Time Per Request: 98.424 (MEAN) Time Per Request: 9.842 (Mean, Across All Concurrent Requests) Transfer Rate: 3696.43 Received Connection Times (MS) Min Mean [+/- SD] Median Max Connect: 1 2 3.6 1 23 Processing: 63 94 21.5 90 173 Waiting: 57 89 21.6 84 166 Total: 64 96 21.5 92 174

pros:

• There are everywhere where there is Apache;
• Does not require any additional configuration;
• Very simple tool.

Minuses:

• Very simple tool;
• Only the web server performance tests: only one URL is polls, does not support the load scenarios, it is impossible to simulate a user load and evaluate the performance of the project from all sides - both from the point of view of infrastructure and in terms of development.

Joe Dog Siege

Free

Official site .

A little more difficult aB And the necessary tasks do much better.

The scenario file is set URLs and testing requests. If the script is large by volume, then you can make all requests in a separate file and in the command to specify this file when testing:

# cat urls.txt # urls file for siege # - http://www.bitrix24.ru/ http://www.bitrix24.ru/support/forum/forum1/topic3469/?pagen_1\u003d2 http: // www .bitrix24.ru / Register / Reg.php Post Domain \u003d Test & Login \u003d Login http://www.bitrix24.ru/search/ post

The command indicates the number of users -s, the number of repetitions -r and the delay between the hits -d.

The result can be displayed in the log file or immediately into the console in real time:

HTTP / 1.1 200 0.44 secs: 12090 bytes \u003d\u003d\u003e Get / http / 1.1 200 0.85 secs: 29316 bytes \u003d\u003d\u003e Get / support / Forum / Forum1 / Http / 1.1 200 0.85 secs: 29635 bytes \u003d\u003d\u003e Get / support / Forum / Forum1 / http / 1.1 200 0.34 secs: 12087 bytes \u003d\u003d\u003e Get / [...] Done. Transactions: 100 hits Availability: 100.00% Elapsed time: 12.66 secs Data transferred: 1.99 MB Response time: 0.64 secs Transaction rate: 7.90 trans / sec Throughput: 0.16 MB / sec Concurrency: 5.02 Successful transactions: 100 Failed transactions: 0 Longest transaction: 1.06 SHORTEST TRANSACTION: 0.31

You can also take from the ACCESS Log of the URL Web Server, which we went to real users and emulate the load of real users.

pros:

• Multi-threaded;
• You can specify both the number of requests and duration (time) of testing - i.e. you can emulate user load;
• Supports the simplest scenarios

Minuses:

• Resource-intensive;
• Few statistical data and does not emulate such custom scenarios as limiting the speed of user requests;
• It is not suitable for serious and large-scale testing in hundreds and thousands of streams, because he in itself is a resource-intensive, and on a large number of requests and flows a lot loads the system.

Apache Jmeter.

Free

Official site

Main features:

• Written in java;
• HTTP, HTTPS, SOAP, Database Via JDBC, LDAP, SMTP (S), POP3 (S), IMAP (S);
• Console and GUI;
• Distributed testing;
• Test plan - XML \u200b\u200bfile;
• Can handle the web server log as a test plan;
• Visualization of results in the GUI.

The results are displayed in graphical form:

pros:

• Cross-platform, because written in java;
• Very flexible, a lot of protocols are used, not only a web server, but also the base;
• Controlled through the console and the GUI interface;
• Using direct logs of Apache and NGINX web server as a script with the ability to vary the load on these profiles;
• A fairly convenient and powerful tool.

Minuses:

• Resource-intensive;
• On long and heavy tests often drops for various reasons;
• Stable operation depends on the environment and configuration of the server.

Tsung

Free

Official site

Main features:

• Written in Erlang;
• HTTP, WebDAV, SOAP, PostgreSQL, MYSQL, LDAP, Jabber / XMPP;
• Console (GUI through side plugin);
• Distributed testing (millions of users);
• Test phases;
• Test plan - XML;
• Plan entry using Tsung Recorder;
• Monitoring of test servers (Erlang, Munin, SNMP);
• Tools for generating statistics and graphs from work logs.

Using your own scripts that handle log logs, you can withdraw various testing reports:

pros:

• Because it is written on Erlang, it is well scaled, depends on the resources allocated;
• Can perform the role of a distributed system, on a large number of machines;
• A large amount of system tested is not only web servers and databases, but also, for example, an XMPP server: can send messages, tests with authorization;
• Control through the console, but, thanks to the support of plug-ins, you can control and using a third-party plugin with the GUI interface;
• The presence in the TSUNG Recorder tool kit is a kind, Proxy server through which you can walk on a test site and record immediately as a load profile;
• Generation of various testing graphs using scripts.

Minuses:

• No GUI interface;
• Only * NIX system.

WAPT

Paid

Official site

Main features:

• Windows
• Paid (there are trial for 30 days / 20 virtual users);
• Recording plan testing from desktop and mobile browsers;
• Dependence in testing plans (subsequent URL depending on server response);
• Imitation of real users (delays between connections, limiting the speed of compounds).

The report can be displayed both table and schedule.

Not every modern user computer system In everyday work, faces the concept of "load testing". It is mainly familiar to web developers and all those who use resource-intensive programs. Nevertheless, sometimes knowledge in this issue can be useful and ordinary users. Let's try to figure out why all this is needed.

Load and goals

First of all, it is worth clearly distinguishing the types of such tests. Conditionally, they can be divided into two classes: checking computer "iron" with the highest possible or excessive load on each component and (websites with elements of forecasting, separate programs, etc.).

It goes without saying that the testing of sites is directly connected with the testing of the servers, which hosts information, as well as with virtual web servers, which are created in the process of working with specialized programs like Denwer.

Load testing programs and their tasks

As you can see, the relationship is very strong here. And if we talk about the "hardware", the system of load testing using special utilities allows you to precisely identify potential problems when working, so to speak, in the easiest. Modern computer games with their requirements can easily load the system to such a state that it will stop working at all. Therefore, before installing such software on a computer, you can conduct a series of tests to determine whether the game can "pull" the game. According to the results, the decision to install the program is made. In principle, the same applies to applications involving complex mathematical calculations, and design work, since the load on the same processor or RAM compared to the usual state of the system can be exceeded at times.

As for the second class, the testing of the site and server can act as a kind of universal means of predicting their behavior in real functioning. For example, it can be an emulation of the simultaneous request for access to a large number of users. As you know, for this principle there are DDoS attacks when the server or the site does not have time to handle too much appeals. Load testing of the server or site will be discussed in more detail later. In the meantime, we will deal with computer "hardware". This applies not only to domestic or working terminals, but also real physical server systems.

Test processor

Let's start, perhaps, from the heart of any computer - the central processor. It is no secret that it is the problems in his work in most cases lead to the saddest consequences. Very often it is associated with overheating. Load testing allows you to create extreme conditions. And then you can see how it will affect its work.

It goes without saying that carrying out load testing of this type implies the use of certain utilities. Today you can count hundreds and thousands. But, according to most experts, the leader in this area is the PRIME95 application, which can be applied to the processors, and to RAM strips. But the main direction is the verification of the processor chipset.

When using the utility, it is recommended to close all active applications and disable automatic (sleep) so that the computer does not turn off the computer in the process of checking. Now you need to simulate the processor the stringent conditions (and the program can do it, like no other, really set chips in the most difficult conditions). The test itself is activated from the options menu where the Torture Test section is selected. There will be specified types of operations. The most interesting here are Blend tests (simultaneous load and on the processor, and on the "RAM"), as well as Small FFT and LARGE FFT (increasing the load on the processor by unloading RAM).

How to determine what load testing has passed successfully? Unified opinion There is no, but it is believed that if during at least 4 hours of errors or failures in the chip work, this component is sufficiently resistant to excessive loads. But it also happens that the failures may appear much later, so if you have a pretty reserve of time, it is better to increase the test period up to 24 hours (errors may appear and after half a day).

Verification of RAM

No less important is the load testing of "RAM", which performs the functions of the so-called second violin. To do this, the MEMTEST86 + app is best suited, which is the best one to date.

To work correctly with it, you need to create boot disk or flash drive and download a computer terminal from such a carrier. After activating the test, it will take quite a lot of time. You can just leave the computer for the night. This should be enough.

Determining the behavior of the graphics adapter

With graphics, it is also worth conducting a test, since video adapters for excessive load are often the cause of computer failures. The perfect tool here will be the program FURMARK.

This utility is able to heat the graphic chip much stronger than it will make some three-dimensional game with system requirements above average. As practice shows, the conditions are created such that the video card can begin to collect already in a period of 15 to 30 minutes after the start of testing.

In addition, special utilities designed for specific games can be used. For example, test applications like ALIEN VS PREDATOR, S.T.A.L.K.E.R.R. Or something else like that. As a rule, they are distributed completely free, and with their help you can accurately establish how the system will behave after installing the original gaming package.

For what you need testing servers and sites

Now a few words about what is the testing of the site and a web server. About one aspect (DDOS attacks) has already been said. Now let's look at this question on the other hand.

The tests of this type can be attributed to some extent to marketing tools for predicting user behavior. For example, you can simulate the situation of the behavior of a certain amount (maximum / peak) people when entering the site, find out how many pages can be viewed whether an email will be involved, for example, in the process of ordering goods, as information can be used to identify visitors, will allow Whether to provide simultaneous access to the site to users at a certain point in time, will be in demand confirmation of user powers by a third party (for example, when entering the number bank card) How effective will be the introduction of Java applets or the use of the protected connection of HTTPS and so on.

Web Server Test Questions (Software) and Internet Resources Created

In principle, almost the same tasks sets both load testing of the server. However, here the emphasis is made a purely technical aspect. Tests allow you to identify whether several users have the same IP, to clarify the response time to send requests, find out how the entire system is reacting to the protected or unprotected connectionWhat will be the speed of access while simultaneously sending too much requests and so on.

In this case (both for the site, and for a web server), many advise use powerful package Under the name OpenSta (System Architecture Test), which allows not only checking, but also break tasks to the components for each individual structure element using the Script Modeler script modeling tool. It is noteworthy that after creating such a model, you can even check the connection via the SSL protocol (the so-called name server must be launched). In addition, the results can be saved in the Repository Host section, and the tests are combined into the appropriate groups.

What is the result?

In principle, it is very brief information For load testing, because the tests themselves, as well as programs that allow them to carry out, can be found very much. Let's just say: here the most popular utilities are presented and the very essence of the question is considered. I think, after reading any user, at least a little will be promoted in understanding issues related to load testing.

As they grow and complicate sites and applications, the main problem of developers becomes ensuring high performance. All modern studies suggest that the number of visitors, sales growth and traffic increase directly depends on the performance of the site. Therefore, it is important to pay attention to how fast users can access the site in the browser.

Over the past few years, many advanced methods and technologies have been developed in the field of performance optimization. Many of these methods are intended to reduce the size of the downloadable web pages, JavaScript optimize and limit the number of individual HTTP requests.

This article will tell about the basic concepts and open tools to optimize productivity. With it, you can find out how quickly your server responds to user requests, and develop an individual plan.

Basic concepts

First you need to get acquainted with basic terms and concepts.

• Delay is an indicator of how fast the server responds to customer requests. Usually measured in milliseconds (MS). The delay is also often called a response time. The lower this indicator, the faster the server processes the request. The delay is measured on the client side from the moment the request is sent before the response is received. This indicator includes network resources.
• The bandwidth is the number of requests that the server can process for a certain period of time. Usually this indicator is measured in queries per second.
• The percentile is a way to group results percentage of the entire dataset.

Basics of load testing

Load testing is a server performance measurement technology, which is to send imitated HTTP traffic to the server. This allows you to find answers to such questions:

• Is there a resource server enough (memory, CPU, etc.) to handle the expected traffic?
• Is it sufficiently quickly reacts the server to provide a good user experience?
• Is the application work effectively?
• Does the server be vertical or horizontal scaling?
• Are there particular resource-to-post pages or API calls?

Load testing is performed by running a special software on one computer (or in the machine cluster). This software generates a large number of requests and sends them to a web server on the second computer (or in another infrastructure). There are many such tools, later we will look at some of them. At the moment we will focus on general terms that will be relevant no matter what means for load testing you will choose. Normal load testing software is used to determine the maximum number of queries per second, which can process the server. To do this, the server is sent as possible large quantity requests; Then you need to check how many of them the server was able to process successfully.

This allows at the base level to determine the maximum features of the server, but this will not provide a lot of information about delays, daily performance and user experience. The overwhelled server can return a thousand answers per second, but if the processing of each response takes ten seconds, users will probably won't wait.

The general trend is as follows: the higher the load (the more requests per second), the higher the delay. To get a more real picture of the server delay at a given load, it will be necessary to test it several times with different queries. Not all applications for testing the load are capable of this, but a little later we will read the WRK2 (this is a command line tool for testing a load that can perform this function).

How to determine a reasonable delay rate?

Website download time in the range of 2-5 seconds - the usual thing, but part of the time associated with a web server delay is usually about 50-200 milliseconds. The perfect delay rate is individual for each site. It depends on a large number of factors (audiences, market, site objectives, availability of a user interface or API, etc.). Keep in mind: Most studies show that every small speed bit is taken into account in performance, and even completely imperceptible improvements lead to improved results as a whole.

Load testing planning

To understand how the server and the web application works and how they react to the load, several common actions can be taken. First, during testing you need to track correct system resources. Then you need to determine the maximum number of requests per second, which can process this server. Also determine throughputAt which the server delay will lead to low performance and poor user experience.

1: Monitoring Resources

Software for load testing will collect and provide information about requests and delays. But there are some other system indicators that need to be monitored to understand what resources does not have enough server when working with large volumes of traffic.

This is mainly related to the load of the processor and free memory: the monitoring of this data during a high load will help you to make more informed decisions on how to scale the infrastructure and where to focus efforts when developing an application.

If you already have a ProMetheus, Graphite or CollectD monitoring system, you can collect all the necessary data.

Read also:

If there is no such system, connect to the web server and use the following command-line tools for real-time monitoring.

To monitor the available memory, use the FREE command. In combination with the Watch command, the data will be updated every 2 seconds.

Flag -H Displays numbers in a readable format.

total USED FREE Shared Buffers Cached
Mem: 489m 261m 228m 352k 7.5m 213m
- / + Buffers / Cache: 39m 450m
SWAP: 0B 0B 0B

The dedicated number in the output represents free memory after subtracting buffer and cache. New fREE version Displays other results:

Total USED FREE Shared Buff / Cache Available
MEM: 488M 182M 34M 14M 271M 260M
SWAP: 0B 0B 0B

The new Available column is calculated in different ways, but usually represents the same metric: the current amount of available memory for applications.

To monitor the use of CPU on the command prompt, there is a MPStat utility that displays the number of CPU free resources. By default, the MPStat utility is not installed in Ubuntu. You can install it using the following command:

sudo apt-get install sysstat

When you start MPStat, you need to set the data update interval in seconds:

It will display the header string, and then the statistics string, and will be updated every two seconds:

Linux 4.4.0-66-Generic (Example-Server) 08/21/2017 _x86_64_ (4 CPU)
08:06:26 PM CPU% USR% NICE% SYS% IOWAIT% IRQ% Soft% Steal% Guest% GNICE% IDLE
08:06:28 PM All 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00
08:06:30 PM All 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00

The% IDLE column shows which percentage of CPU resources is not used. The processor load is often divided into different categories (User CPU and SYSTEM CPU).

2: Determination of the maximum response speed

As mentioned earlier, most load testing programs are particularly well suited to find the maximum speed of the web server response. As a rule, it is necessary to specify only the competitiveness and duration of testing.

Competition is an indicator that displays the number of parallel connections that the server can process. The default value of 100 is suitable in most cases, but you can choose an individual value. To do this, you need to check the MaxClients, MaxThreads Server and other similar parameters.

You will also need to select the URL for testing. If your software can only process one URL at a time, it is worth performing multiple tests for different URLs, since processing requirements can vary greatly depending on the page. For example, the requirements for downloading the home page of the site and the product pages are different.

Some software for load testing allows you to immediately specify several URLs that need to be verified. This allows you to more accurately imitate real traffic. If you have data on the use of the site (from analytical software or server logs), you can apply this data in testing.

Overting URLs, run testing. Make sure the software very quickly sends requests. If the software allows you to select a query speed, select the value that will almost certainly be too high for your server. If the software allows you to set a delay between requests, reduce this value to zero.

The use of processor and memory resources will increase. Free processor resources can reach 0%, and the client can get a connection error. This is normal because the server works at the limit of opportunities.

When testing is over, the software will display statistical data, including the number of queries per second. Pay attention to the response time: this indicator is likely to be very bad, since the server must be extremely overloaded during the test. Therefore, the number of requests per second is not an exact indicator of the maximum bandwidth of the server, but this is a good start for further research.

Then you need to repeat testing to get additional information About how the server works at the resource limit.

3: Determination of maximum bandwidth

At this stage you need to use software that can speed up the download slightly to check the server performance on different levels bandwidth. Some programs allow you to specify a delay between each request, but it makes it difficult to determine the exact bandwidth.

Here you can refer to the WRK2 tool, which allows you to specify the exact number of requests per second.

Take the maximum speed of the requests you have identified at the previous stage, and divide it on 2. Start another test with new data and pay attention to the response time. Is the indicator in an acceptable range?

If so, increase the value to the maximum and repeat the test until the delay reaches the maximum value that you consider acceptable. This will be the actual maximum response rate that your server can process.

Tools for load testing

There are many open source software packages for load testing servers. In addition, there are many paid Serviceswhich can automatically create graphs and reports based on data obtained during testing. These services are greatly suitable for large sites that need to generate high load for testing a large infrastructure.

However, some of the open tools can also operate in cluster mode. Consider several of the most popular open source tools.

AB Tool

(or ApacheBench) is a simple command line single-threaded tool for testing HTTP servers. Initially, it was designed as part of the Apache HTTP server, but it can be used to test any HTTP or HTTPS server.

Since it is single-threaded, the AB tool cannot use multiple processors to send a large number of requests. It will not work if you want to fully load a powerful web server.

The basic call to the AB command looks like this:

aB -N 1000 -C 100 http://example.com/

The -N flag sets the number of requests. Flag -s sets competitiveness. Then you need to specify the URL to be tested. The output (excerpt from which is shown below) indicates the number of requests per second, request time and a list of percentile response time:

. . .
Requests Per Second: 734.76 [# / SEC] (Mean)

Time Per Request: 136.098 (MEAN)
Time Per Request: 1.361 (Mean, Across All Concurrent Requests)
TRANSFER RATE: 60645.11 Received
PERCENTAGE OF THE REQUESTS SERVED WITHIN A CERTIAIN TIME (MS)
50% 133

66% 135

75% 137

80% 139

90% 145

95% 149

98% 150

99% 151

100% 189 (Longest Request)

Jmeter.

JMeter is a powerful and multifunctional application for load and function testing from Apache Software Foundation. Functional testing is to check the output of the application.

JMeter offers Java graphical interface to configure test plans.

Test plans can be written using the JMeter proxy server and a regular browser. This allows you to use traffic in tests, which more accurately imitates the actual operation of the server.

JMeter can display information about percentiles in HTML reports and other formats.

Siege

Siege is another command line tool for load testing. It looks like AB, but has several additional features. Siege - a multi-threaded tool that provides a relatively high throughput. It also allows you to specify several URLs for load testing at once. The basic call looks like this:

siege -c 100 -T 30s http://example.com/

The flag is indicated by competitiveness. The -T flag determines the duration of testing (in this case - 30 seconds). Siege displays the average response time and request speed:

. . .
Transactions: 5810 Hits
Availability: 100.00%
Elapsed Time: 29.47 secs
Data Transferred: 135.13 MB
Response Time: 0.01 secs

Transaction Rate: 197.15 TRANS / SEC
Throughput: 4.59 MB / Sec
Concurrency: 2.23.
. . .

Siege does not provide percentiles for delay statistics.

Locust.

Locust is a tool for load testing on python-based basiswhich provides a web interface for monitoring the results in real time.

Locust testing scenarios are written using Python Code, which provides additional benefits to those who are familiar with this programming language.

Locust can also be launched in a distributed mode: You can start the cluster from the Locust servers, which will create a high load of your server. This allows you to perform high-quality load testing of the entire web server infrastructure.

Locust can provide detailed statistics in CSV files that can be downloaded.

Tool WRK2.

wRK2 is a multi-threaded command line tool for load testing capable of loading with a given frequency of requests. It can provide detailed delay statistics and supports scripts in the Lua programming language.

wRK2 is called by the WRK command:

wRK -T4 -C100 -D30S -R100 --LatenceNCY http://example.com/

The -T parameter determines the number of streams (in this case, they are 4, here you need to use the number of processor kernels of your server). The -c parameter indicates the number of simultaneous requests (here 100). The -D flag determines the duration of testing (30 seconds). The -R flag indicates the frequency of requests per second (100). A detailed delay output will provide the -Latence flag.

. . .
Latency Distribution (HDRHISTOGRAM - Recorded Latency)
50.000% 5.79ms
75.000% 7.58ms
90.000% 10.19ms
99.000% 29.30ms
99.900% 30.69ms
99.990% 31.36ms
99.999% 31.36ms
100.000% 31.36ms
. . .

Conclusion

In this article, we considered the terminology and the basic concepts of load testing, familiarized with the test planning and considered some of the available open tools for testing.

By defining infrastructure performance, you can use this information to try to improve the response time and reduce the load on the server. You may decide in favor of vertical or horizontal scaling. You can optimize your web server configuration: change the number of supported connections, workflows or streams. You can also optimize the caching of frequently used data, reduce the load on the database and query time.