Figure 7.3 shows the desktop client interface of our thick-client. It is imple-mented in C#.Net Winforms.
Figure 7.3: Experimental system with thick client interface
Desktop client application was extended to load testing tool. While Web-Socket is a new protocol, there are none load testing tools that can act over WebSocket and none extensible with some subprotocols. This client allows us to do real benchmarks of Weda against REST and SOAP over HTTP SOS ser-vice. Legacy SOAP/REST endpoints are also invoked and used in benchmarks as baseline.
Following parameters can be set for each test case. User performing the tests should choose between endpoint types. Weda endpoint only react with SOAP encoding at the moment. Then Operation parameter muset be set.
Server can perform PING or GetCapabilities operation or operation workflow with GetCapabilites and GetFeatureOfInterest operation called when response for GetCapabilities received.
Test Case is limited to specified duration limit. There can be various run strategies simulating many types of load over the time allowing to test the performance of the target service under a number of conditions.
The Simple Strategy runs the specified number of threads with the specified delay between each run to simulate a “breathing space” for the server. For example if you want to run a functional test with 10 threads with 10 seconds delay, set Clients to 10, delay to 10000. The Simple Strategy is perfect for Baseline testing or evaluating competing devices. User can use it to assert the basic performance of the service and validate that there are no threading or resource locking issues. By ramping up the number of client threads we do more elaborate load testing or long-running soak tests.
Burst strategy will perform nonlimited amount of requests during the burst duration. This strategy is specifically designed for Recovery Testing and takes variance to its extreme. It does nothing while the Burst Delay, then runs the configured number of threads for the “Burst Duration” and goes back to sleep.
Here we could set the number of threads to a high value (20+) to simulate an onslaught of traffic during a short interval, then measure the recovery of our system with a standard baseline.
Client Strategy lets us linearly change the number of threads from one level to another over the run of the LoadTest. It’s main purpose is as a means to identify at which level certain statistics change or events occur, for example to find at which ThreadCount the maximum TPS or BPS can be achieved or to find at which ThreadCount functional testing errors start occurring. LoadTest containing basic performance-related assertions.
These values gets measured:
• clientsrunning – number of client threads that are actually running
• t_limit – duration limit for Test Case
• t_starttime – time at which the Test Case was started by user
• t_endtime – time at which the Test Case was stopped due to canceling the operation or duration limit
• t_lasttime – indicator of last invoke, in GUI refreshed every 5s to indicate liveness of the running task
• Response time refers to the time it takes to complete a request from the time it reaches the server machine to the time the last byte is returned to the client. This measurement does not take into account latency between the end-user machine and the system.
• t_last – the time (in milliseconds) taken by the last sample
• t_min - the minimum time (in milliseconds) taken by the sample
• t_max - the maximum time (in milliseconds) taken by the sample
• t_avg - the average response time is a good number to know, but from a performance requirement perspective, it normally doesn’t drive SLA’s.
Average time taken will be
Average time = Total time taken by all samples / Count of samples
• t_90th - is the 90th percentil value below which 90% of the samples fall.
The remaining samples too at least as long as the value. This is a stan-dard statistical measure. It is driven by SLA (service level agreements) contracts, which tell the customer “you can expect 90% of the requests to finish within x time”
• stddev - Standard Deviation is a measure of the variability of a data set.
This is a standard statistical measure.
• tps_abs – absolute throughput is based on actual time passed:
tps_abs = Count of samples / seconds passed,
i.e. a TestCase that has run for 10 seconds and handled 100 request will get a TPS of 10
• tps_rel_avg - relative throughput is based on average execution time , tps_rel_avg = ( 1000 / avg ) ∗ threadcount ,
for example avg = 100 ms with ten threads will give a TPS of 100
• bytes - the number of bytes processed by the test step.
• bps_abs - the bytes per second processed by the test step. i.e. a TestCase that has run for 10 seconds and handled 100000 bytes will get a BPS of 10000.
• bps_rel_avg - the average number of bytes per request * tps;
bps_rel_avg = ( bytes / Count of samples ) ∗ TPS,
i.e. a total number of received bytes of 1000000 for 10 requests with a TPS of 100 would give (100000/100 * 100) = 1000000 BPS.
• bytesin - the number of bytes processed by the responses.
• bytesout - the number of bytes processed by the requests.
• cnt - the number of times the test step has been executed.
• cnt_aftertimeout - the number of times the test step has been executed with duration longer than 30s
• err - count of errors in the samples
Big and time consuming task with the thick client application was to de-velop full multithreaded framework that allows concurrent sampling by many threads, well locking and with GUI thread not influencing the results.
. Conclusion
In this chapter, the author presents the application (experimental system) of the WEDA architectural style for developing “GIS on the web” solutions.
There are many applications for this system from small ones (e.g. warning the public or farmers within a range of 10 km before an approaching storm or hail) to bigger ones (monitoring of emissions in real time by informing the public to close windows and alerting the relevant authorities after exceeding permitted limits) or building automata that can warn before some critical event occurs.
With this technology, results from very different sensor types can be processed together and cross calculated. Users can define their own preferences for what to monitor and alert for. The resulting trend analysis can be made available on a global level and deployed in the cloud environment. End-developer can build feature-rich viewers and systems (such a task was not in the scope of this work).
Thick client exprimental system was built and used as load testing tool be-cause for the WebSockets and its subprotocols there is none available. It was time consuming task to create fully multithreaded tool that doesn’t distort the result because of well designed locking, concurrent processing and dealing with GUI thread that should not influence the results. Collected results are used in the next two chapters.
. Distribution of response time instability
. Motivation
The response time is the most visible performance index to users of computer systems. End-users see individual response times, not the average. Therefore the distribution of response times is important in performance evaluation and capacity planning studies. The inability of the webservices involved to guaran-tee a certain response time and performance and the instability of the commu-nication medium can cause timing failures, when the response time or the tim-ing of service delivery (i.e., the time durtim-ing which information is delivered over the network to the service interface) differs from the time required to execute the system function. In result some users may get a correct service, whereas others may perceive incorrect services of different types due to timing errors.
These timing errors can become a major cause of inconsistent failures usually referred as the Byzantine failures [57]. Uncertainity of performance can be ref-ered as the unknown, unstable, unpredictable, changeable characteristics of measured system. Issues of response time analysis are becoming more impor-tant as models are developed that make explicit predictions about the shape of the response time distribution. Results for such a measuring can be used for making decision about the sytem architecture or configuration such as setting timeouts. For example today’s knowledge in the area shows us that the RTs are not normally distributed [92] and they are highly skewed.
We measured the request processing time (RPT) (by a service) and the net-work round trip time (RTT), i.e. response time RT = RPT + RTT in store and forward devices (RFC 1242). In our work we followed methodologies de-scribed at [88,92,35]. Then we investigated how instability changed during 3 hours. With collected data we found the way to predict and represent perfor-mance uncertainity of Weda by employing one of the theoretical distributions, used to describe such random variables like Weda’s response time.