Results

The load tests had a big influence on the overall system. The main changes were made to how Riak was being accessed. We discovered which Riak features are expensive and even discarded some completely. This section describes those discoveries and justifies our decisions.

4.3.1.1 Load balancing

One of the first issues that appeared with load testing was that the system hardly scaled at all. Once we compared the number of reductions between machines in the test cluster, we realised one machine was getting much more load than others, explaining the bad scaling result. Test sessions were being allocated to random machines but we were using too few sessions, resulting in an uneven distribution between backends. By changing our load test setup to create more sessions, we ensured that the test load was evenly distributed among backends.

Once this was fixed, the number of reductions on different machines indicated even balance of work between machines.

4.3.1.2 Riak & backend relation

First of all we were interested in how to setup our cluster - how many Riak nodes does a backend require to perform its best and should Riak and backend nodes be on the same machine?

Fig. 4.1 shows the different setups we tested. The x-axis is the time the tests were running and the y-axis the throughput. The flow included all game play operations like game creation, joining, order submission and messaging. Registration and login are not part of it. The graphs are the different setups, they are labeled with XR or XB. Meaning on machine X there is a Riak node or a Backend node¹.

As you can see most of them performed quite bad. Fig. 4.2 shows only the best setups. These three setups performed more or less the same. One of these setups is a cluster consisting of three machines each with a Riak and a backend node. Since this is also the simplest setup we decided to use setups with X machines each having a a Riak and a backend node in the future.

4.3.1.3 Degradation

The major issue with the previous results was the obvious performance degradation over time. Fig. 4.3 shows that this occured for any cluster size

11B 2R means there is a backend node on machine 1 and a Riak node on machine 2.

Figure 4.1: Load tests of different cluster setups.

Figure 4.2: The best graphs from Fig. 4.1.

Figure 4.3: Load tests for 1-6 machines.

Preliminary prototype benchmarks did not show this pattern, which made us compare our system and the prototype we benchmarked. The main difference was the usage of eLevelDB as Riaks backend storage in the system, whereas the prototype used bitcask, the default storage backend.

The only reason we used eLevelDB was it being the only backend support-ing secondary indices. We decided to do a comparison of those two backends, which required to remove the usage of secondary indices. Therefore we replaced it with Riak search. As you can see in Fig. 4.4 our assumption proved to be correct. Both start of the same but the throughput of eLevelDB version drops quickly, whereas the bitcask version remains almost stable.

4.3.1.4 CAP controls

Another small but effective change was the adaptation of the Riak CAP controls. Many writes to Riak have a quite low priority. The message logging for example is such a case. The system does not need to await confirmation of the write. Fig. 4.5 displays the increase in throughput this gave to our system.

4.3.1.5 Key filtering

Despite those improvements the system did still not scale as well as hoped.

The Riak community² recommended us to take a closer look into our Riak search usage, since it has not been as well tested as other parts of Riak.

2IRC channel #riak on freenode.net

Figure 4.4: Comparison of the storage backends bitcaks and eLevelDB.

Figure 4.5: Improvement by changed CAP controls.

The system used Riak search quite extensively, especially after the change from eLevelDB. In many parts of the system it just replaced secondary indices.

The full text search was not really necessary, as we just needed to find values without knowing the unique id. Therefore we tried to replace Riak search in parts with Riaks key filtering. For messages for example we introduced keys of the form < unique − id > − < f rom > − < to >. With key filtering it is then possible to query the messages for a certain user without the use of Riak search.

The advice from the Riak community turned out to be very valuable.

Riak search should be avoided for values that are being written often, as the indexing is too expensive to scale well. Unfortunately though we no longer have the data to show a comparison graph.

4.3.1.6 Final results

Fig. 4.6 shows the performance of the final system. The y-axis is the amount

� � � �

�

������

������

������

������

�������

�������

�������

�������

�������

������ ����� �������

������ ����� �����������

������ ����� ��������

Figure 4.6: Scaling.

of model users the system can handle. Where a model user is assumed to perform about 20 requests a day and all users are distributed evenly throughout the day. The blue one is just for comparison and displays how perfect linear scaling would look like. The yellow graph shows the performance of the system right after we switched from riak search to key filtering. With one machine it can handle around 400k model users, and with 4 machines a bit more than double the amount. The green one is the system at the end of this project.

Unfortunately the performance dropped a lot. The cause for this is a single commit that introduced an increased usage of riak search again.

Unfortunately there was no time left to get the system back to its previous performance. Still the system performs quite well and 400k model users is most likely more than the global count of diplomacy players.