Institutionen för datavetenskap
Department of Computer and Information Science
Examensarbete
To what extent does Ruby on Rails affect
performance in applications
av
Martin Nordén
LIU-IDA/LITH-EX-A—15/061—SE
Linköpings universitet Institutionen för datavetenskap
Examensarbete
To what extent does Ruby on Rails effect
performance in applications
av
Martin Nordén
LIU-IDA/LITH-EX-A—15/061—SE
2015-10-20
Handledare: Anders Fröberg
Examinator: Erik Berglund
Instituitionen f¨or datavetenskap TABLE OF CONTENTS
Table of contents
Table of contents 3 Abbreviations 7 Summary 9 1 Introduction 11 1.1 Motivation . . . 11 1.2 Purpose . . . 11 1.3 Problem at hand . . . 11 1.4 Delimitations . . . 11 2 Background 13 3 Theory 15 3.1 Active Record . . . 15 3.1.1 The N+1 issue . . . 153.1.2 Active Record SQL statement creation . . . 16
3.1.3 Indexes within Active Record . . . 17
3.1.4 Null constraints . . . 17
3.1.5 SQL caching . . . 17
3.2 Ruby . . . 17
3.2.1 Ruby is an interpreted language . . . 18
3.2.2 Static analysis, evaluation and safety . . . 18
3.2.3 Garbage collection, i/o processing and thread operations . . . 18
3.2.4 Performance speeds fetching data . . . 19
3.3 Ruby on Rails . . . 19
3.3.1 Model-view-controller pattern . . . 19
3.3.2 Caching in Ruby on Rails . . . 19
3.3.3 Cache stores . . . 20
3.3.4 Security features . . . 20
3.4 Conclusions of theory study . . . 20
4 Method 21 4.1 Pre study and design . . . 21
4.1.1 Implementation . . . 21
4.1.2 Test cases . . . 21
4.1.3 Stress tests . . . 21
4.1.4 Testing performed by people . . . 22
4.2 Implementation . . . 22
4.2.1 Test system nginx/passenger configuration . . . 22
4.2.2 Test system Ruby on Rails performance environment . . . 22
4.2.3 Test unit Huawei P8 . . . 22
4.2.4 Computationally complex data structure . . . 22
4.2.5 Computationally semi-complex data structure . . . 24
Link¨opings universitet
Instituitionen f¨or datavetenskap TABLE OF CONTENTS
4.2.7 Ruby on Rails caching . . . 25
4.3 Evaluation . . . 25
4.3.1 Performance tests . . . 25
4.3.2 Test with mobile device . . . 26
4.3.3 Stress tests . . . 26
4.3.3.1 Computationally complex data structure . . . 26
4.3.3.2 Computionally semi-complex data structure . . . 26
4.3.3.3 Computionally simple data structure . . . 26
4.3.4 Active Record N+1 issue . . . 27
4.3.5 SQL statement creation . . . 27
4.3.6 Ruby on Rails caching . . . 28
5 Result 29 5.1 Performance tests . . . 29
5.1.1 Computationally complex data structure . . . 29
5.1.2 Computationally semi-complex data structure . . . 29
5.1.3 Computationally simple data structure . . . 29
5.2 Test with mobile device . . . 29
5.3 Stress tests . . . 29
5.3.1 Computationally complex data structure . . . 29
5.3.2 Computationally semi-complex data structures . . . 30
5.3.3 Computationally simple data structure . . . 30
5.4 Active Record N + 1 issue . . . 30
5.5 SQL statement creation . . . 31
5.6 Ruby on Rails caching . . . 31
6 Discussion 33 6.1 Method . . . 33
6.1.1 Literature . . . 33
6.1.2 Test system nginx/passenger configuration . . . 33
6.1.3 Test system Ruby on Rails performance environment . . . 33
6.1.4 Test unit Huawei P8 . . . 33
6.2 Evaluation . . . 33
6.2.1 Performance tests . . . 33
6.2.2 Test with mobile device . . . 34
6.2.3 Stress tests . . . 34
6.2.4 Active Record N + 1 issue . . . 34
6.2.5 SQL statement creation . . . 34
6.2.6 Ruby on Rails caching . . . 34
6.3 Result . . . 35
6.3.1 Performance tests . . . 35
6.3.1.1 Computationally complex data structure . . . 35
6.3.1.2 Computationally semi-Complex data structure . . . 35
6.3.1.3 Computationally simple data structure . . . 35
6.3.2 Test with mobile device . . . 35
6.3.3 Stress tests . . . 35
6.3.3.1 Computationally complex data structure . . . 35
6.3.3.2 Computationally semi-complex data structure . . . 36 4
Instituitionen f¨or datavetenskap TABLE OF CONTENTS
6.3.3.3 Computationally simple data structure . . . 36
6.3.4 ActiveRecord N + 1 issue . . . 36
6.3.5 SQL statement creation . . . 36
6.3.6 Ruby on Rails caching . . . 36
7 Conclusions 37 7.1 Further work . . . 37
8 References 39 Appendices 41 A Result of stress tests 41 A.1 Computationally complex structure . . . 41
A.2 Computationally semi-complex structure . . . 42
A.3 Computationally simple structure . . . 43
B Result of caching test 44 B.1 Computationally complex structure . . . 44
Instituitionen f¨or datavetenskap ABBREVIATIONS
Abbreviations
Ruby on Rails - RoR Garbage collection - GC
Instituitionen f¨or datavetenskap SUMMARY
Summary
Ruby on Rails(RoR) is a web application framework, the objective of this thesis is to study any positive or negative performance implications regarding RoR, the Ruby language and the object-relational mapping Active Records. This was achieved in two different ways, first a theoretical study on the subject was conducted. Secondly a project was implemented in RoR and evaluated based on the criteria found during the theoretical study.
The results of the evaluation and study are as follows. Active Record have some issues that seem to be easily fixable, others demand more attention. The garbage collection(GC) in Ruby does not seem to impede performance much and overall RoR and Ruby does a good job. The caching functionality in RoR if used correctly can greatly improve overall performance. If performance is not of utmost importance and can to an extent be traded for elegance in code, RoR is a great fit.
Instituitionen f¨or datavetenskap INTRODUCTION
1
Introduction
1.1 Motivation
In recent days network applications and websites have exploded in numbers. Applications should be flexible, dynamic and fast, with no unnecessary delays. The application development frameworks of today are here to create applications satisfying these demands. These tools are also supposed to ease the development process for the developer, balancing ease of use with performance for the perfect combination.
When considering the development of mobile devices such as smart phones and tablets that has access to the internet anywhere. These devices can access web site and applications everywhere, adding more strain the the server managing all these requests. This extra strain is an issue amongst systems of today.
1.2 Purpose
The purpose of this thesis is to study whether RoR is a good and suitable choice for an application that focus on high performance. This thesis will help giving a better understanding of the RoR framework and its strengths and weaknesses. This evaluation will also study how to use the framework in the most performance effective way. The thesis will study how the different areas of the framework such as the object-relational mapping tool Active Record and the Ruby language affect the framework regarding performance. If found, this thesis might lead to suggestions of how to enhance performance of the framework or when using it.
1.3 Problem at hand
• What impact does Ruby on Rails have on performance for an application?
• How does Ruby the programming language influence Ruby on Rails performance?
• To what extent does Active Record the Ruby on Rails object-relational mapping affect perfor-mance?
1.4 Delimitations
To be able to perform a suitable amount of work, this thesis is limited to focus only on the framework RoR. The focus of the thesis is the back end capabilities of the framework. The study will target performance and no other qualities are deemed interesting for this study.
Instituitionen f¨or datavetenskap BACKGROUND
2
Background
For this thesis a back end application is to be studied. The demands and specification for this application has been administered by Jemac Sweden AB in the form of a thesis job. The complete work involves creating a graphical user interface for controlling heat pumps and temperature sensors from all possible devices. This thesis is focused on creating the back end for this interface in Ruby on Rails.
The back ends main entities are called system owner, partner, devicelocation and user. The
larger part in this thesis is to create a structure for managing the system owner and partners and their relation to devices and users. This structure will be organised as a tree structure with system owner at the top and partners below it. A partner at a certain level will be able to control all other partners below the partner in its own tree structure as well as the devices and users belonging to these partners. The system owner will be able to control all partners, users and devices.
Within the project three different kinds of users exist. These being system owner users, partner users and end users. The user type associated to a user dictates which other users and devices it can reach. System owner users will have access to all devices and users. Partner users will have access to all users and devices within its partner tree. End users can only access the device linked to that end user. Another smaller entity is also involved when deciding access, this entity is called user role. Where user types decides upon which devices and users can be reached, user roles determines how the user can interact and modify the object. Different user roles can as an example be system admin, installer or technician and these roles will have different access rights.
Device locations are objects reflecting the physical units. Through one of these objects the setting for its physical counterpart can be changed. A device location object can also be used to display current settings and other necessary information passed from that device.
The end product will consist of two different web portals. One of the portals will be for end users and installers. in this portal an end user can access its device and see settings for it and change attributes such as temperature. It will also be possible for installers through this portal to perform an initial installation of a device and connect this device to the correct end user.
The second portal will be a support portal for all involved parties within the project. From here a partner or the system owner can control and update settings on all its belonging devices and users.
Instituitionen f¨or datavetenskap THEORY
3
Theory
To better understand the strengths and weaknesses of RoR and be able to assess them, an in-depth theory search and evaluation will be performed in this section. Three big components have been decided to be the focus of this evaluation. The first component is the object-relational mapping tool of Ruby, Active Record. In this section it will be studied how Active Record operates and if any default settings could impact performance, furthermore the communication between Ruby and Active Record will be evaluated.
The second component is the Ruby language. It will be studied whether the Ruby language is suitable as a high performance language, it may be compared to other similar languages to further investigate its strengths and weaknesses.
Lastly the RoR framework itself will be studied. This will involve investigating if RoR enhance or degrade performance and why. Additionally how and if the model-view-controller pattern affect performance will be examined. The study will also analyse if the framework offers any features that will improve or downgrade performance for an application.
3.1 Active Record
One major area to evaluate when studying performance for a framework such as RoR is data man-agement and access. Ruby uses the SQLite3 database and RoR uses Active Record to communicate with the database and its tables[15].
3.1.1 The N+1 issue
Active Record utilises lazy relations between tables[11], this means that data from another table that is included into the original table through a relationship will not be accessed unless the original table is explicitly told to do so. To clear things up, here is an example with two tables:
person
id f irst name last name living location telephone Table 1: Example structure for person table.
living location
id address zip code city
Table 2: Example structure for living location table.
The table persons consist of persons and their contact information, it does not store the address information for a specific person but rather a relationship to another table with this information. This table is called living location and stores address, zip code and city.
Link¨opings universitet
Instituitionen f¨or datavetenskap THEORY
be performed. The issue arises when the living location for a person is inquired, then one query will be performed regarding the person and another second query asking for that living location from living location. Two queries instead of one might not seem like a big deal, but the issue is not called the 1 + 1 issue, it is called the N + 1. When the whole person table is queried and this query includes the living location for every individual, first a single query for all persons will be processed and for each individual there will be an additional query for living location, this adds up to a total of N + 1 queries.
This can fortunately be solved by good programming, this code for example will create four SQL queries:
P e r s o n . f i n d ( [ 1 , 2 , 3 ] ) . e a c h {
| p e r s o n | p u t s p e r s o n . l i v i n g l o c a t i o n . a d d r e s s }
Whereas this code will only create one SQL query[12]:
P e r s o n . f i n d ( [ 1 , 2 , 3 ] , : i n c l u d e => : l i v i n g l o c a t i o n ) . e a c h {
| p e r s o n | p u t s p e r s o n . l i v i n g l o c a t i o n . a d d r e s s }
The issue with this approach however is that the programmer will have to keep track of all places where this problem can arise.
3.1.2 Active Record SQL statement creation
Another performance issue with Active Record is its statement creation. This means the way it translates Ruby code to SQL code to be queried in the database. Active Record translates Ruby code on a line by line basis[10]. When one line has been translated, processed and the result has been returned, it continues onto the next line. An example to illustrate:
a d d r e s s t a g = ’%Rydsvagen% ’ #f e t c h a l l i d : s on Rydsvagen
l i v i n g w h e r e = LivingWhere . where ( ’ a d d r e s s LIKE ? ’ , a d d r e s s t a g ) . p l u c k ( : i d )
#f e t c h a l l p e r s o n s l i v i n g on Rydsvagen
p e r s o n s = P e r s o n . where ( l i v i n g l o c a t i o n : l i v i n g w h e r e )
This Ruby code will in turn become this SQL code: SELECT l i v i n g w h e r e . i d FROM l i v i n g w h e r e WHERE ( a d d r e s s LIKE a d d r e s s t a g )
SELECT p e r s o n s . ∗ FROM p e r s o n s
WHERE ( p e r s o n s . l i v i n g l o c a t i o n IN l i v i n g w h e r e )
Instituitionen f¨or datavetenskap THEORY
As the example shows, the Ruby code is as mentioned processed one line at the time and a SQL statement is created and then the next line is processed. The more effective way to process this would be some form of abstract interpretation, where Active Record read more lines and first then creates a suitable query. In this case a simple join would suffice, but it might not always be so easy with more advanced queries.
Another issue with this is that the construction of queries is not as coherent as the semantics of SQL. Active Record creates multiple SQL statements and uses the Ruby heap to carry results between these queries. This can result in huge IN (...) clauses in the SQL query and can possibly bury the SQL parser with data[10]. This also results in a lot of performance expensive and frequent changes between the Ruby scope and the SQL scope.
This issue is more complicated to solve than the N+1 issue. A solution could be to skip Active Record altogether and use a framework which utilises abstract interpretation when creating the trans-lation rather than line by line interpretation.
It is also possible to solve by creating more advanced statements rather than small short ones. This will minimise the time for SQL to perform queries as well as scope changes between Ruby and Active Record, but this could be hard for complicated queries and some queries might be missed. 3.1.3 Indexes within Active Record
Active Record does not use default indexes for any attributes except primary keys[11]. Searches in a SQL database when not using a index is significantly slower than when using an index. In a big database the result of this will be a lot of time spent looking for the correct record. This issue is however easily fixed by explicitly creating indexes for suitable attributes when creating the database. 3.1.4 Null constraints
Active Record does not by default use null constraints for columns[11]. This will increase the shear size of the database and through that add time to processing of queries if that column is involved in the query. This is easily fixable by code.
3.1.5 SQL caching
Caching of frequently queried records and results help improve the overall performance of a database since this reduces the number of trips into the database. Active Record has nearly no capabilities for handling of caching[13].
3.2 Ruby
Evaluating performance of a language is always a complicated task, questions such as what is perfor-mance really and how to evaluate it needs to be addressed. There are also some different perspectives that can be used.
Link¨opings universitet
Instituitionen f¨or datavetenskap THEORY
3.2.1 Ruby is an interpreted language
Ruby is an interpreted language and an initial approach can be to compare how an interpreted lan-guage manages against a compiled lanlan-guage. Interpretation means that the code is interpreted and evaluated during runtime and compilation means that the code is compiled into a executable machine code file before runtime[7].
It is rather obvious that interpretation has a negative effect on performance compared to compi-lation. Even more so when considering that compilation has been around for a long time with plenty of time to optimise performance of compilers[7]. However the line between compilation and interpre-tation is not as clear today since some interpreted languages compile code into an intermediate form such as byte code and interpret only that intermediate media during run time for performance boosts. The regular Ruby compiler does not use an intermediate form for interpretation[7].
There are high performance implementations for Ruby called YARV and Rubinius. These imple-mentations compiles the Ruby source code to byte code that is used during the interpretation[7]. If performance is necessary, this can be something to have in mind.
3.2.2 Static analysis, evaluation and safety
Ruby’s open classes, dynamic typing and the other meta-programming features impedes compiler optimisation of the language[5]. The reason is that these features makes it absolutely necessary for the Ruby compiler to have the complete source code to assure that context is established, otherwise evaluation might yield incorrect results. This means that for any partial evaluation in the interpreter, the compiler need all source code. When context is known analysis is performed top-down[5].
Since Ruby is a dynamic language and user input and data can be changed at runtime, it is not even possible to partially evaluate every expression and the compiler needs to figure out what to evaluate[5]. This is a question of what is safe to evaluate and what needs to be evaluated with user given input. Safe operations usually means that it does not change any context or dependencies. This is not the case in Ruby since that definition is to strict and Ruby depends on changing and modifying context. The definition for a safe method in Ruby is if the receiver and arguments are known, the block is safe, no states shared between different parties are changed and the method called is considered safe[5]. The block is safe if its expressions are safe. With this in mind the task to deciding which methods are safe in context is time consuming. The interpreter loads 16 modules, 147 classes and 8236 methods in these classes. The one upside is that the Ruby library is written in C for performance[5].
C and other static languages were made to cut down creation time on the cost of performance in regards to program in assembly[5]. Ruby was made for the same reasons to the same price. Cutting down time creating applications and sacrificing performance.
3.2.3 Garbage collection, i/o processing and thread operations
Ruby have suffered a lot due to its GC, i/o operations and thread operations[6]. this can easily be observed when setting a Ruby program to operate on a large file and see it eat away CPU cycles[6]. Ineffective memory control and the use of an unnecessary amount of cycles during operations definitely will limit the performance of the language.
Before Ruby 2.1, the default GC malloc limit was 8MB[9]. This number was set many years ago when this was enough for any ruby application. This was fixed in Ruby 2.1, where multiple new variables for specifying the limit was introduced[9].
Instituitionen f¨or datavetenskap THEORY
3.2.4 Performance speeds fetching data
Interpreted and compiled languages is different and has different upsides and downsides and comparing performance between them might not be fair, especially when considering that Ruby is written in C. It might be a lot more interesting to find out how Ruby compares against other dynamic languages, it would be even more interesting when these languages are used for the same purposes. These results are gathered from an article comparing four different dynamic languages in regards to performance for fetching data from databases[4]. The languages are Groovy, Scala, Ruby and Python.
The tests were performed with databases containing 1 million objects. The languages will fetch different data types such as linked lists, tree structures and queues.
In these comparisons Ruby was the fastest for fetching tree structures and was near the fastest for the other data types[4]. Conclusions from this is that the Ruby language can compete rather well against other interpreted languages in this area.
3.3 Ruby on Rails
Ruby on Rails is a framework for creating web applications that uses the model-view-controller pat-tern. Active Record is used to connect programming objects to database records. RoR is mostly preconfigured and the programmer should not have to spend hours configuring the system[1].
3.3.1 Model-view-controller pattern
On this subject, I have not to find any documentation whether this has any impact on performance. Although since most frameworks today uses this pattern, it is a reasonably guess that the pattern is the optimal way of doing these types of frameworks.
3.3.2 Caching in Ruby on Rails
Ruby on Rails features multiple different ways to cache data. Static pages can be cached entirely in the web server. This means that when a new request for this content is received, the web server can directly send back the cached page instead of inquire Ruby on Rails for the content. Caching is a very useful way of positively influence performance of an application. Through caching a lot of request that otherwise would inquire the whole chain of command of RoR and Active Record can stop at the web server and cut a lot of time from this otherwise time consuming process[2].
As of RoR 4, action caching, that is caching of actions within a controller has been replaced with key-based cache expiration. Key-based cache adds data from the database model to a unique key. A specific key will always return the same data. Whenever the data is updated, a new key is added to the updated data[2].
In the dynamic applications of today, static page caching can feel rather outdated. Fragment caching can be used as a smarter solution. Fragment caching caches specific parts of a page. When a new view is requested, all the static parts can be returned by the web server and only the dynamic data is queried by RoR[2].
Ruby on Rails support Low-Level Caching, caching of certain search constants and results. SQL caching is also allowed within RoR[2].
Link¨opings universitet
Instituitionen f¨or datavetenskap THEORY
3.3.3 Cache stores
To utilise caches, the information needs to be stored somewhere. Ruby on Rails does offer a multitude of choices. The standard implementation for cache stores is the file system, but the cache store can also be configured to use the considerably faster memory of the machine[2].
There are also centralised caches for applications. One of these being Danga’s Memcached server, this is bundled in the Dalli gem per default and used by a lot of high performance sites. If the application uses JRuby, then Terracotta’s Ehcache could be an even faster option[2].
3.3.4 Security features
Ruby on rails have some default security features which obviously will impact performance, with that said I would not recommend removing these if you do not know what you are doing. The first is the protection against cross site request forgery(CSRF) attacks. This protection adds a token to all requests that is then checked at the server to make sure that the correct individual sent the request and that the request is correct[14].
The second default protection is SQL injection protection. It automatically removes special SQL characters such as the ’ character[14].
3.4 Conclusions of theory study
Active Record does seem to have a lot of performance impacting issues, the N+1 issue and the way SQL statements are created seems like the worst. With this in mind, good programming can solve most these issues making for a faster database access tool. If this still is not enough, Active Record can always be changed for another framework more focused on performance.
Ruby is slower than compiled languages, but that is hardly something to be surprised about. Even though Ruby does not use any intermediate form of interpretation it does still seem to fare well against other similar languages when measuring times fetching data. This is as I see it probably one the most important areas to be fast in when operating with the world wide web. If additional performance is needed other implementations of Ruby can be used.
Ruby on Rails have some default security features that dwindle performance, but these can almost be considered essential and is most likely part of similar frameworks written in other languages. Ruby on Rails have several different cache features that will increase performance and could possibly do so by much. Different cache stores can also be used, anything from the slower but bigger hard drive to an exclusive server created for performance.
All in all, I believe that the the framework can be used effectively.
Instituitionen f¨or datavetenskap METHOD
4
Method
4.1 Pre study and design
4.1.1 Implementation
In the background chapter there is an overview for a back end to an application. First this back end will be implemented and this implementation is the to be studied for performance and to gather results in how Ruby on Rails perform. The application will consist of a lot of different routes and various data to be queried. The application is a so called one page application, meaning that HTML and javascript is only loaded once and all other requests is responded to with Json code. This will further diminish times to render views and get a more accurate time frame for responses to requests. The end system is platform independent, meaning basically any device with an internet connection can access it. Tests for this will be included, testing desktops, tablets and smart phones. The database of the implementation will be filled with appropriate test data to accomplish a more realistic setting for these tests.
4.1.2 Test cases
To ensure that a system can manage the deployment demands, tests needs to test this. In this case tests will be written for multiple devices and different complexities of the data queried. Different complexity is here referencing both shallow queries fetching a simple data structure and deep more complex queries gathering complex structures. The theory issues will be tested as much as possible. For the evaluation of results and response times an appropriate time for the request will be used as a reference, this reference will take into consideration the complexity of the request.
The N+1 issue will be tested. One test can be set up to ignore the issue and another tests tries to limit the issue as much as possible but still querying the same data, lastly compare the response times for these requests. The ActiveRecord SQL statement creation will also be tested, this can be done where one test consists of regular Ruby code performing a task and the other test is a specified SQL query performing the same task. This will take into consideration booth the fact that more SQL queries are run and that the scope is switched between Ruby and SQL frequently.
The cache functionalities within Ruby on Rails will be tested. One test suite with certain requests is set up and first tests this suite with cache functionality activated and then turn it off and run the suite again, compare results. Different cache types and stores could be tested.
If time allows, test with different Ruby implementations such as YARV and JRuby could be performed.
4.1.3 Stress tests
Besides the ordinary tests and test cases described above, a stress test suite will be created where multiple users are involved. These users will in perform multiple queries and queries of complexities. Users will only query information that user is entitled to and this test suite will cover a big part of all data within the database. The suite will record response times, creating median and average times and also times for worst case scenario. It might be interesting to test this suite with the cache functionality activated to see how much time can be cut down. As this will be an automated suite it might prove difficult to act as different devices and platforms, but if possible this is how it will be done.
Link¨opings universitet
Instituitionen f¨or datavetenskap METHOD
4.1.4 Testing performed by people
Since this is a web application that is to be used by people, tests of perceived performance on some individuals may be performed. Multiple individuals may be used at the same time for more simulta-neous requests to the server. During these tests it could be tested booth with a predetermined route for the test subject and also free roaming. After these tests a form regarding perceived response times and overall performance could be used.
Due to the nature of people the exact response time can not be measured during these tests, but rather how the individual perceived an individual response and the complete experience through the application.
4.2 Implementation
For the evaluation of the RoR framework, an implementation is described under 2 Background. Here an overview of the implementation is described. This implementation will be tested based on findings from 3 Theory. In this section the test system and configurations are described and also pieces of code for specific tests as well as necessary commands.
4.2.1 Test system nginx/passenger configuration
This test system is a macbook pro 2013. It has an Intel i5 cpu with two cores and operates at 2.4 GHz. The test system has 8 Gb of DDR3 RAM and a SSD. The operating system on the machine is Mac OS X Yosemite.
The webserver is an Nginx version 1.8.0 and Phusion Passenger version 5.0.15 setup. The installed Ruby on Rails version is 4.2.1 and the installed Ruby version is 2.1.2p95.
Test structures and implementations are described under 4.2.4, 4.2.5, 4.2.6 and 4.2.7. In this test configuration the client and server is the same machine.
4.2.2 Test system Ruby on Rails performance environment
This test system uses the exact same hardware and operating system as the nginx/passenger configu-ration, but instead of nginx this environment uses a gem called perf-test. A gem is a packaged Ruby application installed under the Ruby version. Perf-test is a performance environment and can display other important data than the nginx/passenger configuration.
This environment uses the same test data as the nginx/passenger configurations. In this test configuration the client and server is the same machine.
4.2.3 Test unit Huawei P8
To test how RoR acts against a mobile device. A Huawei P8 phone was used agains the RoR framework. This client is connected through a wireless network to the test server.
4.2.4 Computationally complex data structure Computationally complex data structure:
Instituitionen f¨or datavetenskap METHOD system_owner partner_1 partner_2 partner_3 partner_31 partner_11 partner_12 partner_111 partner_1111 partner_21 partner_311
Figure 1: Computationally complex data structure
As seen in Figure 1, the complex structure is a tree structure. system owner is the top node and other nodes are of partner type. A partner can have multiple parents and children within the tree. Only top partners are directly connected to system owner. There exists only one system owner. Example of structure: [ p a r t n e r : sy s t e m o w ne r , c h i l d r e n : [ [ p a r t n e r : p a r t n e r 1 , c h i l d r e n : [ . . . ] ] , [ p a r t n e r : p a r t n e r 2 , c h i l d r e n : [ . . . ] ] , [ p a r t n e r : p a r t n e r 3 , c h i l d r e n : [ . . . ] ] ] ]
Link¨opings universitet
Instituitionen f¨or datavetenskap METHOD
The system owner and partner entities have the same fields in the database:
partners
id name contact inf ormation city
Table 3: Table structure for partners and system owner table.
This is a computationally heavy structure and not the most common structure amongst web applications. In this project there was a focus on easy to read code over absolute top performance, thus my implementation might not be the most performance effective.
4.2.5 Computationally semi-complex data structure Computationally semi-complex structure:
partner_1 partner_11 partner_12 partner_111 partner_1111 device_1 device_2 device_3
Figure 2: Computationally semi-complex data structure
The goal with this structure is to collect all devices connected to a partner. This includes the devices connected to the partner and all devices connected to a partner below in that partners branch. For example, if fetching devices for partner 1, device 1 will be shown as a direct device and device 2 and device 3 will be shown as child devices.
This structure looks like:
Instituitionen f¨or datavetenskap METHOD [ d e v i c e 1 , c h i l d d e v i c e s : [ d e v i c e 2 , d e v i c e 3 ] ]
First an array of all devices for partner. Then an new array under the name child devices with all devices belonging to partners below in the tree structure. This structure will involve 20 partners and a total of 60 devices.
The device locations table looks like this: device locations
id address city conf iguration temperature alerts Table 4: Table structure for device locations table.
This structure needs to traverse the complete branch of the partner, however this is done in a more performance effective way than the complex structure. It was easier to create a more effective solution here.
4.2.6 Computationally simple data structure
The simple structure is an array of device location objects. The device location objects are described under the chapter 4.2.5. This array will consist of 120 device location objects.
4.2.7 Ruby on Rails caching
The caching used in this evaluation is action caching. To implement action caching first modify the following line to true:
c o n f i g . a c t i o n c o n t r o l l e r . p e r f o r m c a c h i n g = true
in the used environment file, that is production.rb, development.rb or test.rb. Secondly in each respective controller where an action should be cached add the line:
c a c h e s a c t i o n : a c t i o n
where :action is the action to be cached.
4.3 Evaluation
4.3.1 Performance tests
With the help of the ruby gem rails-perftest, routes using the data structures in 4.2.4, 4.2.5 and 4.2.6 were tested and evaluated. Each test regarding each structure were tested 1000 times and a mediate
Link¨opings universitet
Instituitionen f¨or datavetenskap METHOD
based on these tests was calculated.
Rails-perftest generates data concerning wall times(real world time elapsed during test run, affected by processes running on system), process times(time taken by process, unaffected by other processes on system). The test gem also generate information about GC runs and time spent in GC.
4.3.2 Test with mobile device
To test how RoR handle performance when mobile devices is used I choose to test the Huawei P8 phone against the framework. I limited the test to the computationally complex structure. From what I could found there were no good test suite for the phone so I accessed the route and measured times by hand. I did 50 measures and calculated a mediate.
4.3.3 Stress tests
The stress tests is performed by the software httperf[16]. Httperf allows for concurrent rapid requests to a web server and simulating multiple sessions. Httperf then outputs information about the connection, request rates and response times. Httperf requested three different routes, one for each data structure mentioned in 4.2.4, 4.2.5 and 4.2.6. Httperf simulated 25 sessions, all sending 100 requests each adding up to a total of 2500 requests for each route.
4.3.3.1 Computationally complex data structure /partners is the route for the complex structure.
The request:
h t t p e r f −−hog −−s e r v e r 1 2 7 . 0 . 0 . 1 −−p o r t =3000
−−u r i =/ p a r t n e r s −−w s e s s = 2 5 , 1 0 0 , 0 . 1 −−r a t e =25
In this scenario the structure consisted of 80 partners and the biggest depth in the tree was eight. 4.3.3.2 Computionally semi-complex data structure
/partners/10/devices is the route for all devices belonging to partner 10. The request:
h t t p e r f −−hog −−s e r v e r 1 2 7 . 0 . 0 . 1 −−p o r t =3000
−−u r i =/ p a r t n e r s /10/ d e v i c e s / −−w s e s s = 2 5 , 1 0 0 , 0 . 1 −−r a t e =25 There are 20 partners to collect devices from. These partners have a total of 60 devices amongst them. The biggest depth of this branch is six.
4.3.3.3 Computionally simple data structure /devices/all fetches all devices as an array.
The request:
h t t p e r f −−hog −−s e r v e r 1 2 7 . 0 . 0 . 1 −−p o r t =3000
−−u r i =/ d e v i c e s / a l l −−w s e s s = 2 5 , 1 0 0 , 0 . 1 −−r a t e =25 There are 120 devices in the database.
Instituitionen f¨or datavetenskap METHOD
4.3.4 Active Record N+1 issue
To test the N + 1 issue, two tables were used. One table called partner users and the partners table described in 4.2.4.
partner users:
partner users
user id partner id
Table 5: Table structure for device locations table.
user id is a foreign key to a user and partner id is a foreign key to a partner. The users table is irrelevant for this evaluation. Two different code pieces were used to test this, the first one did not solve the N + 1 issue:
P a r t n e r U s e r . f i r s t (N ) . e a c h {
| p a r t n e r u s e r | p u t s p a r t n e r u s e r . p a r t n e r . name }
The other code piece solved the N + 1 isse:
P a r t n e r U s e r . i n c l u d e s ( : p a r t n e r ) . f i r s t (N ) . e a c h {
| p a r t n e r u s e r | p u t s p a r t n e r u s e r . p a r t n e r . name }
The : partner property is a foreign key to the partner users partner. The code prints the name of each partner belonging to the partner user fetched. N is the number of partner users which is fetched. These code fragments were tested with N = 1, 5, 10, 20, 30, 40, 50 and 60. For each N every test was run a total of 1000 times and a mediate of these results was calculated.
4.3.5 SQL statement creation
During this evaluation two different pieces of code were used. These code pieces had the same func-tionality but different ways of achieving this funcfunc-tionality.
The first fragment:
@pa rtne r = P a r t n e r . where ( name : ” T e l e n o r ” ) . f i r s t
@ u s e r i d s = @pa rtn er . p a r t n e r u s e r s . p l u c k ( : u s e r i d ) . f l a t t e n @ u s e r s = User . where ( i d : @ u s e r i d s )
The second fragment:
@ u s e r s = User . j o i n s ( : p a r t n e r ) . where ( p a r t n e r s : {name : ” T e l e n o r ” } )
Link¨opings universitet
Instituitionen f¨or datavetenskap METHOD
SELECT ” p a r t n e r s ” . ∗ FROM ” p a r t n e r s ” WHERE ” p a r t n e r s ” . ”name” = ?
ORDER BY ” p a r t n e r s ” . ” i d ” ASC LIMIT 1 [ [ ”name” , ” T e l e n o r ” ] ] SELECT ” p a r t n e r u s e r s ” . ” u s e r i d ” FROM ” p a r t n e r u s e r s ”
WHERE ” p a r t n e r u s e r s ” . ” p a r t n e r i d ” = ? [ [ ” p a r t n e r i d ” , 3 4 ] ] SELECT ” u s e r s ” . ∗ FROM ” u s e r s ” WHERE ” u s e r s ” . ” i d ” IN ( 1 3 , 1 4 , 3 3 )
The second fragment translated into this SQL code: SELECT ” u s e r s ” . ∗ FROM ” u s e r s ”
INNER JOIN ” p a r t n e r u s e r s ” ON ” p a r t n e r u s e r s ” . ” u s e r i d ” = ” u s e r s ” . ” i d ” INNER JOIN ” p a r t n e r s ” ON ” p a r t n e r s ” . ” i d ” = ” p a r t n e r u s e r s ” . ” p a r t n e r i d ” WHERE ” p a r t n e r s ” . ”name” = ? [ [ ”name” , ” T e l e n o r ” ] ]
What can be observed is that even though booth code fragments do the same, fetching all users belonging to a partner. The first fragment does it in three different queries whereas the second does it in one. The Ruby fragments were tested a 1000 times each and mediates were calculated.
4.3.6 Ruby on Rails caching
The caching test is performed on the complex structure under the same circumstances as the stress testing of this structure but with action caching activated.
Instituitionen f¨or datavetenskap RESULT
5
Result
5.1 Performance tests
5.1.1 Computationally complex data structure • Wall time: 133 ms
• Process time: 131 ms • GC runs: 1
• GC time: 6 ms
5.1.2 Computationally semi-complex data structure • Wall time: 94 ms
• Process time: 95 ms • GC runs: 0
• GC time: 0 ms
5.1.3 Computationally simple data structure • Wall time: 27 ms
• Process time: 29 ms • GC runs: 0
• GC time: 0 ms
5.2 Test with mobile device
• Average response time: 171.5 ms
5.3 Stress tests
5.3.1 Computationally complex data structure Summarised data:
• Size of response: 8301 B • Request rate: 12.3 requests/s • Average response time: 1909 ms
Link¨opings universitet
Instituitionen f¨or datavetenskap RESULT
5.3.2 Computationally semi-complex data structures Summarised data:
• Size of response: 5040 B • Request rate: 81.5 requests/s • Average response time: 194 ms
The complete result of the stress test can be seen under the appendice result of stress tests. 5.3.3 Computationally simple data structure
Summarised data:
• Size of response: 31966 B • Request rate: 35.2 requests/s • Average response time: 594 ms
The complete result of the stress test can be seen under the appendice result of stress tests.
5.4 Active Record N + 1 issue
The result of these following tests are summed up in this graph:
Active Record N+1 Issue
T ime (ms) 0 7,5 15 22,5 30 Samples (N) 1 5 10 20 30 40 50 60 N+1 Issue N+1 Solution
Figure 3: Comparisons between N+1 issue and N+1 solution. 30
Instituitionen f¨or datavetenskap RESULT
Already at 60 records, a rather small number for fetching data in a database, the solution to the N + 1 issue was more than four times faster than the issue.
5.5 SQL statement creation
• Unoptimised code execution time: 5.6 ms • Optimised code execution time: 4.9 ms
• Difference: Optimised was more than 14% faster
5.6 Ruby on Rails caching
Summarised data:
• Size of response: 8294 B • Request rate: 210.6 requests/s • Average response time: 9 ms
Instituitionen f¨or datavetenskap DISCUSSION
6
Discussion
6.1 Method
6.1.1 Literature
The literature is composed of scientific articles, web articles and technical documents. The scientific articles are articles published in one or more scientific journals and sources I deem credible. These are the thesis main source of information.
The scientific papers contribute with pure facts, however this information could from time to time be difficult to understand. This is where the web articles excel. The web articles is not used for new information, but rather contributes the same information in an easier to read context. These articles also added another perspective to the issue.
From the technical documents, I could extract information about how certain features operates, for example the RoR cache functionality. From this information conclusions about performance could be drawn.
6.1.2 Test system nginx/passenger configuration
When evaluating the RoR framework, the system used is important to take into consideration. The hardware is new and most likely good for this evaluation, however the operating system is not a server operating system, but one created for personal use. This probably means some extra unnecessary processes running at any given time for GUI and other unnecessary features for this evaluation. This could impede performance.
All the software used is one of the more recent releases. This test setup is a real configuration that will act like a real production environment. Nginx is a lightweight web server that if something should have a positive influence on performance compared to if a heavier web server was used.
The test data is in a sqlite3 database, possibly it could have been better to use a production database but I do not think it will have any big effect on performance.
6.1.3 Test system Ruby on Rails performance environment
This environment uses the same data and database as in the test system 6.1.2. This environment however generates other information about the test runs than 6.1.2, for example information about the garbage collection.
Since two different configurations is used for evaluation, it could be an issue if one is considerably faster than the other. Fortunately these configurations tasks differs and hopefully will not be an issue. 6.1.4 Test unit Huawei P8
A new phone with new hardware, since the computations are server-side, the client should not im-pede performance. This unit is connected through wireless network, this will have some impact on performance.
6.2 Evaluation
6.2.1 Performance tests
These tests were performed in the Ruby on Rails performance environment. These test results sets a baseline for the other tests, mostly the stress test to compare too. These will also show how effective
Link¨opings universitet
Instituitionen f¨or datavetenskap DISCUSSION
or not the GC in RoR and Ruby is.
These tests only measured time spent in the back-end and no involvement with the front-end. This gives a better estimate of performance for the framework than if the entire chain from back-end to front-end were to be measured.
These tests were tested a total of 1000 times and a mediate was measured, but these tests were performed one after one so the back-end could focus on one test at the time. Therefor no load was considered during these tests, the results could rather be seen as best possible expectation.
6.2.2 Test with mobile device
Firstly I tested the device against the RoR framework to get similiar results as the performance test, but it was not the exact same test procedure. These tests were tested per hand, noted and a mediate was calculated. This approach could differ from the actual performance environment, but I tried to make them as similar as possible. These tests were also only run 50 times compared to the 1000 times for the performance tests, this gives a less accurate mediate.
6.2.3 Stress tests
The stress tests were performed with 25 users acting at the same time, sending a new request when the previous one was responded too. 25 might seem like a rather small number, one of the reasons for this amount was that the operating system of the machine seemed rather repellent to a greater number of users. I would have preferred to simulate more than 25 users.
Even at 25 users the data gathered through these tests can still show how the RoR framework would scale, even for bigger applications with more data.
These users will be sending requests right after each response is received. However it will only send the same request every time. This will test how the the server will respond during load, but not how it would react during a real lite situation. It would be a lot more complicated setting up an actual scenario were the simulated users acted as real users.
6.2.4 Active Record N + 1 issue
These tests were also tested under the Ruby on Rails performance environment and in the same way as the performance tests. As in 1000 tests and then a mediate was calculated.
For these tests I do not think it is necessary to consider load at all, that would only make it harder to see what response times are depending on the N + 1 issue and the fact that there is load on the server.
6.2.5 SQL statement creation
These tests were tested under the Ruby on Rails performance environment, stress and load was not considered a factor. The interesting data was whether frequent changes between the Ruby and Active Record environments would impede performance. This was easier and more accurately tested in this environment.
6.2.6 Ruby on Rails caching
The caching tests tested the most complex structure 4.2.4 to see caching under the most extreme circumstances. That is a very server heavy test and the caching moves a lot of the work away from the back-end server to the nginx/passenger configuration. Since the nginx/passenger is involved this
Instituitionen f¨or datavetenskap DISCUSSION
test will also act more like a real world application.
This test setup might seem simple and could possibly have yielded better results if the test users acted more as real life users, but as said before is a lot harder to setup and conclusions can still be drawn from these tests.
There are other cache functionalities that could be tested and different cache stores too see which gives the best configuration. For this test the regular cache store was used.
6.3 Result
6.3.1 Performance tests
6.3.1.1 Computationally complex data structure
A rather positive result compared to my expectations. This is not a structure commonly used in web development and thus RoR could have been very poorly optimised for it. But the time 133ms is comparable to the semi complex structure at 94 ms. Another note to consider is that the company I did this for preferred good looking code over top performance and it is my belief that this solution could have been more effective.
Another issue with this result is that the test base might be to small, a larger test base could possibly scale very badly.
The garbage collection in this case use about 4% of the time. This garbage collection was in the theory described as poorly optimised, 4% is not that bad. However if it were to be used repeatedly in a bigger query the time could stack up.
6.3.1.2 Computationally semi-Complex data structure
This was actually worse than expected. There were less SQL queries performed and a smaller data set than 6.2.1.1 but it took almost the same time to get the response.
For some reason garbage collection does not even seem to be used here. If that is true, some rather heavy operations needs to be used to even enquire about garbage collection.
6.3.1.3 Computationally simple data structure
Compared to the other tests this was a somewhat expected result, considerably faster than both. Garbage collection does not seem used here.
6.3.2 Test with mobile device
On a mobile device these tests averaged at 171.5 ms compared to 133 ms from the performance tests for the same structure. This is a difference of about 29% slower. Some of this could come from the different ways these were tested, or maybe mobile devices just are slower. A big part probably comes from the wireless connection. It still is a small difference and completely acceptable.
6.3.3 Stress tests
6.3.3.1 Computationally complex data structure
Since the benchmark was sending a new heavy load requests to the server whenever a response was received I was expecting slower response times. But almost two seconds compared to 130 ms seems much. It is not exactly proportionally slower but 14 times slower with a total of 25 simultaneous
Link¨opings universitet
Instituitionen f¨or datavetenskap DISCUSSION
requests. The payload was rather small and should not be a limiting factor. The real performance issue here most probably is the amount SQL queries.
It should however be considered that this is not a request that should be requested as frequent as this, if it were the the case, caching is to recommend. So these times can be seen as somewhat of a worst case scenario.
6.3.3.2 Computationally semi-complex data structure
Excellent results, at 25 simultaneous requests the average response time was not even twice as slow compared to the performance test. The data was about half as big as the complex test but still these response times is really impressive. This could be considered somewhat of a standard request, that is one that will be asked quite frequently.
6.3.3.3 Computationally simple data structure
This was at 25 simultaneous requests almost 22 times slower, that is not good, however the payload is large compared to the other data structures which is probably a factor. The payload is almost 32 kB, that is a lot of data to send over a network.
Each request only creates one SQL query and no heavy calculations, so I think the size of the payload is what creates these slow response times.
6.3.4 ActiveRecord N + 1 issue
The result show that when fetching 60 objects through a foreign key, the unoptimised code is already 4 times slower than the optimised. As 60 objects is a rather small amounts of objects and as the graph shows, this problem increases a lot by each extra object, this issue should be taken seriously.
The good thing is that this issue is rather easy to solve, as shown in the code example, it is a single extra word in the code. As long as the programmer is aware of this issue and do not miss it, this should not be left untreated.
6.3.5 SQL statement creation
At only 2 additional SQL queries, the unoptimised code is 14% slower than the optimised code. This performance decrease will happen the more SQL queries are run. This does not seem as big a performance loss as the N + 1 issue, but could still hurt performance.
The issue here is that when the code base increase, writing code that utilize the best SQL solution will become harder and harder. If this is a really big issue, I would recommend another solution than manually write SQL queries. There are other frameworks that can be used rather than Active Record. 6.3.6 Ruby on Rails caching
These results clearly shows that caching if used correctly can be an extremely powerful utility. It can cut down a lot of unnecessary load from the server and in return respond a lot quicker to most requests.
The programmer needs to be well aware of how the code operates and what will in the application be frequently asked for. There is no idea in just caching everything. The correct items should be in the cache and it could be difficult to predetermine what these are.
Instituitionen f¨or datavetenskap CONCLUSIONS
7
Conclusions
If performance is of the utmost importance for the project, I am not certain that Ruby on Rails is the perfect fit. With that said, I do believe that it is possible to create a fast applications in Ruby on Rails. Active Records have some issues but can to an extent be fixed by clever programming. Ruby and RoR performs well during the tests overall and is not impeding performance too badly. Caching is a feature that really can improve performance in applications. It acts well when queried from mobile devices as well.
If the framework still do not seem fast enough, there are plenty of options and configurations. Active Record can be changed entirely to another framework. There exist multiple different caching solutions and different faster cache stores than the one tested in this thesis. There exist more perfor-mance focused Ruby implementations such as YARV and Rubinius.
7.1 Further work
There are several approaches to how the framework could be further tested. A good example would be to test it in a bigger environment, more simulated user and more data in the database, this could help giving an even more accurate picture of how the framework operates on that scale. To build on this, adding users on different platforms would further help testing this area.
There are also several other ways that would yield a deeper understanding of performance and RoR. The Ruby language could be more comprehensively tested. There are more cache methods to test and multiple different cache stores.
Instituitionen f¨or datavetenskap REFERENCES
8
References
[1] Geer, D. (2006). Will Software Developers Ride Ruby on Rails to Success? Computer 39(2), 18-20. [2] Caching with Rails: An overview, guides.rubyonrails.org/caching with rails.html
[3] Purer, K. (2009). PHP vs. Python vs. Ruby – The web scripting language shootout
[4] Bhat, S, M., Nair, G, D., Bansal, D., J, V., (2012). Data structure based performance evaluation of emerging technologies — A comparison of Scala, Ruby, Groovy, and Python. CSI Sixth International Conference on Software Engineering (CONSEG) 1/ 1/2012, 1-5.
[5] Keep, A., Chauhan, A., (2008). Concrete Partial Evaluation in Ruby. eScience ’08, 346-347. [6] You Used Ruby to Write WHAT?!, www.cio.com/article/2437037/developer/you − used − ruby − to − write − what − −.html
[7] Rei, L., Carvalho, S., Alves, M., Brito, J. (2007). A look at dynamic languages.
[8] Lessons Learned in Large Computations with Ruby, nirvdrum.com/2009/09/17/lessons−learned− in − large − computations − with − ruby.html
[9] Ruby 2.1 Garbage Collection: ready for production, samsaf f ron.com/archive/2014/04/08/ruby− 2 − 1 − garbage − collection − ready − f or − production
[10] Grust, T., Mayr, M. (2013). A Deep Embedding of Queries into Ruby. Data Engineering (ICDE), 2012 IEEE 28th International Conference on, 1257-1260.
[11] Ku, K-I., Kang, S-J., Chung M., Kim, W-Y., Choi, W. (2008). Evaluation and Optimization of ActiveRecord to implement the Personalized Software Service Platform. Advanced Communication Technology, 2008. ICACT 2008. 10th International Conference on (Volume:3 ), 1736-1766.
[12] Real world Rails, Part 3: Optimizing ActiveRecord, www.ibm.com/developerworks/library/wa− rails3/index.html
[13] Hibernate vs. Rails: The Persistence Showdown, www.theserverside.com/news/1364757/Hibernate− vs − Rails − T he − P ersistence − Showdown
[14] Ruby on Rails Security Guide, guides.rubyonrails.org/security.html [15] Ruby on Rails, rubyonrails.org
Instituitionen f¨or datavetenskap APPENDICES
Appendices
A
Result of stress tests
A.1 Computationally complex structure
Link¨opings universitet
Instituitionen f¨or datavetenskap APPENDICES
A.2 Computationally semi-complex structure
Figure 5: Caption
Instituitionen f¨or datavetenskap APPENDICES
A.3 Computationally simple structure
Link¨opings universitet
Instituitionen f¨or datavetenskap APPENDICES
B
Result of caching test
B.1 Computationally complex structure
Figure 7: Caption
!
