A comparison of performance in scripting for content creation in games

(1)

Bachelor of Science in Digital Game Development June 2019

A comparison of performance in

scripting for content creation in games

Gustav Pettersson

Faculty of Computing, Blekinge Institute of Technology, 371 79 Karlskrona, Sweden

(2)

The thesis is equivalent to 10 weeks of full time studies.

The authors declare that they are the sole authors of this thesis and that they have not used any sources other than those listed in the bibliography and identified as references. They further declare that they have not submitted this thesis at any other institution to obtain a degree.

Contact Information:

Author(s):

Gustav Pettersson

E-mail: gupe16@student.bth.se

University advisor:

Petar Jer˘cić

Department of Computer Science

Faculty of Computing Internet : www.bth.se

Blekinge Institute of Technology Phone : +46 455 38 50 00 SE–371 79 Karlskrona, Sweden Fax : +46 455 38 50 57

(3)

Abstract

Context. When working in a content creation software within a pipeline, some tasks can often become repetitive and time consuming. In order to reduce the time these tasks take to complete, scripting is often an option to automate the tasks. This thesis aims to investigate the performance of different scripting interfaces in such a software using Autodesk Maya and its scripting interfaces.

Objectives. The objectives of this thesis are to answer the questions if there is a clear performance difference between different scripting possibilities within Maya.

Methods. Using the different scripting languages available in Autodesk Maya, scripts were implemented and the time the execution of the scripts took were collected and analyzed and compared to see if there is a clear difference in their performance.

Results. The results from the tests show that there is a difference in performance between the different test implementations in the different languages, but the difference is more based on what way the interfaces are used rather than the interfaces themselves.

Conclusions. The conclusions that can be drawn from the results are that, while there is a difference in specific cases, the results are not enough to say that there is a general difference or gain in performance between the different interfaces. The results presented might be interesting for anyone working within Maya or it’s scripting interfaces.

Keywords: Scripting, Maya, Pipeline

i

(4)

(5)

Acknowledgments

The author would like to thank Petar Jer˘cić for the supervision and guidance through- out the work with this thesis.

iii

(6)

(7)

List of Figures

2.1 Visual example of the vertex iteration test . . . 6

2.2 What the script achieved . . . 8

2.3 Example of one version of the renaming scripts . . . 8

2.4 The script which created the objects to be renamed . . . 8

2.5 Example of one of the vertex iterations tests as run by the "Testing script" . . . 9

2.6 How the renaming tests were run by the testing scripts . . . 10

2.7 How the tests were run . . . 11

3.1 A table of averages of the vertex iteration tests. A green color in the column indicates that the time was faster, red indicates that it was slower. . . 13

3.2 The two comparisons for vertex iteration over 72 vertices for which the null hypothesis that they are equal could not be rejected . . . 14

3.3 Bar graph of the averages of the different vertex iteration tests. The Y-axis represents time in seconds. . . 14

3.4 Bar graph of the averages of the different xform versions of the vertex iteration tests. The Y-axis represents time in seconds. . . 15

3.5 Bar graph of the two lowest averages of the different versions of the vertex iteration tests. The Y-axis represents time in seconds. . . 15

3.6 Table showing the averages in seconds of the renaming test. . . 16

3.7 Bar graph showing the average speed of the rename test. The Y-Axis is the average speed measurement. . . 16

3.8 The result table of the T-tests for renaming . . . 17

3.9 The resulting average time in seconds of the matrix multiplication tests 18 3.10 The results of the T-tests between the groups of the matrix multiplication tests . . . 18

3.11 Bar graph representation of the averages. The Y-axis of the graph represents seconds. . . 18

3.12 Bar graph representation of the two lowest averages. The Y-axis of the graph represents seconds. . . 18

A.1 Table of T-test results for 8 vertices . . . 30

A.2 Table of T-test results for 8 vertices continued . . . 31

A.6 Table of T-test results for 242 vertices continued . . . 35 vii

(10)

B.1 The script to run the tests in MEL . . . 42

B.2 The script to run the tests in Python . . . 43

B.3 The Test script in MEL . . . 44

B.4 The Test script in MEL continued . . . 45

B.5 The Test script in Python . . . 46

B.6 The Test script in Python continued . . . 47

B.7 The Test script in Python continued . . . 48

B.8 The different test functions in MEL . . . 49

B.9 The different test functions in Python . . . 50

B.10 The different test functions in Python continued . . . 51

viii

(11)

Chapter 1 Introduction

Scripting is used to a wide extent in the creation of content for games and films.

While scripting could refer to somewhat different things in various contexts, for the purpose of this writing it will be used to describe code of varying length used to automate and/or improve manual tasks of varying difficulty or size. Such scripts can be a part of a “Pipeline”[16][17]. Pipeline will here be used as a general description of the process of creating assets that ends up in a game, certain concepts will be appli- cable to the creation of films, as the two productions can share aspects of production.

Improving a pipeline can be a very costly and time consuming process, with lots of hours of work put in to it and is subject to a lot of change to adapt to changes in software.[16] As such, when offered the possibility of doing the same thing in different ways via scripting, whether it be with different application programming interfaces(API) or languages within a software or two different softwares completely.

Getting the most performance out of existing scripts in such a pipeline would be something worth striving for in order to optimize a pipeline of any sort including scripts to its fullest by reducing the time it takes to complete a task.

It is also relevant for the people working within such a pipeline that the scripts at hand have relatively good performance. A quote from the book Maya python by Adam Mechtley and Ryan Trowbridge for games and films states this well:

“No artist would want to use a script that takes a few minutes to execute”[19]

In order to be able to have scripts perform well in a pipeline, there needs to be some knowledge of the performance of the scripting implementations available. Such possibilites of scripting to automate tasks are available in many different content creation softwares, such as Blender[1], zBrush[7] and Autodesk Maya.

Autodesk Maya is an industry standard software for creating content for games and films. It is used in many different areas of content creation for modeling, ani- mating, rendering and more. Maya is created in a way that allows users to extend its functionality further via plug-ins and scripting. These scripts and plugins can be created via a language specific to Maya, Maya embedded language(MEL), various Python modules which access the core code of Maya or wrap MEL, or plugins created

1

(12)

in Python or C++.

Python is a widely used language outside of Maya as well, it was added as an extra possibility for scripting usage in Maya 8.5[20]. It provides the user with some extra possibilities when scripting in Maya as it allows the user to import other modules that exists in Python. Below are some more in-depth explanation of the scripting capabilities available for use when scripting in Maya to best suit the needs of what the user is trying to achieve.

• MEL

MEL is an abbreviation for Maya Embedded language. It is a scripting language created solely for the purpose of scripting in Maya. MEL is used to implement many of the different tools that are accessible upon start up via the GUI[10].

• Maya cmds(maya.cmds)

Maya.cmds is a Python wrapper for MEL[13] added by Autodesk themselves.

This extends what MEL is by allowing a user to leverage the functionality of the Python language while at the same time having access to the functionality in MEL.

• Maya python API (maya.api.OpenMaya)

OpenMaya is a Python wrapper for the Maya C++ API. There are two versions of this in Maya 2018, maya.api.OpenMaya, referred to as Maya Python API 2.0, and maya.OpenMaya, referred to as Maya Python API 1.0. The latter of the two is an older version and the former is still under development. In Maya 2018, the 2.0 version does not have all of the functionality as 1.0[13]. OpenMaya can provide the user with more functionality than the other interfaces because it can access Maya on a lower level.

• PyMEL

PyMEL is one of the Python implementations shipped with Maya. It was created by a third party with the motivation that Maya’s current Python scripting possibilities were unpythonic and awkward to use. PyMEL has a lot of focusing on object-orientation and Python within Maya in order to make the scripting work in a more intuitive way. [4] PyMEL derives some commands from maya.cmds and it also make use of the OpenMaya module.

MEL and the Python modules listed above will for the purpose of this writing be referred to as "the interfaces".

Given the information that all of the above mentioned capabilities of scripting exists in Maya and it being an industry-standard software, making the most out of the interfaces available is something that is of relevance for anyone working with them. Thus, the purpose of this thesis is to investigate if there is a clear difference of performance between the different interfaces. The measurement of performance is the time it takes to execute a task with a script.

(13)

1.1. Related Work & Motivation 3

1.1 Related Work & Motivation

Pipelines are a subject of research for many big game and film studios, in order to improve the pipelines performance. Golembeski et al. show in their work "PipelineX:

A Feature Animation Pipeline on Microserves" [17] how they set up a pipeline of mi- cro services as well as the gains and benefits of the usage of such a pipeline. While there is no specific focus on the scripting within a pipeline in their work, it mentions scripting as being a part of pipelines.

Baillet et al. show another example of how they improved their pipeline in their work "Forging a New Animation Pipeline with USD"[14] which uses Maya in combination with other tools. Which illustrates that Maya is a software that is used and modified by its users to fit their pipeline needs, which motivates the usage of Maya as the scripting software in this thesis.

As for Maya, being an industry standard software with a large amount of users have been a subject of several books, one of these are "Maya Python for Games and Film: A Complete Reference for the Maya Python API" by Adam Mechtley and Ryan Trowbridge which introduces the user and dives a bit deeper into the scripting and plug-in capabilities of Maya with Python. In the Introduction section of this book there are some quick mentioning of the performance speed of MEL compared to the Python API[19] and the differences in the implementations, though it quickly moves on to other things.

The Autodesk documentation states that the performance of the Maya API will

"often" be better than that of the commands interfaces(MEL, maya.cmds), and that C++ API will provide much better performance. This is somewhat of a vague statement and though it mentions that the API will provide better performance in some cases[8], there is still not much information provided regarding the performance. The information does however point to OpenMaya as having the best performance of the interfaces tested for this thesis.

Other than some blog posts e.g[21] [18] there have not been much previous work done in the way of performance comparisons between the different interfaces. These blog posts, as well as the book, are at the time of writing a few years old, meaning that the interfaces may very well have changed since then.

The related work mentioned above is what motivates the use of Maya and it’s scripting for this thesis. Seeing as there is a lack of investigations of performances in previous work, the aim of this thesis is a more specific investigation of the performance of the interfaces available for the user when using Maya as a content creation software.

(14)

1.2 Research question

The research questions which this thesis aims to answer are:

• Does the time to execute tasks of the Python interfaces(OpenMaya, maya.cmds and PyMEL) show a clear difference?

• Are any of the Python interfaces faster in performance compared to MEL?

To answer these questions, scripts was created in Maya and tested. The scripts in the different interfaces complete the same tasks. The time it took for each of the scripts to execute was measured, compared and discussed. The results from these tests are presented and discussed.

(15)

Chapter 2 Method

This chapter will explain how the tests of the different interfaces were implemented and tested as well as how the data from these tests were collected and analyzed to provide the results given from this investigation.

2.1 Initial research

The first step for the work of this thesis was to study what work had previously been done in the area. This study did not lead to much of a result as there was no related work found except for some blog posts[21, 18] and forum discussions. Some related work can be found in the book by Adam Mechtley and Ryan Trowbridge where there is an almost anecdotal comparison between the time it takes for a script to run when implemented in MEL or OpenMaya in the introduction[19]. There is also the statement in the Autodesk documentation that OpenMaya "often" will perform better than the other interfaces [8]. Other than these mentioned items, no related research in the area of scripting performance comparison was found.

2.2 The tests

The following are the tests that was implemented for the performance testing. Be- cause of the lack of related work, choosing which tasks for the scripts to perform were based on different tasks that one could potentially find useful when working in a software such as Maya:

• Vertex Iteration Test

Iterating over the vertices of a mesh and changing their translation, a more visual example can be seen in fig 2.1. This was implemented because it is a common task in scripts such as these to want to iterate over any sort of relevant data. The changing of the translation of the vertices is just one example of something that such iteration can be useful for.

5

(16)

Figure 2.1: Visual example of the vertex iteration test

• Matrix Multiplication Test

Tests the speed of matrix multiplication in MEL, PyMEL and OpenMaya.

maya.cmds has no implementation of matrix multiplication and was therefore excluded from this test. This test was implemented because it is a common thing to want to do in many different 3D-graphics applications to want to multiply matrices together.

• Renaming test

Renames parts of all names in a selected hierarchy in Mayas outliner, if certain requirements are met. This test was implemented as the "real world usage"

test. While the other tests are mainly constructed to simply test performance, this test both compares performance, and has a real world application in that it renames something similar to what you may want to do when importing and exporting files between Maya and other softwares in a pipeline.

The tests were all set up in Maya’s script editor and run from there. First, each of the tests were written and tested individually with a time measurement to in order to see that everything was working as intended. A secondary script was then created that ran the tests and stored the time each test took and wrote the results to Comma-separated values(CSV) file. This script will be referred to as "The test script". The test script contained MEL and python functions and is what was run to gain the final results. The reason for writing the results to a CSV file was that the data could easily be put into Microsoft Excel for further analysis.

2.2.1 The vertex iteration tests

This test, in all the different versions, iterate over a mesh’s vertices and translates them +1 in the Y direction of the Maya scene. As with many other things in Maya, there are several ways of achieving this same result. The implementations used here to move the vertices are not all the different ways to move a vertex. These tests were set up in different ways in the different languages in order to test their performance, and also to see if there is a significant loss of using the different commands that are the same e.g xform and PolyMoveVertex. The different ways that was used to move

(17)

2.2. The tests 7 the vertices are explained in the list below.

• Xform

Is a command that is implemented in MEL, maya.cmds[12] and a version derived from maya.cmds in PyMEL[5]. This command can be used to access all kinds of different transformation data connected to an object within Maya.

• PolyMoveVertex

PolyMoveVertex is a command implemented in MEL, maya.cmds and a version derived from maya.cmds in PyMEL[6]. This command is used to move individual vertices. It takes a vertex as an input parameter and translates it according to flags sent along the command.

• Meshvertex.translateby

This is unique for PyMEL in the tests. It operates on the vertex as an object in Maya and translates the vertex based on input flags.

• OpenMaya Setpoint

This sets the position of a vertex accessed as an object. In order to use this in the same way as the other implementations, another function from the Open- Maya module called Getpoint was also used to get the current position[3], and the translation was added it’s current translation.

This test was run on meshes of different vertex count. The different vertex counts were 8, 72, 242, 520 and 1020. The mesh was for the testing a Cube with different subdivisions. The reason for testing on the different amounts of vertices was to see if there was an inconsistent difference in the languages when scaling the different tests.

It is also mentioned on the API introduction that OpenMaya would give better performance for bigger data sets, which this is meant to address[8]. The test was run 100 times for every vertex count being tested. And the total time for each of the tests is what was saved as a time measurement.

2.2.2 Renaming test

The renaming test renames all items below a selected item in a a selected hierarchy in Maya as it is seen in the "Ouliner" which is a UI element which shows the items in a Maya scene in a list. The first object in a hierarchy is selected and when the test is run all the items below it are renamed partially based on an input name to another input name. An input to this function with “ImportedName” and “RenamedName”

would change the name according to the example in figure 2.2. Figure 2.3 show one of the scripts versions

(18)

Figure 2.2: What the script achieved

Figure 2.3: Example of one version of the renaming scripts

The renaming test was also tested to see if it scaled well with more objects in Maya to rename. For the simplicity of the testing, a standalone script created joints in Maya which would act as the targets to be renamed. The intervals of the renaming tests were 50, 100 and 900. Initially, the final interval for this test was 1000, but one of the renaming scripts would not run if there were too many joints to rename. The test was run 100 times for each of the different test intervals.

The objects which this test was run on were created with a very simple MEL script which creates joints in the Maya scene in a hierarchy. The script is shown in figure 2.4 below. In figure 2.4, a number of 50 joints are created as an example.

Figure 2.4: The script which created the objects to be renamed

2.2.3 The matrix multiplication test

The matrix multiplication test run a function that multiply together matrices, the matrices that are multiplied together are passed in to the test function, the matrices are then multiplied together 100 times in each test.

This test was implemented in OpenMaya, PyMEL and MEL because these are the only ones of the four interfaces that have matrix multiplication implemented when shipped with Maya. This test was run in a slightly different manner, than the others.

Because the initial testing proved to have very low run-times, most of the time being 0, a matrix multiplication is done 100 times per test and the test are run 1000 times.

The reason for running this test 1000 times are to get many points of measurements for the later comparison in Excel as explained in section 2.4, as they are still very low.

(19)

2.2. The tests 9

2.2.4 The testing script

A secondary script was set up that had functions for running the different tests.

The testing script sets the scene to a state so that the tests can be run, starts the timer, runs the test, stops the timer and stores the run time value. The difference between the Python and MEL version of this script is that the Python version stores the timer’s values in lists and then prints them to a CSV file after the test has run, whereas the MEL version prints the timer values directly to file.

For the vertex iteration test in Python, the testing script first creates a list for all of the different tests run-times to be stored in. It then runs a loop which, for each test, creates a new cube for the vertex iteration test to run on. The test was created in this way so that each of the tests iterations when run are run under the same circumstances, in this case meaning a completely new Cube of a certain subdivision.

What differs between the MEL and the Python version are that MEL writes the timed results to file instantly inside the function instead of saving them to a list first.

Figure 2.5: Example of one of the vertex iterations tests as run by the "Testing script"

The number of iterations, as well as the subdivision of a cubes Y- and X- axis(which sets the vertex count) are sent as input parameters to this test. After the vertex iteration test that is currently being run has finished, the time it took to run is stored in its respective list if it is the Python version, or printed to file if it is the MEL version, the cube is then removed from the scene.

The renaming tests are run in another function which iterates a number of times based on an input parameter. For every iteration it starts a timer, runs the rename test, stops the timer, and store the timer values in the same way as for the vertex iteration test, both in the Python versions and the MEL versions.

For each new version of the tests within the loop the names of the targets for being renamed are named so that the next test can be run, e.g PyMEL version renames from "Character1" to "Character2", the maya.cmds version then renames from "Character2" to "Character3" and the OpenMaya version renames back to

"Character1" from "Character3". This allowed the script to loop through the iterations in the same fashion as the vertex iteration test.

When the initial testing as described in section 2.2 was performed, the his- tory, which is in many cases stored for objects in Maya, of the object being renamed seemed to affect the performance of the script. In order to mitigate this the

(20)

"cmds.DeleteHistory()" in Python and "DeleteHistory" in MEL was added which seemingly solved the problem.

Figure 2.6: How the renaming tests were run by the testing scripts

2.3 The timer

The timer used for the testing was a command implemented in MEL and the maya.cmds module called timerX, it is stated on the documentation page of MEL and its corresponding Python version that this is useful for timing the length of operations.[11].

The maya.cmds version was used for the python tests and for the MEL tests the MEL version was used. While there are different ways of timing the performance with other timers or using profilers, "timerX" was deemed as sufficient for the testing as it is implemented in both maya.cmds and MEL.

A Python profiler was initially used to measure the time of the tests, but in order for it to be used, the MEL scripts had to be called from Python. This was not something that would represent how MEL is used by it self and might change the result from an actual usage scenario of MEL which could have affected the results.

2.4 Running the tests and collecting the data

After all the test functions had been completed, the tests were run from Maya’s script editor in the same way in both Python and MEL. The figure 2.7 below show the Python example. The script shown runs the tests in sequence and outputs the information to the a file. During the time which the tests were run testing system was not used for anything else so as to not affect the results of the tests. The string

"path" of the input parameters is a file path to where the output CSV file should be placed.

(21)

2.5. The testing system 11

Figure 2.7: How the tests were run

After the tests had finished running, the data from the written CSV file was imported to Microsoft Excel and analyzed using analysis of variance test(ANOVA) using the Analysis ToolPak[2], the ANOVA tests showed unequal variances in all cases and T-Tests assuming unequal variance with a null hypothesis of the tests being equal was run for each of the different test pairs.

All the code used for the tests can be found in appendix B.

2.5 The testing system

The tests were all run on a Lenovo Ideapad Y700 laptop running Maya version 2018.

The following are the specifications of the CPU, memory and operating system of the computer.

• Processor: Intel i7-6700HQ, 2.60GHz, 4 cores

• Operating system: Windows 10 version 10.0.17763

• 8 GB 2133 MHz DDR4 RAM

(22)

(23)

Chapter 3 Results

In this chapter the averages of the tests run times and their corresponding results from T-Testing will be presented. This is what will be considered as the measurement of performance. Because of the size of the full tables of the t-tests, some results will be presented in this chapter as simplified graphs showing the averages of the test runs. The entire tables of the T-tests can be found in Appendix A. All the averages are shown as tables in this chapter in figures 3.3 for the vertex iteration test, 3.6 for the renaming test and 3.9 for the matrix multiplication test.

3.1 The vertex iteration results

OpenMaya were the fastest by average for all of the different vertex counts except for 8 vertices where MEL xform and maya.cmds xform had the lowest average. Figure 3.1 shows the average speed for all of the tests of the vertex iteration.

Figure 3.1: A table of averages of the vertex iteration tests. A green color in the column indicates that the time was faster, red indicates that it was slower.

With a null hypothesis of all of the results data being equal, the T-tests showed with a P-value of less than 0.05 certainty that all the different pairs of tests were significantly different for vertex counts 242, 520 and 1020, see appendix A for the entire tables of T-test results. For 72 vertices, the difference between MELPoly- MoveVertices and CMDSPolyMoveVertices versions is not enough to reject the null hypothesis, which can be seen in figure3.2, but in all other cases it is.

13

(24)

Figure 3.2: The two comparisons for vertex iteration over 72 vertices for which the null hypothesis that they are equal could not be rejected

In the 8 vertex iteration tests the results showed in many more cases that the speed of the tests were not different enough to say with certainty that the performance is different, for the full list of results for 8 vertices, see A.1 and A.2 which illustrates all of the T-test pairs.

Figures 3.3 - 3.5 further illustrate the difference in performance of the different tests. These results in combination with the t-tests show that, with overall certainty, there is in fact a difference in performance between the test versions of the vertex iterations scripts. The results show that OpenMaya is the best performing one and that the different PolyMoveVertex versions are much slower.

In order to further illustrate the differences in average speeds, figure 3.4 and 3.5 show the lower average results as separate graphs as the bars are practically invisible in the first graph with all the averages.

Figure 3.3: Bar graph of the averages of the different vertex iteration tests. The Y-axis represents time in seconds.

(25)

3.2. The renaming test results 15

Figure 3.4: Bar graph of the averages of the different xform versions of the vertex iteration tests. The Y-axis represents time in seconds.

Figure 3.5: Bar graph of the two lowest averages of the different versions of the vertex iteration tests. The Y-axis represents time in seconds.

3.2 The renaming test results

The MEL version of the renaming test was the fastest in all test intervals. Figure 3.6 shows the averages of the tests of all the rename tests.

(26)

Figure 3.6: Table showing the averages in seconds of the renaming test.

Figure 3.7 show the results of the rename tests averages in a more visual manner.

Figure 3.7: Bar graph showing the average speed of the rename test. The Y-Axis is the average speed measurement.

The results from the renaming tests when put together and analyzed with T-tests show in all cases except two that there is a difference in performance of the different tests, see fig 3.8 for the results of the t-tests.In the cases of maya.cmds and the Open- Maya version, the statistical differences are not enough to reject the null hypothesis of the two being equally fast in performance for 50 and 100 renames but in the case of 900 the difference is convincing.

(27)

3.2. The renaming test results 17

Figure 3.8: The result table of the T-tests for renaming

Seeing that MEL is the fastest one and there is a significant difference between all the other tests(as illustrated by figure 3.8), the results of this test show that MEL has the best performance.

(28)

3.3 The matrix multiplication test results

The results of the Matrix multiplication test show that OpenMaya is the fastest implementation of matrix multiplication. The T-test shows that the null hypothesis of the results being equal can be rejected with great certainty. Figure 3.9 show tha average time in seconds of the tests. Figure 3.10 illustrates the T-tests results between the MEL, OpenMaya and PyMEL implementations. The graphs in figures 3.11 - 3.12 illustrates the differences in a more visual manner.

Figure 3.9: The resulting average time in seconds of the matrix multiplication tests

Figure 3.10: The results of the T-tests between the groups of the matrix multiplication tests

Figure 3.11: Bar graph representation of the averages. The Y-axis of the graph represents seconds.

Figure 3.12: Bar graph representation of the two lowest averages. The Y-axis of the graph represents seconds.

(29)

Chapter 4 Analysis and Discussion

The results given from the tests that was run for this thesis show that there is a difference in performance between the different languages, which answers the first research question:

• Does the time to execute tasks of the Python interfaces(OpenMaya, maya.cmds and PyMEL) show a clear difference?

The answer to this question is yes, in these test cases. However, the results are not general enough to claim that one of the languages have better performance than the others. The results points to OpenMaya as having the best performance, which could very well be the case, given that it is the closest to the core of the software and according the the Maya help page "Often" provide better performance.[8].

The second research question:

• Are any of the Python interfaces faster in performance compared to MEL?

The results show that in two out of the three tests that OpenMaya have better performance than MEL, but MEL still has better performance than the rest of the interfaces in some cases. This question would therefore require more research to be answered. On the basis of the results of this investigation, the question is unanswered.

What can be stated from the results from the vertex iteration test, which is the main contributor of results, are that in a usage scenario where the scripts performance matters, it is more important how something is implemented using the interfaces, rather than which interface is used. Such as in the case of PolyMoveVertex compared to xform, where xform had significantly faster run times in all the 3 interfaces it was implemented in except for the 8 vertices iteration test.

It should also be taken into consideration that while the different interfaces can in many ways achieve the same results, they are created with different purposes.

OpenMaya provides a user with more functionality for, in some cases, more complex implementation, wheras the others provide some shortcuts to tasks which would in OpenMaya be needed to implemented by the user. One example of this is the case of the renaming script which is included in the appendix B, where the function to get all children are implemented for OpenMaya but come as implemented features in MEL, PyMEL and maya.cmds. PyMEL also has the advantage in implementation of

19

(30)

being created with object oriented programming in mind, thus being easier to work with for someone who is used to object oriented programming, compared to MEL which is documented by Autodesk as something that can "trip up" programmers[9]

with it’s syntax.

The fact also stands that the time measurements compared in the tests that were run are often very low, meaning less than a second. This can show that there is significant difference statistically. The statistical difference is not something that a user of a script would ever notice in a similar case. One example can be seen in figure 3.8 between the MelRename and OpenMaya rename. The difference is significant when analyzed with a T-test(P value of 2.60562E-11) but as a user would not necessar- ily notice the 0.0343 second average difference. This means, when working with the software to create content that optimizing should be done when optimizing is needed.

The only case where the time is significant is that of the PolyMoveVertex implementation which scales incredibly poorly compared to the other implementations.

This way of moving the vertices can however be used in a different way with list slicing in Python and could show different results when used in another way.

(31)

Chapter 5 Conclusions and Future Work

The conclusions drawn from the work presented in this thesis are that the different interfaces have some difference in their performance, but it cannot be said definitively if either of them have generally better performance. The results points towards OpenMaya as having the best performance, which is very much in line with what is stated in the documentation as stated in section 1.1, that it will "often" have better performance[8]. According to the results, there is also a hint that unless there is something needed for the task at hand which is not implemented in maya.cmds or MEL, either should be chosen over PyMEL. According to the results, MEL should be the first choice, based on only performance, it does of course depends on which syntax the creator of the script is most comfortable in. The results presented in this thesis might be of interest for anyone working within Maya and it’s scripting interfaces.

The results given in this thesis only show a difference in performance between the different tests that was implemented. Changing the implementation might yield very different results. The most useful conclusion to be drawn from the results given are that it is more important to choose which features within either of the interfaces to use rather than which of the interfaces themselves.

Because of the many different tasks which scripting can be used for further research would be needed in order to gain more definitive answers about the performance of the different interfaces. Considering that Maya is a software with a lot of possibilities for scripting, the results of further investigation of the kind presented in this thesis will more than likely not yield results in general, but only for the specific tests done.

Since the results of further testing similar to the method presented in this thesis would most likely provide similar answers when trying to answer the questions in a general manner, focusing the performance of scripts in a certain area of content creation could provide more beneficial answers to the various questions. This could also be tested in a way that would allow answers to questions when using two or more different softwares rather than being specific to Maya. For example a script that modifies something within a scene in Blender and Maya could be compared, as this would provide a more general answers to questions regarding scripting in a Pipeline.

21

(32)

Another aspect that would perhaps be more suitable for further work is the actual usage of the interfaces compared to each other. An example of this would be a user study of what users think are more readable or easy to use could be conducted, another case study which can inspire how to perform such a study was made by Bar- išić et al.[15]. In that study usage of languages are compared. Such a study would of course need to be changed to fit the purpose of the interfaces used in Maya or another software, but following the methodology used therein could give answers to some questions about the user interaction with the interfaces.

(33)

Chapter 6 Limitations

As a pipeline can take many different forms for different purposes and include many different procedures and softwares some limitations must be set. Though the only comparisons that was made was within Autodesk Maya, there are other softwares capable of extension via scripting to suit different pipelines such as Blender[1]. For the scope of this thesis only Maya and it’s Python modules and MEL implementations was researched as the scripts within a pipeline.

In Maya a user can also create C++ plugins with the same functionality as the scripts tested in this study. This is not included in the testing for performance because it is already documented that it would have better performance than both Python and MEL on the official Autodesk Website.[8]

Maya python API 1.0 was excluded from testing. The reason for this was that it is stated on the Maya help Page that Maya API 2.0 has better performance, while missing some functionality of 1.0.[13]

Investigating the performance gains or losses of the different implementations of scripting languages within Maya is a fairly hard thing to do as there are often several different ways of doing the same thing even within the same language implementation(MEL or Python) as well as because of Mayas extensive scripting possibilities. It was therefore required for the scope of this specific investigation to limit the testing to a smaller amount of tests. The tests are limited both in amount of tests and by the scope of what they test.

The matrix multiplication was very limited due to the fact that there is no implementation of the maya.cmds version. This is something that could have been further tested by implementing matrix multiplication in python for maya.cmds for the testing purpose.

Further limitations include the amount of measurements in each of the tests. In the cases where the tests does not show statistical difference, running more tests might prove that the tests, in all cases, have a statistical difference.

One final limitation of this thesis is that the tests were only run on a single computer and the results can therefore not provide a general answer to the questions.

Running the tests on several different computers might yield further results or show 23

(34)

a difference in the results that are gained.

(35)

References

[1] Blender Manual blender introduction. https://docs.blender.

org/manual/en/latest/advanced/scripting/introduction.html#

general-information. Last Accessed: 2019-05-16.

[2] Microsoft office support page use the analysis toolpak to perform complex data analysis. https://support.office.com/en-us/article/

use-the-analysis-toolpak-to-perform-complex-data-analysis-6c67ccf0-f4a9-487c-8dec-bdb5a2cefab6?

NS=EXCEL&Version=90&SysLcid=1033&UiLcid=1033&AppVer=ZXL900&

HelpId=xladdin.chm1793&ui=en-US&rs=en-US&ad=US. Last Accessed:

2019-05-16.

[3] Openmaya 2.0 documentation. {http://help.autodesk.com/view/MAYAUL/

2018/ENU/?guid=__py_ref_class_open_maya_1_1_m_fn_mesh_html}, note=Last Accessed: 2019-05-16.

[4] Pymel documentation: Pymel for maya. https://help.autodesk.com/

cloudhelp/2018/JPN/Maya-Tech-Docs/PyMel/index.html. Last Accessed:

2019-05-19.

[5] Pymel documentation: pymel.core.general.xform. {https://help.autodesk.

com/cloudhelp/2018/JPN/Maya-Tech-Docs/PyMel/generated/functions/

pymel.core.general/pymel.core.general.xform.html?highlight=xform}, note=Last Accessed: 2019-05-19.

[6] Pymel documentation: pymel.core.modeling.polymovevertex. {https:

//help.autodesk.com/cloudhelp/2018/JPN/Maya-Tech-Docs/PyMel/

generated/functions/pymel.core.modeling/pymel.core.modeling.

polyMoveVertex.html}, note=Last Accessed: 2019-05-19.

[7] Zbrush docs: Zscripting. {http://docs.pixologic.com/user-guide/

customizing-zbrush/zscripting/}, note=Last Accessed: 2019-05-21.

[8] AUTODESK.Help. Introduction. http://help.autodesk.com/view/MAYAUL/

2018/ENU/?guid=__files_API_Introduction_htm. Last Accessed: 2019-05- 19.

[9] AUTODESK.Help. Mel for programmers. https:

//knowledge.autodesk.com/support/maya/learn-explore/

caas/CloudHelp/cloudhelp/2018/ENU/Maya-Scripting/files/

GUID-0F7C50D1-FF45-4868-8EBC-FE044F54B82E-htm.html. Last Accessed:

2019-05-21.

25

(36)

[10] AUTODESK.Help. timerx documentation. hhttps:

//knowledge.autodesk.com/support/maya/learn-explore/

caas/CloudHelp/cloudhelp/2018/ENU/Maya-Scripting/files/

GUID-60178D44-9990-45B4-8B43-9429D54DF70E-htm.html. Last Accessed:

2019-05-19.

[11] AUTODESK.Help. timerx documentation. {http://help.autodesk.com/

cloudhelp/2018/ENU/Maya-Tech-Docs/Commands/timerX.html}, note=Last Accessed: 2019-05-16.

[12] AUTODESK.Help. xform documentation. {https://help.autodesk.com/

cloudhelp/2015/ENU/Maya-Tech-Docs/Commands/xform.html}, note=Last Accessed: 2019-05-16.

[13] AUTODESK.Help. Python in maya. https://

knowledge.autodesk.com/support/maya/learn-explore/caas/

CloudHelp/cloudhelp/2018/ENU/Maya-Scripting/files/

GUID-C0F27A50-3DD6-454C-A4D1-9E3C44B3C990-htm.html, 2018. Last Accessed: 2019-05-19.

[14] Aloys Baillet, Eoin Murphy, Oliver Dunn, and Miguel Gao. Forging a new animation pipeline with usd. In ACM SIGGRAPH 2018 Talks, SIGGRAPH

’18, pages 54:1–54:2, New York, NY, USA, 2018. ACM.

[15] Ankica Barišić, Vasco Amaral, Miguel Goulão, and Bruno Barroca. Quality in use of domain-specific languages: A case study. In Proceedings of the 3rd ACM SIGPLAN Workshop on Evaluation and Usability of Programming Languages and Tools, PLATEAU ’11, pages 65–72, New York, NY, USA, 2011. ACM.

[16] Cristian S. Calude, Alasdair Coull, and J. P. Lewis. Can we solve the pipeline problem? In Proceedings of the Fourth Symposium on Digital Production, DigiPro ’14, pages 25–27, New York, NY, USA, 2014. ACM.

[17] Dan Golembeski, Ray Forziati, Ben George, and Doug Sherman. Pipelinex: A feature animation pipeline on microservices. In Proceedings of the ACM SIG- GRAPH Digital Production Symposium, DigiPro ’17, pages 1:1–1:4, New York, NY, USA, 2017. ACM.

[18] mark. Mel vs pymel vs openmaya vs maya.cmds vs c++. http://radks.

blogspot.com/2012/08/performance-comparison-mel-vs-pymel-vs.html, 8 2012. A blog post, Last Accessed: 2019-05-21.

[19] Adam Mechtley and Ryan Trowbridge. Maya Python for Games and Film: A Complete Reference for the Maya Python API, chapter INTRODUCTION, page xxii. CRC Press, 2011. ISBN: 9780123785787.

[20] Adam Mechtley and Ryan Trowbridge. Maya Python for Games and Film:

A Complete Reference for the Maya Python API, chapter INTRODUCTION, page xv. CRC Press, 2011. ISBN: 9780123785787.

(37)

References 27 [21] Yantor3d. cmds vs. openmaya. https://yantor3d.wordpress.com/2017/08/

11/cmds-vs-openmaya/, 8 2017. A blog post, Last Accessed: 2019-05-21.

(38)

(39)

Appendix A

T-Test results from Vertex Iteration test

29

(40)

Moving 8 Vertices

Variable 1 MELPolyMoveVertices MELPolyMoveVertices MELPolyMoveVertices

Variable 2 MELXForm CMDSPolyMove CMDSXForm

t Stat 3.701716183 -1.613421778 3.701716183

t Critical two-tail 1.979124109 1.972595079 1.979124109

P(T<=t) two-tail 0.000319763 0.108321302 0.000319763

Variable 1 MELPolyMoveVertices MELXForm MELXForm

Variable 2 PyMELXForm CMDSPolyMove CMDSXForm

t Stat -0.416207096 -5.016138902 0

P(T<=t) two-tail 0.677711543 1.91722E-06 1

Variable 1 MELXForm CMDSPolyMove CMDSPolyMove

Variable 2 PyMELXForm CMDSXForm PyMELPolyMoveVertices

t Stat -4.05152882 5.016138902 -0.36346952

P(T<=t) two-tail 8.94938E-05 1.91722E-06 0.716647913

Variable 1 CMDSXForm CMDSXForm CMDSXForm

Variable 2 PyMELPolyMoveVertices PyMELTranslateBy OpenMaya

t Stat -6.096997698 -4.209603749 -0.260489338

P(T<=t) two-tail 1.33112E-08 4.84807E-05 0.794768431

Variable 1 PyMELPolyMoveVertices PyMELTranslateBy PyMELTranslateBy

Variable 2 PyMELXForm OpenMaya PyMELXForm

t Stat 1.679512753 3.956403706 1.79096E-15

P(T<=t) two-tail 0.094629371 0.000121571 1

Variable 1 MELPolyMoveVertices MELPolyMoveVertices CMDSPolyMove

Variable 2 PyMELTranslateBy OpenMaya OpenMaya

t Stat -0.425620477 3.457820374 4.80433983

P(T<=t) two-tail 0.670846677 0.00072444 4.39665E-06

Variable 1 MELXForm MELXForm PyMELPolyMoveVertices

Variable 2 PyMELTranslateBy OpenMaya PyMELTranslateBy

t Stat -4.209603749 -0.260489338 1.713642275

P(T<=t) two-tail 4.84807E-05 0.794768431 0.08817547

Figure A.1: Table of T-test results for 8 vertices

(41)

31

Variable 1 MELPolyMoveVertices Variable 2 PyMELPolyMoveVertices

t Stat -2.132671161

t Critical two-tail 1.972141222 P(T<=t) two-tail 0.034195899

Variable 1 MELXForm

Variable 2 PyMELPolyMoveVertices

t Stat -6.096997698

t Critical two-tail 1.979763763 P(T<=t) two-tail 1.33112E-08

Variable 1 CMDSPolyMove Variable 2 PyMELTranslateBy

t Stat 1.225789336

Variable 1 CMDSXForm

Variable 2 PyMELXForm

t Stat -4.05152882

Variable 1 OpenMaya

Variable 2 PyMELXForm

t Stat -3.815535924

Variable 1 CMDSPolyMove Variable 2 PyMELXForm

t Stat 1.204348428

t Stat 5.824948854

Figure A.2: Table of T-test results for 8 vertices continued

(42)

Moving 72 Vertices

Variable 2 MELXform CMDSPolyMoveVertices CMDSXForm

t Stat 41.06938778 -0.490982283 40.86165982

P(T<=t) two-tail 7.905E-99 0.623985475 1.94158E-98

Variable 1 MELPolyMoveVertices MELXform MELXform

Variable 2 PyMELXform CMDSPolyMoveVertices CMDSXForm

t Stat 32.45531418 -43.06160349 -0.183347481

P(T<=t) two-tail 1.50837E-80 7.241E-102 0.854712996

Variable 1 MELXform CMDSPolyMoveVertices CMDSPolyMoveVertices

Variable 2 PyMELXform CMDSXForm PyMELPolyMoveVertices

t Stat -6.559358761 42.84430314 -16.0008389

P(T<=t) two-tail 4.69629E-10 1.7754E-101 2.35245E-36

t Stat -63.29841328 -28.91517185 3.507550801

P(T<=t) two-tail 3.8994E-119 4.54813E-72 0.000581105

Variable 2 PyMELXform OpenMaya PyMELXform

t Stat 51.82459864 39.85115951 20.53580835

P(T<=t) two-tail 6.3825E-102 3.91951E-92 1.05984E-49

Variable 1 MELPolyMoveVertices MELPolyMoveVertices CMDSPolyMoveVertices

t Stat 14.82709929 52.80964198 56.28697655

P(T<=t) two-tail 9.47618E-34 2.1947E-107 2.4538E-115

Variable 1 MELXform MELXform PyMELPolyMoveVertices

Variable 2 PyMELTranslateBy OpenMaya PyMELTranslateBy

t Stat -29.13053825 3.290725909 35.22997515

P(T<=t) two-tail 1.41645E-72 0.001219174 5.63201E-84

(43)

33

t Stat -15.73747682

Variable 1 MELXform

t Stat -63.56258276

Variable 1 CMDSPolyMoveVertices Variable 2 PyMELTranslateBy

t Stat 15.99888089

Variable 2 PyMELXform

t Stat -6.381484941

t Stat -10.77391831

Variable 1 CMDSPolyMoveVertices Variable 2 PyMELXform

t Stat 33.98932456

t Stat 88.26032922

(44)

Moving 242 Vertices

Variable 2 MELXform CMDSPolyMoveVertices CMDSXForm

t Stat 377.7287062 -3.4131794 361.0740599

P(T<=t) two-tail 9.5033E-184 0.000836951 1.1255E-204

Variable 1 MELPolyMoveVertices MELXform MELXform

Variable 2 PyMELXform CMDSPolyMoveVertices CMDSXForm

t Stat 322.4623688 -695.6535817 -3.049374532

P(T<=t) two-tail 2.5877E-235 4.4028E-297 0.002641308

Variable 1 MELXform CMDSPolyMoveVertices CMDSPolyMoveVertices

Variable 2 PyMELXform CMDSXForm PyMELPolyMoveVertices

t Stat -20.61004453 611.4263184 -53.11719549

P(T<=t) two-tail 6.22329E-45 0 1.4867E-115

t Stat -589.7822907 -140.6374861 11.19927887

P(T<=t) two-tail 3.7746E-299 5.5171E-198 7.20798E-23

Variable 2 PyMELXform OpenMaya PyMELXform

t Stat 488.6021416 152.0796241 93.69266962

P(T<=t) two-tail 4.7237E-307 1.5038E-204 2.4895E-144

Variable 1 MELPolyMoveVertices MELPolyMoveVertices CMDSPolyMoveVertices

Variable 2 PyMELTranslateBy OpenMaya PyMELXform

t Stat 290.9264587 367.1341341 485.1522055

P(T<=t) two-tail 2.0091E-182 1.173E-205 3.6503E-289

Variable 1 MELXform MELXform PyMELPolyMoveVertices

t Stat -168.29813 9.824335242 598.0152707

P(T<=t) two-tail 1.033E-208 1.71918E-18 1.1381E-301

(45)

35

t Stat -38.2933301

Variable 1 MELXform

t Stat -649.4757096

Variable 1 CMDSPolyMoveVertices Variable 2 PyMELTranslateBy

t Stat 511.3509962

t Stat -16.76714625

Variable 1 OpenMaya Variable 2 PyMELXform

t Stat -25.78284057

Variable 1 CMDSPolyMoveVertices Variable 2 OpenMaya

t Stat 620.597385

t Critical two-tail 1.972079034

P(T<=t) two-tail 0

Variable 1 PyMELPolyMoveVertices Variable 2 PyMELTranslateBy

t Stat 498.729577

A comparison of performance in scripting for content creation in games