Unity Mono – A Very Simple Benchmark

When I say the Mono version that ships with Unity is slow, people get curious. I’d say it’s substantially slower compared to the recent versions of the MS .NET Framework. This is a quick post so I’m not going into the details of individual numbers for the IL or JIT or CLR overhead.

If you’re not familiarized with JIT-compiled languages, each have it’s own peculiarities. In case of C#, along with other .NET languages, it’s compiled into an intermediary language called IL. The IL code is JIT-compiled at execution time by a “virtual machine” (CLR in case of .NET) which I think technically can be considered an interpreter of kinds (if that helps you wrap your head around it) that outputs native code. This happens on-the-fly on a per-function basis (in most cases), and can cause the infamous “JIT spikes” the first time you call a function – but there are ways to “pre-JIT” your functions beforehand to prevent that. The advocates of high-performance JIT-compiled language promise they are capable of outputting native code that’s even faster than fully compiled binaries because the JIT compiler can ‘tune’ its output specifically for each architecture, unfortunately in real-life applications they’re never faster than hand-optimized C++ code, and the only example I’ve seen of that is the Farseer physics engine, but it’s heavily hand-optimized so you can’t compare it directly.

I’ve benchmarked addition, subtraction, multiplication, division and modulo operations with uints+uints and floats+uints types (all 32 bit). Here are the results (seconds, Unity 5.3.5f1 and .NET 4.6.1 on my testing rig):
unitymono

Analyzing that graph you can notice that both lines are more or less parallel until the last 2 results. In other words, the distance between them is roughly ‘absolute’; that’s an evidence of VM overhead. Unity took about 5 extra seconds to compute each of these tests. In the last 2 tests we got mixed results, but I was expecting those operations to be much slower than to the other ones. If we had a literal value in the operations there, for example x/2, the IL compiler probably could optimize it to use multiplication instead, but since it’s a variable value that’s not possible, so MultFloat costing the same as DivFloat in Mono is a mystery to me.

Perhaps the explanation is that Mono is ignoring the instructions NOT to optimize the functions, and it’s applying its very poor IL optimization to it anyway, that is an especially valid hypothesis when you allow the IL compiler to optimize the code, in which case the Mono numbers are exactly the same, but the numbers from MS .NET are very different:
optimized
In the last result Mono is 24 times slower, however, the last 5 results shouldn’t be used for comparison without inspecting the IL code. Please note this is 100% automated, I just commented out the “MethodImpl” attributes in the code below… Without a careful analysis I can only make conjectures: maybe the IL compiler was smart enough to discard the operations altogether; maybe the JIT compiler is using some extended instruction set (very unlikely to make that difference).

In the future I’m going to do more extensive tests (collections and whatnot) and look at the IL code generated by both compilers to compare them. I think this synthetic benchmark doesn’t reflect the reality very well (well, they never do), because I feel that Unity is much slower than that in real-life, or maybe I’m just biased… the numbers never lie though. Something to consider is that the benchmark doesn’t generate garbage, and the garbage collection in Unity is probably what is lagging behind the most.

I know that Unity has a very old Mono version because of licensing issues with Xamarin. Now that Microsoft bought Xamarin, I hope that changes quickly. In any case, it would be nice if Unity could use the MS .NET Framework on Windows platforms because it’s even faster than the latest Mono versions I’ve tried. But perhaps I’m asking too much.

Here’s the fairly simple code:

For the Unity version, besides using an empty scene with not even the default sky, I’ve added some extra code to ensure a fair comparison:

Leave a Reply