Objective C v C Speed / Benchmarks

I've been becoming increasingly interested in Objective C recently (as an alternative to C++). It is a much cleaner object-oriented extension to C that dodges a lot of of the pitfalls present in C++.

One of the "complaints" about Objective C is that message passing (Objective C's method of communicating with objects) is slower than C++ and C. Because of the nature of Objective C, this is likely true. I set out to discover just how much Objective C's dynamic messages would cost me.

A blog post by Jonathan Hohle showed some interesting numbers for a recursive Fibonacci calculator. Hohle's post showed that Objective C's message passing costs pushed the runtime of the Objective C version of the code up to nearly triple that of the C version. Using a few hacks (storing a function pointer to the result of a message) reduced the difference to double.

Hohle's results left me skeptical and wanting a bit more, so I created my own little benchmark with an Objective C and a C implementation of a linked list. I tried to keep the codes as close to each other as possible, only introducing differences when an Objective C construct should be used (objects, most notably).

The code for the linked list implementations is exactly what you'd expect it to be, but it is available for download (as well as scripts to run the benchmark yourself) at the bottom of this post.

Before anyone jumps to any long-shot conclusions, obviously these benchmarks aren't perfect: not even close. I don't have access to the wide range of machines that would be needed to sufficiently gauge what is going on. I'm also aware that only testing a linked list implementation is not exhaustive. I was interested in the speed of various data structures like a linked list, so that's what I tested. Take everything below with a handful of salt.

The actual benchmark code (that utilized the linked list) looks like this:

C:

#include "linked_list_c.h"
#include <stdlib.h>
#include <stdio.h>


int main(int argc, char** argv) {
	int SIZE = atoi(argv[1]);

	// make a new linked list
	struct linked_list* ll = ll_create();
	
	int* d = calloc(SIZE, sizeof(int));
	int i;
	for (i=0; i < SIZE; i++) {
		d[i] = i;
		ll_add(&(d[i]), ll);
	}
	
	printf("Size of list: %d\n", ll_length(ll));
	int sum = 0;
	while (ll_length(ll) != 0) {
		sum +=  *(int*)ll_pop(ll);
		struct linked_list* lli = ll_create();
		for (i=0; i < sum; i++) {
			ll_add(&(d[1]), lli);
		}

		while (ll_length(lli) != 0) {
			sum -= *(int*)ll_pop(lli);
		}
		free(lli);
	}
	printf("Sum: %d\n", sum);
	
	
	free(d);
	free(ll);
	return 0;
}

And its Objective C counterpart:

#include "linked_list.h"
#include <stdlib.h>
#include <stdio.h>



int main(int argc, char** argv) {
	int SIZE = atoi(argv[1]);
	
	// make a new linked list
	LinkedList* ll = [[LinkedList alloc] init];
	
	int* d = calloc(SIZE, sizeof(int));
	int i;
	for (i=0; i < SIZE; i++) {
		d[i] = i;
		[ll add: &(d[i])];
	}
	
	printf("Size of list: %d\n", [ll length]);
	int sum = 0;
	while ([ll length] != 0) {
		sum +=  *(int*)[ll pop];
		LinkedList* lli = [[LinkedList alloc] init];
		for (i=0; i < sum; i++) {
			[lli add: &(d[1])];
		}

		while ([lli length] != 0) {
			sum -= *(int*)[lli pop];
		} 

		[lli release];
	}
	printf("Sum: %d\n", sum);
	
	
	free(d);
	[ll release];
	return 0;
}

If you're anything like me, you prefer the look and feel of the Objective C code, but only if the performance costs aren't too high.

I tested my code on two different systems: an i7 920 Linux box (Debian stable Linux 2.6.32-5-amd64) and a MacBook Air 13" (OS X 10.7.3, Darwin 11.3.0). Since the majority of my development targets Linux, I wanted to test both GCC and Clang on the Linux box. Apple's default compiler was used on the MacBook Air.

There is quite a bit of test data, and I've made it available in a publicly-accessible Google spreadsheet here: https://docs.google.com/spreadsheet/ccc?key=0AmQWGSF3o-iidEtCR2Yxa1lPR2FhUDZRc0wyRXVGckE

The first system/compiler tested is Clang, an LLVM compiler. The compiler options for both the GCC and Clang tests were provided by the gnustep-config utility.

Clearly, the Objective C version is slower. It might be more helpful to look at a graph of the percent differences between the two implementations:

Despite some strangeness with low data sizes, Clang's compilation of my Objective-C implementation runs around 30% slower than Clang's compilation of my C implementation. The strange behavior at lower ranges is likely due to the initial overhead of the first message passed to an object.

Clang's Objective C is about 30% slower than Clang's C

Similar tests for GCC:

Once again, Objective C is obviously slower, but the gap looks quite a bit smaller. The percent-difference graph is more helpful.

GCC's Objective C binary seems to approach 25% slower than C as opposed to Clang's 30%, possibly due to the maturity of GCC.

GCC's Objective C is 25% slower than C

Since GCC's C binaries tend to execute faster than Clang's, it is no surprise that GCC's Objective C binaries also execute faster than Clang's Objective C binaries. The difference, however, is not that large.

Generally, GCC's Objective C binary seems to be 4-5% faster than Clang's Objective C binary, although the shape of the graph seems to indicate that the difference may shrink even more as the problem size increases.

GCC's Objective C seems to be 4-5% faster than Clang's Objective C

One would imagine that Apple's compiler would be especially good at compiling Objective C, given Apple's large commitment to the language. Benchmarks of the same code ran on a MacBook Air are below.

It appears as though Apple's Objective C compiler produces binaries that are around 18% slower than Apple's C binaries. The shape of the graph suggests that the percentage may decrease further with an increased problem size.

Apple

In conclusion, Objective C sucks a bit of the performance out of your application if you depend on objects/messages for high-use data structures. While these numbers certainly have their issues (there's no real comparison of the Apple compiler to any other compiler), they do suggest that performance-critical chunks of code ought to remain in C (or, *shutter*, Fortran).

In the HPC world,Greenspun's tenth rule seems to hold true: "any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp." Meaning, of course, that certain higher-level constructs (like objects) inevitably become needed. So while Objective C shouldn't be replacing our high-performance data structures yet, it could be used to implement some of the more complex parts of our applications.

Perhaps in a scenario when I would take advantage of distributed objects, I would consider Objective C.

There's no doubt that Objective C is a well thought-out and elegant language, and it is possible that there are various constructs that, when implemented in Objective C, would actually be faster than their written-by-me C counterpart. Until I run into one, I'll just continue hammering away in C.

While it is out of the scope of this post (and many others have done a much better job), GCC did seem to produce faster C binaries than Clang.

You can grab the code here. Reddit comments.

Last updated by at .

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>