RightJS 1.5: 6-8 times faster than jQuery
Posted on 9/12/09 by Felix Geisendörfer
My journey of mastering procrastination has led me to an interesting article on Hacker News today:
RightJS 1.5: 6-8 times faster than jQuery
(The title has been updated to "RightJS Version 1.5.0 Is Out" since I started writing this.)
Wow, I thought! This sounds like an excellent example of cargo cult science, one of my favourite subjects.
I mean I love innovation in this field just like everybody else. But seriously, jQuery is not exactly know for being slow & heavy. So anybody claiming a 6-8x speed improvement must have achieved an unbelievable breakthrough. Either that, or he must be using using the cargo cult method.
Applying the cargo cult method to performance testing is rather simple, which probably explains its popularity. You pick a random series of tests that can be run against the various implementations you want to compete with. Then you spend a few hours hacking away at your implementation until it is the clear winner. Don't give up if it becomes too hard, just tweak the test cases to slightly favor your implementation. It's really as easy as that.
I can totally understand why people are doing that. The opposite would mean that you have to apply the scientific method, which is really cumbersome. First you have to collect data, lots of it. In our case that means performing very detailed analysis and profiling of a large enough set of real world JavaScript applications. Using this data set, you should be able to answer questions like: What are the most common selectors people use? What DOM operations are popular? Which of those are actually relevant to the performance of the analyzed applications? With those answers you can attempt to come up with a series of tests that will rank the various implementations according to their performance. But actually writing those tests will be very hard. Should you use the most distinct and sexy way in each implementation? Or should you use the most effective techniques people have come up with?
Luckily there is a third option. It is called specialized benchmarking. You start by admitting that the things you are going to test are purely based on your curiosity about them, possibly because they are related to the particular problem YOU care about. Make it very clear that the outcome of those tests should in no way be seen as an indicator for overall performance and try to hide them from people who don't know what that means.
Specialized benchmarking will possibly not answer anybodies questions other than your own, but it beats the hell out of the cargo cult method.
Let's examine why the RightJS performance tests get it wrong and what they could do about it. From this point on I will only refer to material on their page, the post on Hacker News was just how I heard about them.
First of all, they claim that their performance benchmarks are there "To give you some ideas about the project quality and abilities". I think that should be changed to: "We are especially fast for the following operations (...), those however are not proofen to be good indicators for general performance in JS projects.". It's kind of like weight loss advertisement. You can show pictures of people who lost 50 pound, but you gotta put that "* Results are not typical" note there. This way people can pause for a second, and remember that there are no magic bullets to weight loss and consider their purchase with that in mind.
After that, they could start to decide whether some of their tests are worth keeping, and if so, make sure that they are as scientific as possible. I'll just use their test #1 as an example, but check the test suite for yourself, to see that this pattern is repeated throughout the entire thing.
Testing jQuery DOM building (343ms*):
for(var i = 0; i<250; i++){
$("<ul id='setid" + i + "' class='fromcode'></ul>")
.append("<li>one</li>")
.append("<li>two</li>")
.append("<li>three</li>")
.appendTo("body");
}
return $("ul.fromcode").length;
}
Testing RightJS DOM building (80ms*):
for (var i = 0; i < 250; i++) {
document.body.appendChild(
new Element('ul', {
'class': 'fromcode', id: 'setid'+i
}).insert([
new Element('li', {html: 'one'}),
new Element('li', {html: 'two'}),
new Element('li', {html: 'three'})
])
);
}
return $$('ul.fromcode').length;
}
I smell cargo! First of all, why is RightJS using a native DOM method, document.body.appendChild, and jQuery has to use .appendTo('body')? Those are two radically different operations, and just to see how radical lets make the following change:
Optimized jQuery DOM building I (194ms*):
for(var i = 0; i<250; i++){
document.body.appendChild(
$("<ul id='setid" + i + "' class='fromcode'></ul>")
.append("<li>one</li>")
.append("<li>two</li>")
.append("<li>three</li>")[0]
);
}
return $("ul.fromcode").length;
}
Ouch, an error rate of 43% against jQuery. Let's try harder:
Optimized jQuery DOM building II (72ms*):
for(var i = 0; i<250; i++){
document.body.appendChild(
$(
"<ul id='setid" + i + "' class='fromcode'>"+
"<li>one</li>"+
"<li>two</li>"+
"<li>three</li>"+
"</ul>"
)[0]
);
}
return $("ul.fromcode").length;
}
If this was a presentation I would have an LOLCat saying "jQuery rulez" right now. But luckily this isn't and I'll try to reason scientifically about this.
jQuery is NOT faster in this example. Don't believe the numbers you see. They have been meaningless all along. The reason for that is simple: While initially it looked like we were performing the same test with jQuery as we were with RightJS, we never actually did! The jQuery example, from the beginning, was creating DOM elements from HTML strings, while RightJS was wrapping the document.createElement API. This is not the same thing and you cannot learn anything from comparing apples to oranges.
The truth as far as this test case is concerned? Well, jQuery simply does not have a document.createElement wrapper. Thus you cannot compare it to implementations that do. And why should you? DOM building like this is largely useless, given excellent alternatives such as John' Micro -Templating engine.
Just to show how useless this test was from the beginning, here is my not so paradox implementation that outperforms the pure DOM test:
Testing Pure DOM building (37ms*):
for(var
body = document.body,
ul = document.createElement("ul"),
li = document.createElement("li"),
i = 0,
fromcode;
i < 250; ++i
){
fromcode = ul.cloneNode(true);
fromcode.id = "setid" + i;
fromcode.className = "fromcode";
fromcode.appendChild(li.cloneNode(true)).appendChild(document.createTextNode("one"));
fromcode.appendChild(li.cloneNode(true)).appendChild(document.createTextNode("two"));
fromcode.appendChild(li.cloneNode(true)).appendChild(document.createTextNode("three"));
body.appendChild(fromcode);
};
return utility.getSimple.call(body, "ul.fromcode").length;
}
Optimized jQuery DOM building III (36ms*):
var elements = '<div>';
for(var i = 0; i<250; i++){
elements = elements+
"<ul id='setid" + i + "' class='fromcode'>"+
"<li>one</li>"+
"<li>two</li>"+
"<li>three</li>"+
"</ul>";
}
$(elements+'</div>')
.children()
.each(function() {
document.body.appendChild(this);
});
return $("ul.fromcode").length;
}
As you can see, the cargo cult method is quite powerful : ).
Anyway, I don't want to discourage the development of RightJS in any way. I think it's awesome that there are libraries that are trying to compete with jQuery.
It is really hard to do meaningful performance testing and infinitely easy for some random punk like me to come along and point out all the flaws. To me, even trying to do a general purpose performance test against 6 (!) implementations, that is pure bravery. So in case you decide to do something similar, just admit the odds you are up against and people will be very forgiving and engaged.
Comments, hate mail & suggestions are welcome!
-- Felix Geisendörfer aka the_undefined
* Results not typical - Some recent version of Firefox on my Laptop, picking random samples from runs that looked good!
Did you like this blog post? If so, please consider subscribing to the Blog RSS feed.
You can skip to the end and add a comment.
secondly you do your construction off-dom.
basically what you test in your code is the body.appendChild method performance. it has nothing to do with jQuery performance at all.
Nik: Thanks for playing : )
you're welcome. write again.
Nik: I thought you were kidding. Nevermind ...
Felix: sorry didn't really read the article, though you're serious with the jQuery's test tweaking and was quick on my reaction 8)
Then I've read it and I agreed, the tests technique of taskspeed is quite a bool. Especially I love the part where RightJS rips of the pure-DOM test 8). Dojo and YUI in the tests had done most of the things in pure dom too, that's why they look fast too. Then it's really affected by the css-selectors performance.
And it's complete off-ballance. But well. It not all shitty, at some points like add/remove class you might have some sort of comparison.
But anyway you should agree that RightJS in those tests was more or less honest in the way the things implemented.
BTW: This is me the author of RightJS
Nik: Yeah, I don't think RightJS is cheating. I just saw a totally flawed comparison and you know how it is when you think somebody is wrong on the internet : ).
Keep up the great work and don't let people like me discourage you! I absolutely love new interesting libraries, and RightJS looks great!
Thans Felix, but it's too late to give up anyway 8)
Will read you now, you're funny boy!
As I said on HN; lets not blame the RightJS developer - he/she never seems to have explicitly made a 6x better claim! Or even any kind of "faster than X claim".
It was an editorial mishap by whoever posted it to HN :(
On the other hand I agree with the rest of your post :)
Errant: Yeah, the RightJS site itself doesn't make that claim. But it's easy to see why somebody would look at it, and draw the same conclusion as the poster on HN did from it. Personally I'd be very happy if Nik just adds a bit more explanation and context to the way the results are presented.
Well, I think you've kind of blown this out of proportion. That HN heading was not posted by Nikolay, it was by some guy who was hyping. The reality is, Nikolay has some performance tests he uses to make sure Right.JS is fast. It's probably more of a regression test suite than anything. Of course, you probably wouldn't know what a test suite was so it's understandable you wouldn't realize that.
If you actually went to rightjs.org and read it, he says it's fast, not the fastest. The fact that he has some tests to make sure it's fast is a bonus. The fact he gave you the code is double bonus, and more than you've done here.
You have no verifiable data available, no full runnable test suite, and no indication of the browsers it runs in.
I'm beginning to think you work at the CRU. :-)
Whether you think his tests have problems or not doesn't matter. He's testing something, and he's giving you the full skinny, entire data, and the code he runs. That's mountains more disclosure than you've got in your blog post right here.
Felix: most of the pages on the site actually were done just off hands when the project was rolling out, and the site by itself is more like a technological demo.
I'm working with a designer now and going to rework/rewrite most of its parts, so I'll write something more or less meaningful over there soon.
Just wanted to point out that as soon as jQuery 1.4 is released, the statement "Well, jQuery simply does not have a document.createElement wrapper" will no longer be true. However, you're absolutely correct for the current (1.3.2) release.
http://github.com/jquery/jquery/blob/master/src/core.js#L77-91
Zed Shaw: Of course I've blown it out of proportions, I'll be the first to admit it. By taking an argument to the extreme you learn all of its weaknesses and flaws, and you get much more passionate and interesting responses, such as yours. I really think RightJS is doing some awesome stuff. I agree with you on everything except your assumption about my knowledge on test suites : )
Nik: Cool, is the source of it up somewhere? I'd certainly volunteer to provide a patch for putting the benchmark page into perspective to make up for my rant : ).
Mike: Cool, I didn't know that! Thanks for pointing it out.
Felix: yup. it's on github next to all other rightjs related stuff
http://github.com/rightjs/rightjs-home
but don't make it too smart though, I still want people to put a healthy amount of thoughts on the matter, question it and came to their own conclusions ;)
I really use the benchmarks as regression tests mostly to watch RightJS own progress from version to version, find weak spots, etc. And then, there are actually tests for Prototype and Mootools and they are more or less valid and can be correlated with RightJS, case all three frameworks have similar api and do things in similar way.
So try to not make a show out of it.
Nik: No worries, I would probably just add a note saying that it's a regression suite and shouldn't be seen as as a way of comparing the general speed of those frameworks. Would that be ok?
Felix: Well I think you actually can, but as a blunt direct comparison on a bunch of weird stuff.
The thing as I see it, the test by itself is not a threat and it has a good amount of truth in it. The problem is the interpretation and here I rather agreed with Zed you kinda blow it out of the proportions. I see your point and irony, but the thing is that if you're a smarty pants and know what are you doing, performance in JavaScript will never be a problem to you with any framework, case you always can fall back to the pure dom and do some mambo-jumbo around.
In reality most of developers are not hardcore programmers, and they know about JavaScript just enough to make a simple ajax call or make an animated navigation menu. Those people will do what they see on the surface of the public API or another words, do things blunt and directly. And in this light those tests do actually reflect the general speed of those frameworks.
So it's kinda tricky business. At one side you can interpret the tests as general speed measures, on the other the taskspeed util by itself rather idiotic and unbalanced, without any real value in each separated test and this makes that ridiculous summary result of 8x speed up.
I actually had a paragraph over there saying that dojo and yui are cheating in the test and jQuery just don't have the elements building functionality and blah blah blah, but it was boring and I nuked it.
Anyway, I'm being long. Try what you have on mind. I always can correct where I don't agree. As the result we can come to some sort of compromise
WebReflection here, Pure DOM author.
Pure DOM has been explicitly created to perform tasks in the best optimized way using DOM natives, except in a single test where it is explicitly asked to use innerHTML.
We need to do few considerations here:
1 - 36 ms VS 37 ms are a fool if you call it outperform
2 - Pure DOM does NOT want to use innerHTML. We all know that innerHTML is faster and your test is a cheat. If I cheat with Pure DOM bye bye baseline (innerHTML is not exactly what I define Pure DOM, HTML5 a part, but I may consider an innerHTML based version so you won't outperform *anything* with whatever library it is, is this what you want?)
for(var
el = document.createElement("div"),
outperform_this = [],
i = 0, length = 250; i < length; ++i
)
outperform_this[i] = '<ul class="fromcode" id="setid' + i + '"><li>one</li><li>two</li><li>three</li></ul>'
;
el.innerHTML = outperform_this.join("");
for(var i = 0, fragment = document.createDocumentFragment(), childNodes = el.childNodes; i < length; ++i)
fragment.appendChild(childNodes[i].cloneNode(true))
;
document.body.appendChild(fragment);
for(var childNodes = document.getElementsByTagName("ul"), i = 0, l = 0, length = childNodes.length; i < length; ++i){
if((el = childNodes[i]).className === "fromcode")
++l
;
};
return l;
}
3 - same libraries authors perfectly know which is the TaskSpeed aim: demonstrate how are performances via daily basis code ... there is not such jQuery usage and if there is, you must consider to use directly JavaScript because you are obviously looking for Pure Performances. This YUI! related post should tell you something, nobody complained about TaskSpeed so far: http://www.yuiblog.com/blog/2009/04/13/yui-270-on-taskspeed/
4 - last, but not least, you used the old TaskSpeed implementation, while the new one should be even faster
Best Regards
Moreover, I did not cheat using even querySelectorAll or getElementByClassName where available ... but still, if that is what you got about TaskSpeed, I'll be glad to provide you a proper cheat for the "make" test
just in case ... http://webreflection.blogspot.com/2009/12/taskspeed-real-cheat.html
Zed, thanks for adding absolutely nothing to the discussion and wasting my time.
Just like this comment.
This is awesome, but jquery is not only a DOM-manipulation tool, it's also a big number of plugins, etc. I think it is standard de-facto for now
AFAIK jQuery is using dojos CSS selector engine thus any comparison like the above mentioned would actually be a comparison with dojo yet still everyone indicates jQuery as their "main target"
Actually I believe Jquery is using John Resig's Sizzle css selector engine - which was given to the Dojo Foundation to be maintained outside of the Jquery loop.
They then adopted it, not the other way around. So .. just to be pedantic .. Dojo is using Jquery's selector engine.
Juergen : as such, according to your logic - not mine - any comparison by dojo and their community would actually be a comparison with Jquery. The exact opposite of what you said. That being said, the two frameworks are hugely different and it would also be a bad apples to apples comparison.
This post is too old. We do not allow comments here anymore in order to fight spam. If you have real feedback or questions for the post, please contact us.
You don't do it the same way as others in your examples. First of all you don't construct the elements with framework, you feed the innerHTML property with strings and make your browser construct the elements. It's different.