Code Coverage Analysis soon in CakePHP - Test How Well You Test
Posted on 27/4/08 by Tim Koschützki
Hey folks,
the last week I have been dealing with an implementation of code coverage analysis for the upcoming release 1.2 of CakePHP. For those of you who do not yet know what code coverage is, here is a short excerpt from Wikipedia:
Code coverage is a measure used in software testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing.
There are several different kinds of criteria to code coverage. The two most important ones are line coverage (or statement coverage as wikipedia puts it) and path coverage. Line coverage basically analyzes if each line of the subject-under-test has been executed during the test. Path coverage examines if every possible route through the code has been executed. Since I rely on Xdebug for the implementation we are talking line coverage here.
So how is it going to work? Pretty simple actually. Whenever you run a CakePHP test case Cake assembles information in the background about which lines of your subject-under-test are called. Then it will display all code lines from the subject-under-test and mark those that were "covered" (ie executed by the testcase) and which were not. Lines that play no role for the code coverage percentage, such as multiline comments, will be displayed in a different color with less contrast.
How could an example report look like? Here you go:
I have also implemented a diff-style report. It just lists all the lines that are not covered by your tests and then seven lines before and after each of those. With that view you can really focus on the uncovered lines, but see enough lines around them in order to be able to write a reasonable test case to get the uncovered lines covered.
This is what the diff view looks like for the cakephp core router.test.php:
Within the dev team we are currently discussing these two views. If you favor one of these, can recommend another one or just want to voice your opinion on the looks of them, please do so by commenting on this post.
For your information: Code coverage analysis currently only works for separate test cases only. However, the plan for this week is to get aggregate code coverage working. This means that your group tests are analyzed and the coverage percentage is displayed for each of the test subjects involved.
This is a very valuable tool when developing tests for your cakephp application. Although we are talking line coverage here and not path coverage it is still a pretty good indicator of how thorough your tests are. It might as well encourage some more people to start writing tests if they see a neat report after they have done so. :]
-- Tim Koschuetzki aka DarkAngelBGE
Did you like this blog post? If so, please consider subscribing to the Blog RSS feed.
You can skip to the end and add a comment.
Yeah I also prefer the diff view currently, but having several options is most of the time a good thing.
Regarding different views: Yeah that is possible. I would probably prefer the diff one being the default one and then on the report page there could be various other options.
I asked this in the irc channel and got the answer but I thought it would be helpful for other people to know as well.
How far along is the Line Code Coverage for CakePHP v1.2?
And an additional question, is there a chance a Path Code Coverage will be in the works?
Well I started last week and the basic view is already committed. Tonight I will commit the diff view and some code refactorings, which also make the results a little more accurate. Once I commit this, you just need Xdebug installed and are ready to go already in Cake 1.2. :] So it's pretty far along already in my opinion. ;)
What I will do after aggregate coverage is done is getting a command line report tool for this. Yeah let's do it - that would be pretty neat. :D
And plugins testing needs support, too, but that should not take long.
Well about Path Coverage: That's pretty hard to do. If all of the abov are done, I will look into it for sure.
Okay the new version is committed to Cake.
Have been playing with the code coverage additions for the past few days - great work! and very reassuring to know that the major parts of core are covered pretty well. I feel the diff view is much better
This is *leet*
@Dieter: heh, thanks. :]
@tarique: Yeah I like the diff view. Got any suggestions about it?
It's really interesting how the different parts of the core *measure up* with their coverage. Set class and Http socket seem to be the best. *looks at Felix*
This is very cool, and another good reason I should look at migrating all our tests to the Cake 1.2 test suite (currently using Daniel Hofstetter's suite).
However, for this to work on Windows with magic quoting, you need to clean $_GET first - otherwise all the paths have additional backslashes (ie libs\\set.test.php ). So while the tests are executed no worries, there is no match found from the xdebug_get_code_coverage() call.
Something like the following in app/webroot/test.php works fine.
if (isset($_GET) && ini_get('magic_quotes_gpc') == 1) {
$_GET = stripslashes_deep($_GET);
}
This is really neat stuff. Not only is great for the core developers but fantastic for end developers. I really like the testing focus that cake is taking on. I for one have found a lot of value in all the improvements to the testing suite over 1.1's test suite. Awesome work.
@Grant: Oh yeah I have done a lot of the stuff on the coverage in the meantime. I will post about that later. Be sure to go to cake 1.2 soonish. ;]
About the magic quotes: I doubt this is a problem. In fact I just tested it and it is not. have a look at parseParams($url) in the Dispatcher class.
@Mark: Thanks. :] Nate, the lead dev, wants all classes to go to 75 - 85% code coverage at least. So it will also help with getting all the core code covered.
@Grant: By the way I found a small design issue in Windows IE in the coverage reports due to your hint. Thanks a lot!
Tim - I know about Dispatcher::parseParams, that's where I copied it from :) It's just that the url is not dispatched when requesting through the /app/webroot/test.php, which is why something is needed.
You may be working on this, if so no matter. But with:
- clean 1.2.x branch r6735 test suite
- magic quotes on
- Firefox 2.0.0.14, or IE7
- requested through http://localhost/cake1.2/test.php
the xdebug coverage does not work, as the $_GET['case'] is 'libs\\set.test.php'
Ah okay, I will check it out. :] Thanks for the heads up!
Nice one Tim, this looks great.
As to which representation of the code coverage, or lack thereof, I prefer - its difficult isn't it, because each best suits different situations.
The first example is good if your subject under test is short or your code coverage is quite bad, whereas the latter is better if your subject under test is long and your code coverage is good.
Perhaps you could present one or the other depending on some criteria. A combination of number of lines and the percentage code coverage?
Hey Neil, yeah I can do that. Please give some time. Felix is currently moving to Berlin and I am helping him. Cheers
This is indeed really cool. Where can I find the implementation details? How is your progress on making this report on a group of multiple test cases?
This post is too old. We do not allow comments here anymore in order to fight spam. If you have real feedback or questions for the post, please contact us.
I like both of the sample views. I can see myself using the diff style one more than the full code mainly because it's a lot less reading and scanning. But I can see someone preferring one over the other.
Since the two different views are similar would it be possible to let the developer choose which one to show and set one of the two that the most people like as the default?