Update from the QA frontier

Not only since my election as a member of the PEAR group I have been agitating for a better quality assurance process in PEAR. One year ago, we had constantly over 500 open bug reports (that were not feature requests). By bugging other developers and fixing issues ourselves we were successful - 6 months ago we were at 400 bugs! Unfortunately, the number has been rising since then and exceeded the 500 mark two or three weeks ago - again.

I am sure that, with more time, we will get the number down again. But let's come to the things I wanted to write about in the first way.

Since July this year I have been working on a PEAR-wide unit test suite that integrates all tests available in PEAR and runs them regularly. PEAR's CVS root directory contains a file called AllTests.php that is responsible for this task. All packages with a PHPUnit3 compatible AllTests.php file in their tests/ directory get included into the global test suite and executed.

There are several problems with this way of running tests: Executing all tests in one suite hogs memory, and if one test crashes (Fatal Error), the whole suite stops. I originally chose this way to get a full coverage report over all tested PEAR files. Phing offers methods to combine the coverage output of several suite runs - which is what the suite will be using in the near future.

Another issue was the fact that many packages do not use PHPUnit tests but .phpt ones. The PEAR installer itself has a class to run .phpt test cases, but since that did not help me with my PHPUnit suite, I wrote some patches for PHPUnit that are fully functional since version 3.2.0! This means that the pear suite will soon also execute .phpt tests and generate the coverage analysis for it.

The last but most problematic case was to get the packages work without actually installing them. Most packages do not have the same directory structure in CVS as they have when being installed, which complicates everything. I ended up modifying the include path to list every single package directory in CVS, and symlinking missing root directories in the packages. While it's a really ugly hack, it works.

Beside the unit tests, the daily cronjob on my server checks the package's documentation (if all classes and methods are mentioned in the pear docs) and checks if the php files adhere to the PEAR coding style.

I began logging all results into a database nearly a month ago, and wrote a little script producing a graph of the coding style violations along the timeline. Together with the result overview we now have a nice tool to see where some of the problems in PEAR code are.

Written by Christian Weiske.

Comments? Please send an e-mail. Or Reply or Like.