Tag Archives: Unit Tests

One time the unit tests failed me

Since my initiation into the world of unit testing about a year or so ago, I’ve been a huge fan of unit testing. I write unit tests for almost everything now, rather than doing repeated manual testing, which is sometimes more tedious (and certainly gets boring after a while).

But just now, the unit tests failed me. I’ll explain the case in a moment, but let me just say that, as great as unit tests are, they aren’t the only kind of testing you need. Doing manual beta testing is always needed for good measure.

So here’s what happened. I had written a custom module for my WordPoints plugin, with unit tests, of course. And the unit tests were passing fine.

That was last week. But before I tried it out on a live site, I wanted to test it manually on one of my local setups. And guess what. It didn’t work.

I eventually figured out that the issue was a bug in WordPoints, that was caused by WordPress having a 64 character limit for option names. I knew about this limit, but it hadn’t bitten me yet. I’d actually forgotten whether I’d handled the limit properly or not. But that is by the by.

What struck me after I figured out what was going on was this: “What about the unit tests I wrote, they passed fine!”

Yep, they did. And here’s why. For the unit tests, WordPress’s testcase sets up a database transaction before each test. Of course it doesn’t actually commit the data to the database, instead it rolls back the transaction after the test is done. This way, the database stays clean between tests, so they don’t leak into each other.

And this is why my tests didn’t fail. The data wasn’t actually getting committed to the database, so saving the option value wasn’t failing like it normally would. I assume that the database checks that the table and column names are all good, but doesn’t realize that you’re trying to write 65 characters to a VARCHAR(64) field until the data gets committed (which in this case it doesn’t).

In the end, I really think the issue here is that WordPress should fail, or tell us that we’re _doing_it_wrong(), when we try to use option names that are too long. But until then, we need to be careful, when using dynamic option names, that we check they aren’t too long. Especially since unit tests won’t be enough to catch the issue.

WP Ajax Plugin Unit Testing

The other day I was refactoring some code for an Ajax callback in one of my plugins. I was dreading having to manually test that the changes I was making didn’t break anything, when I had an idea. “Why don’t I have unit tests for this?”, I though to myself. Before I started trying to mock Ajax requests myself, I thought I’d ask Google what was already out there. As it turns out, the WordPress unit test framework already provides a testcase specifically for testing Ajax callbacks. Who knew? Obviously, the developers did, but I hadn’t thought much about it before, except that each time you run the WP unit tests, you get a message that it the Ajax unit tests aren’t being run.

While I found this testcase in the WordPress PHPUnit test source code, what I didn’t find was any tutorials on how it works. So, I just used WP’s AJAX unit tests as an example. Its pretty simple actually.

The WP_Ajax_UnitTestCase

If you’re familiar with plugin unit testing, you know that your testcases need to extend the WP_UnitTestCase class to leverage the full power of the WordPress testing framework. Well, for Ajax tests, its the same thing, only you need to extend a child of WP_UnitTestCase: WP_Ajax_UnitTestCase. This testcase provides a few methods that make the Ajax unit tests simpler.

The most important method is of course the one that actually mocks the Ajax request: _handleAjax(). It takes just one parameter; the name of the Ajax action:

$this->_handleAjax( 'my_ajax_action' );

Setting up for the test

Of course, before you run the Ajax handler, you will need to set up for the test. The first thing you need to do is set up the $_POST data. You need to mock up the $_POSTed data, and then later we’ll check that we get the expected result.

$_POST['_nonce'] = wp_create_nonce( 'my_nonce' );
$_POST['other_data'] = 'something';

Setting the expected exception

The test is performed by running the 'wp_ajax_*' action, and thus invoking the callback. (No request is actually made.) When the Ajax callback function runs, it should finish by calling die() or exit(). Well, actually, it will need to be using wp_die(), because if you call die() you will kill the tests. Lesson: in WP Ajax callbacks, you should use wp_die(). Fortunately, I already was in the callbacks that I was testing, so I was good to go.

Now, when wp_die() gets called, the testcase catches it and prevents it from actually killing the execution of the script. Instead, it throws an exception. The testcase uses two different exceptions, and which one gets thrown depends on whether there was any output from callback (this is caught with a buffer). If there was output, a WPAjaxDieStopException is thrown. Otherwise, a WPAjaxDieContinueException is thrown. These need to be caught. One way that this is done is using the setExpectedException() method provided by PHPUnit.

If, for example, you are expecting the callback to exit without any output, you would do this:

$this->setExpectedException( 'WPAjaxDieStopException' );
$this->_handleAjax( 'my_ajax_action' );

If you need to run some assertions afterward, you’ll need to catch the exception manually instead. For example, your Ajax callback may be supposed to save some data to the database, and you’ll want to check that the data was actually saved.

try {
	$this->_handleAjax( 'my_ajax_action' );
} catch ( WPAjaxDieStopException $e ) {
	// We expected this, do nothing.
}

$this->assertEquals( 'yes', get_option( 'some_option' ) );

Checking for the Correct Output

If your Ajax function outputs some data, you will probably want to check that the expected data gets output. If you are sending a simple response code with wp_die(), you’ll probably want to check that the expected value was output. The message passed to wp_die() is the message of the exception that gets thrown. So one way to check for this is when you set the expected exception:

$this->setExpectedException( 'WPAjaxDieStopException', '1' );

In this case we are expecting '1' to get output to signify a successful result.

If you need to manually catch your exception so you can run some other assertions like we did above, you’ll need to do it differently:

try {
	$this->_handleAjax( 'my_ajax_action' );
} catch ( WPAjaxDieStopException $e ) {
	// We expected this, do nothing.
}

// Check that the exception was thrown.
$this->assertTrue( isset( $e ) );

// The output should be a 1 for success.
$this->assertEquals( '1', $e->getMessage() );

$this->assertEquals( 'yes', get_option( 'some_option' ) );

But if you are outputting your data directly instead of using wp_die(), you need to do it differently. One example might be if you are using wp_send_json_success() to send a JSON response. In this case, the output is caught by _handleAjax() and stored in $this->_last_response. So you could check for the correct JSON response like this:

try {
	$this->_handleAjax( 'my_ajax_action' );
} catch ( WPAjaxDieStopException $e ) {
	// We expected this, do nothing.
}

$response = json_decode( $this->_last_response );
$this->assertInternalType( 'object', $response );
$this->assertObjectHasAttribute( 'success', $response );
$this->assertTrue( $response->success );

Setting the User Role

Many times an Ajax callback isn’t intended for all users. You may be saving something that only users with administrator permissions should be allowed to save. The Ajax test case class provides a handy helper function to let you set the current user’s role. For example, if your Ajax callback should only work for administrators, you may want to test that just any subscriber can’t come along and use it. To do this, you would set the user’s role before calling _handleAjax(), like this:

$this->_setRole( 'subscriber' );

And if you want to make sure that the administrators will have the permissions necessary to use it, you would set the role to 'administrator' instead:

$this->_setRole( 'administrator' );

You can also run as a logged-out user:

$this->logout();

Running Your Tests

Based on the examples above, our final testcase might look something like this:

/**
 * Test case for the Ajax callback to update 'some_option'.
 *
 * @group ajax
 */
class My_Some_Option_Ajax_Test extends WP_Ajax_UnitTestCase {

	/**
	 * Test that the callback saves the value for administrators.
	 */
	public function test_some_option_is_saved() {

		$this->_setRole( 'administrator' );

		$_POST['_wpnonce'] = wp_create_nonce( 'my_nonce' );
		$_POST['option_value'] = 'yes';

		try {
			$this->_handleAjax( 'my_ajax_action' );
		} catch ( WPAjaxDieStopException $e ) {
			// We expected this, do nothing.
		}

		// Check that the exception was thrown.
		$this->assertTrue( isset( $e ) );

		// The output should be a 1 for success.
		$this->assertEquals( '1', $e->getMessage() );

		$this->assertEquals( 'yes', get_option( 'some_option' ) );
	}

	/**
	 * Test that it doesn't work for subscribers.
	 */
	public function test_requires_admin_permissions() {

		$this->_setRole( 'subscriber' );

		$_POST['_wpnonce'] = wp_create_nonce( 'my_nonce' );
		$_POST['option_value'] = 'yes';

		try {
			$this->_handleAjax( 'my_ajax_action' );
		} catch ( WPAjaxDieStopException $e ) {
			// We expected this, do nothing.
		}

		// Check that the exception was thrown.
		$this->assertTrue( isset( $e ) );

		// The output should be a -1 for failure.
		$this->assertEquals( '-1', $e->getMessage() );

		// The option should not have been saved.
		$this->assertFalse( get_option( 'some_option' ) );
	}
}

And our Ajax callback might look like this:

/**
 * Ajax callback to save the 'some_option' option.
 */
function my_some_option_ajax_callback() {

	check_ajax_referer( 'my_nonce' );

	if ( ! isset( $_POST['option_value'] ) || ! current_user_can( 'manage_options' ) ) {
		wp_die( '-1' );
	}

	update_option( 'some_option', $_POST['option_value'] );

	wp_die( '1' );
}
add_action( 'wp_ajax_my_ajax_action', 'my_some_option_ajax_callback' );

Notice that I added @group ajax to the testcase docblock? That isn’t necessary, but you will probably want to do it, because it lets you easily run just your Ajax tests. Also, you will probably want to exclude the Ajax group from running by default, the same way WordPress does. The reason is because the tests can be very slow. The Ajax tests are so slow because all of the default Ajax callbacks in WordPress core get hooked up before each test is run. There are quite a few, and this takes some time. I don’t think this feature is actually necessary, and maybe it will be removed in future, especially if the test suite starts preserving hook state between tests. It also calls 'admin_init' every time, which is also slowing things down.

So that’s it. There really isn’t much to the Ajax unit tests, but it saved me from breaking the code I was refactoring. And hey, after all, that’s what unit tests are for.

Unit Testing WordPress Plugin Uninstallation

If you’re a WordPress plugin developer, there are two things that you should probably be doing. First, you should be writing your plugin with an uninstall routine to clean everything up when a user decides to remove it. Second, you should be unit testing your plugin, to make sure that everything is working. Actually, we could go a step further: You should be unit testing your plugin’s uninstallation script. It kind of follows naturally doesn’t it?

I thought so, so I created a set of tools to make that easier: the WP Plugin Uninstall Tester

From the project README:

The purpose of this testcase is to allow you to make plugin uninstall testing as realistic as possible. WordPress uninstalls plugins when they aren’t active, and these tools allow you simulate that. The installation is performed remotely, so the plugin is not loaded when the tests are being run.

I created these tools after finding that there was a fatal error in one of my plugin’s uninstall scripts. Not that I didn’t have unit tests for uninstallation. I did. But the uninstall tests were being run with the plugin already loaded. So I never realized that I was calling one of the plugin’s functions that wouldn’t normally be available. That’s when I decided to create these testing tools, so my uninstall tests would fail if I wasn’t including all required dependencies in my plugin’s uninstall script.

In addition to providing a realistic uninstall testing environment, it also provides some assertions to help you make sure that your plugin entirely cleaned up the database.

That basically sums up the project. I encourage you to add it to your plugin’s test suite. And it’s hosted on GitHub, so you’re welcome to help make it even better.

WordPress Unit Test Suite Introduces @expectedIncorrectUsage

In an recent effort to clean up the WordPress unit tests, the @expectedDeprecated notation was introduced for testing deprecated functions. Now another new notation was introduced which does a similar thing for _doing_it_wrong(). Usually, you don’t want to be _doing_it_wrong(), but you might want to test that the proper warning is given from your code when someone does it wrong. And that is where @expectedIncorrectUseage comes into play. Now your tests will fail if you are _doing_it_wrong(), which is helpful to make sure you are coding your project correctly. But when you are testing for a _doing_it_wrong(), you are expecting it, so you can add the @expectedIncorrectUsage to your tests. Then your test will fail if you don’t receive the expected _doing_it_wrong() warning.

The usage is simple:

<?php

/**
 * A test class with failing tests.
 */
class Failing_WP_UnitTestCase extends WP_UnitTestCase {

	/**
	 * This test will succeed, because we were expecting to do it wrong.
	 *
	 * @expectedIncorrectUseage register_uninstall_hook
	 */
	public function test_with_expected_incorrect_usage() {
		
		// Only static functions can be registered as an uninstall callback.
		register_uninstall_hook( __FILE__, array( $this, 'a_function' ) );
	}
	
	/**
	 * This test will fail, because we weren't expecting to do it wrong.
	 */
	public function test_with_expected_incorrect_usage() {
		
		// Only static functions can be registered as an uninstall callback.
		register_uninstall_hook( __FILE__, array( $this, 'a_function' ) );
	}
	
	/**
	 * This test will fail, because we don't do this wrong, though it was expected.
	 *
 	 * @expectedIncorrectUseage register_uninstall_hook
	 */
	public function test_with_expected_incorrect_usage() {
		
		// Only static functions can be registered as an uninstall callback.
		register_uninstall_hook( __FILE__, array( __CLASS__, 'a_function' ) );
	}
}

This will be useful for those that are heavily using the WordPress unit test framework, like running plugin unit tests.

WordPress Unit Testing: Deprecated Functions

A new feature has just been introduced into the WordPress test suite: the @expectedDeprecated notation. This notation is modeled after PHPUnit’s built-in @expectedException notation, and allows you to test for the use of deprecated functions or function arguments in your test suites. (You do have a suite of unit tests for your plugin, don’t you?)

This new feature comes as the result of a charge to clean up the WordPress unit tests, lead by @wonderboymusic. I’m really glad to see this happen, because the tests, like any WordPress development environment, really should be run with WP_DEBUG on, and when I tried that a few weeks ago I encountered plenty of notices. Many of those notices were about the use of deprecated functions in the test suite. The use of these functions is intentional, and is meant to test for backward compatibility. The new notation provides a way to test deprecated functions, and at the same time test that the proper deprecated notices are given. Also as a result of this, any time a deprecated function is used in a suite and it hasn’t been @expectedDeprecated, the test will fail. Tests which expect deprecated notices and don’t get them will also fail.

The usage of this new annotation is simple enough. If you have a test case that uses a deprecated function or argument, then add an @expectedDeprecated to the docblock for the test function where the deprecated function or function argument is used. In the below example, we’ll test one of your project’s functions that has been deprecated, and another that has a deprecated argument:

<?php

/**
 * A test case.
 *
 * In case we fail, we test.
 */
class My_Deprecated_Function_Test extends WP_UnitTestCase {

	/**
	 * Test these this deprecated functions are backward compatible.
	 *
	 * They have to give deprecated notices too, or else this won't pass.
	 *
	 * @expectedDeprecated my_deprecated_function
	 * @expectedDeprecated function_with_deprecated_arg
	 */
	public function test_my_deprecated_function() {
	
		$this->assertTrue( my_deprecated_function() );
		$this->assertEquals( 4, function_with_deprecated_arg( 'deprecated' ) );
	}
}

You can also use this when testing for backward compatibility with older versions of WordPress, by marking any deprecated WordPress functions used as @expectedDeprecated.

The notation can also be used in the class docblock, if all of your tests in a test case class will be using the deprecated function.