I’ve been playing around with the Tipfy framework for App Engine. Tipfy is a framework built on top of App Engine’s APIs that provides many features on top of what is currently possible. I won’t go too much into their virtues here.
One thing that’s bothered me is the dearth of a testing guide. More disturbing still is that one of the top search results for unit testing is a groups post of a developer bragging that he doesn’t write tests (let’s hope no one ever has to work with you). Digging around, it’s clear that Rodrigo Moraes, the creator of Tipfy, emphasizes testing in his own app, as can be evidence by the testing package in the Tipfy source repository. I’ve decided to write this quick guide to help other developers to try to save some time having to do the detective work I’ve had to do to get unit tests running.
Shortcut
So – if you don’t want to read, you can just skip ahead and read this code sample which shows an example of how to write tests for the demo “Hello, World” application that comes as part of the Tipfy download.
Getting Started
We’re going to need a few different tools to run tests. Note that we don’t need need them, I just find that using these tools will make our life a lot easier:
- Nose – Nose is a popular Python test discovery and execution tool. Nose will dig through your source directory and run your tests
- Nose GAE plugin – this is the plugin that makes nose play nice with the local App Engine SDK
If you don’t already have these tools installed, go ahead and install them with easy_install:
sudo easy_install nose sudo easy_install nosegae
We’ll also need to make sure tipfy is on our PYTHONPATH. Look for tipfy under YOUR_TIPFY_INSTALL/app/distlib. Here’s what I see as of the writing of this post:
distlib ikai$ ls README.txt babel jinja2 tipfy werkzeug
Add this to your PYTHONPATH by adding a line to ~/.bash_profile (or equivalent on your system):
export PYTHONPATH="/path/to/root/of/tipfy/libraries
If needed, run:
source ~/.bash_profile
Alright, you’re ready to roll. Run a test from the root of your application directory. It’s probably easiest to do this from the directory app.yaml resides in:
nosetests -d --with-gae --without-sandbox -v
Note that this assumes your App Engine SDK lives at /usr/local/google_appengine. If it doesn’t, either symlink it or pass the –gae-lib-root flag.
You only really need –with-gae and –without-sandbox flags, but I like the other flags. Type nosetests –help for a full description of the commands available.
Now let’s write some tests.
Writing tests
Now let’s create a new file for tests. Tipfy has a concept of apps within a project (think Django apps), so for this example, I’ll create a file called tests.py in each app directory for each organization (we’ll have to remember to create a setting in app.yaml to not upload this file, but this isn’t crucial). The responsibility of the tests in this file will be to run the tests for the app it’s colocated with. It’d be equally valid to create a test directory.
Here’s our tests.py:
import unittest
from tipfy import RequestHandler, Tipfy
import urls
class TestHandler(unittest.TestCase):
def setUp(self):
self.app = Tipfy(rules=urls.get_rules(None))
self.client = self.app.get_test_client()
def test_hello_world_handler(self):
response = self.client.get('/', follow_redirects=True)
self.assertEquals(response.data, "Hello BLAH")
def test_pretty_hello_world_handler(self):
response = self.client.get('/pretty')
self.assertTrue("Hello, World!" in response.data)
Let’s talk through what we’re doing here step by step:
def setUp(self):
self.app = Tipfy(rules=urls.get_rules(None))
self.client = self.app.get_test_client()
If you’re using to Python testing, this shouldn’t look too surprising to you. The setUp function is run before each test. We’re doing two things here:
- Initialize an instance of the app. We’ve imported the urls module from this app, so we can call get_rules() on it to get our URL mappings. We’re passing None to this because it expects an app, but as luck would have it, the “Hello World” demo doesn’t actually use this paramter.
- We’re initializing an instance of the test client. This is what we’ll be using to make requests
Now let’s talk about the tests
def test_hello_world_handler(self):
response = self.client.get('/', follow_redirects=True)
self.assertEquals(response.data, "Hello BLAH")
def test_pretty_hello_world_handler(self):
response = self.client.get('/pretty')
self.assertTrue("Hello, World!" in response.data)
In test_hello_world_handler(), we use self.client.get() to make a call to the”/” URL. Note that we’ve passed a follow_redirects argument; we don’t actually need this. This is just something I copied over from Rodrigo’s original testing example. We test to ensure that the response equals the output.
In our second test, we test the “pretty” version of this handler. We look for a String inside, but really it’s up to us how we want to do this. In general, we don’t want to look for an exact match of the output, since this makes our test extremely brittle and we’ll end up either not maintaining or deleting this test.
Advanced users will likely have all the handlers extend a BaseHandler RequestHandler class and call self.render(). We can point the render method to a Mock method, then try to capture the context parameters that were passed. (this is a bit out of scope for this post, but I may follow up this post with some quick samples of how to do Mocking – I like Michael Foord’s Mock library.
Writing tests with the datastore
Let’s do something a bit more interesting. Let’s run some tests with the datastore. We’ll also demonstrate some other ways of testing Tipfy. Let’s consider the following, updated code snippet:
# Install nose and nosegae:
# sudo easy_install nose
# sudo easy_install nosegae
#
# run via:
# nosetests --with-gae --without-sandbox -v
import unittest
from tipfy import RequestHandler, Rule, Tipfy
# Need this import for testing
from google.appengine.api import apiproxy_stub_map, datastore_file_stub
from google.appengine.ext import db
import urls
class Comment(db.Model):
body = db.StringProperty()
class TestHandler(unittest.TestCase):
def setUp(self):
"""
We use this to clear the datastore. Thanks to Gaetestbed for
his example here:
https://github.com/jgeewax/gaetestbed/blob/master/gaetestbed/datastore.py
"""
datastore_stub = apiproxy_stub_map.apiproxy._APIProxyStubMap__stub_map['datastore_v3']
datastore_stub.Clear()
# We're importing rules from the sample app
# The sample app doesn't require an app
self.app = Tipfy(rules=urls.get_rules(None))
self.client = self.app.get_test_client()
def test_hello_world_handler(self):
response = self.client.get('/', follow_redirects=True)
self.assertEquals(response.data, "Hello BLAH")
def test_pretty_hello_world_handler(self):
response = self.client.get('/pretty')
self.assertTrue("Hello, World!" in response.data)
def test_save_comment(self):
class DatastorePostHandler(RequestHandler):
def post(self):
body = self.request.form.get("body")
comment = Comment()
comment.body = body
comment.save()
return "OK"
rules = [
Rule('/ds', endpoint='ds', handler=DatastorePostHandler),
]
app = Tipfy(rules=rules)
client = app.get_test_client()
response = client.post('/ds')
self.assertEquals(response.data, "OK")
comments = Comment.all().fetch(100)
self.assertEquals(1, len(comments))
Revisiting the setUp() method, we see that we have a new line of code:
datastore_stub = apiproxy_stub_map.apiproxy._APIProxyStubMap__stub_map['datastore_v3'] datastore_stub.Clear()
Between test invocations, the datastore stub is NOT cleared. This lets us do it, since the last thing we want is to have state persist between tests. That’s a very bad practice I occasionally see in “clever” attempts to save lines of code. Don’t do it. It causes flaky tests and will give you hours of pain. Reset your state and rebuild it each time.
test_save_comment() defines a handler and a set of rules for our Tipfy instance. We probably won’t be doing this for non-trivial applications, since the whole point is to test some handler code we wrote, but it serves our purpose for this example. We want to test for a side effect – in this case, that a comment was saved. In a more complete test, we would not only test for the number of comments, but we’d also test that the body was saved. Notice the difference in our call to client.post() – this invokes an HTTP POST instead of an HTTP GET.
When we run nosetests with the command above, we get:
$ nosetests -d --with-gae --without-sandbox -v test_hello_world_handler (apps.hello_world.tests.TestHandler) ... ok test_pretty_hello_world_handler (apps.hello_world.tests.TestHandler) ... ok test_save_comment (apps.hello_world.tests.TestHandler) ... ok ---------------------------------------------------------------------- Ran 3 tests in 0.206s OK
And life is good again.
Final notes on testing
I’m not one of these people that believe that 100% test coverage, or even 80% test coverage is needed for a project to be well covered. The payoff for that much coverage often involves lots and lots of code is relatively minor, especially for trivial code paths.
I also see a lot of developers completely isolate each layer of the stack. In the datastore example above, these developers would have completely mocked out the datastore layer. I don’t find this to be a useful practice by default, as you end up testing your mocks and not the code. There are cases where this practice is useful, but in most cases, you will have more confidence in your code if you take the time to define a correct set of fixtures. Where you’ll 100% want mocks are places where you have complex or external services that can be flaky, or when you need to replicate failure conditions that are difficult to programmatically cause in your code.
Don’t think of testing as a replacement for QA because it’s not. In web testing, think of it as a replacement for opening a browser and clicking. When you discover a bug, you write a test for it and try to fix it, because in most cases setting up the error state will be much easier programmatically than manually. You’re always going to have to do browser testing at some point, but it’s time consuming, especially if you need your data in a specific state. You could go the Selenium route for full coverage, but in my experience (people are going to disagree with me on this – get ready for comment/Twitter trolling), Selenium tests, while providing a high level of confidence, also are extremely brittle and are a maintenance nightmare if you have too many of them. You’ll want to write as many tests as you can outside the browser environment and save Selenium for the minority of your user flows that are critical – write Javascript unit tests instead of Selenium tests for client side functionality. I’ve used JsUnit before and heard good things about Jasmine but never had experience with it myself.
And my last tip? Do what works for your team. But do write tests, because it’s one of those practices that will pay off over time if you write AND maintain them well.
- Ikai
