Some time ago the github repo has been prepared to run tests on travis-ci and appveyor. At the same time, it was configured to ship coverage reports to https://codecov.io site.
A little bit of introduction
- coverage is a metric (sometimes not as precise as I'd wish) too see how many parts of a piece of code gets tested
- Web2py ATM includes tests only for gluon code, not for gluon/contrib modules.
- we leverage travis-ci and appveyor to test every commit against python 2.5, 2.6, 2.7 and pypy
- web2py extrapolated pydal in a separate repo, but the same considerations apply...code runs for SQLite, MySQL,Postgresql,mongodb,GAE on travis-ci and mssql on appveyor
Here I'll explain how you can replicate what happens on travis-ci and on codecov.io with your python interpreter.
Web2py executes tests simply with ./web2py.py --run_system_tests . A few of them are run on pydal just to test if something breaks with the integration. Extensive pydal tests are instead run within the separate project
By default, tests are run against a "sqlite:memory" database. To force both web2py and pydal tests to happen on a particular database, you need to set an environmental variable “DB” that is the uri of your database.
E.g.: given that you'd like to run your tests on your own postgresql instance, you'd just need to start a shell and
cd path/to/web2py export DB=postgres://username:password@localhost/test ./web2py.py --run_system_tests # for web2py # python -m unittest -v tests # for pydal
If you're on Windows, instead of “export” you should type “set”
NB: your database should be already there and EMPTY (meaning, no tables in it)
To enable coverage reports, you should have the coverage module installed
pip install --upgrade coverage
Again, web2py smooths out lots of things for you:
./web2py.py --run_system_tests --with_coverage # for web2py
coverage run -m unittest -v tests # for pydal
will run tests and measure coverage for you.
There's a default coverage.ini file right in gluon/tests and a .coveragerc in the root of pydal.
PS: To set your own configuration, just set the env variable COVERAGE_PROCESS_START pointing to the path.
Given all web2py defaults, after running tests with coverage, you'll see a new file in the web2py path that is called roughly “.coverage.something” . Those file are holding all the “raw” data for the coverage report.
Now we should build a “readable” form of this “raw report”.
NB: by default coverage collects all available coverage “runs” to form a single report. It's best to delete all those files, run the tests and generate a report containing only the results from the latest run.
Usually the coverage config is in a file called .coveragerc. We have it in gluon/tests/coverage.ini, so we need to supply the path to the config file. To do this, coverage ships with nice helpers.... so, by default, you should do
cd /path/to/web2py coverage erase #deletes previously collected data ./web2py.py --run_system_tests --with_coverage coverage combine # sums up all coverage reports generated coverage html --rcfile=gluon/tests/coverage.ini # creates a nice html report
Once you created the html report, you can navigate to /path/to/web2py/coverage_html_report/index.html to see it.
That report closely resembles what you have on https://codecov.io/github/web2py/web2py . If you click on a file you'll see what lines are covered by tests and what lines are not (red ones).
How to improve coverage
Choose a piece of code that isn't tested (all the red lines aren't) and figure out a way to write a test for them. Unfortunately if you're not familiar with unittests you'll need to spend a little bit of time to learn the syntax. Existing tests in gluon/tests can definitely help you out in the process: it's not **that** hard :-P
Let's say I'd like to improve tests for pydal's validate_and_insert() function (ATM untested).
Tests are unittests included in gluon/tests/* (or tests/ for pydal).
NB: pydal tests are a little bit different from any other, in the sense that every tests finds an EMPTY database and should leave the database EMPTY when it's finished. So, all pydal tests REQUIRE that you use as a uri the variable DEFAULT_URI (is the one that gets used when you pass the DB env variable), and that you remember to .drop() your tables at the end, so that your tests will never get in the way with other tests.
Use meaningful class names for tests and comment as much as you can to help others see why you built that tests and to verify what condition.
class TestValidateAndInsert(unittest.TestCase): def testRun(self): import datetime from gluon.validators import IS_INT_IN_RANGE db = DAL(DEFAULT_URI, check_reserved=['all']) db.define_table('val_and_insert', Field('aa'), Field('bb', 'integer', requires=IS_INT_IN_RANGE(1,5)) ) rtn = db.val_and_insert.validate_and_insert(aa='test1', bb=2) self.assertEqual(rtn.id, 1) #errors should be empty self.assertEqual(len(rtn.errors.keys()), 0) #this insert won't pass rtn = db.val_and_insert.validate_and_insert(bb="a") #the returned id should be None self.assertEqual(rtn.id, None) #an error message should be in rtn.errors.bb self.assertNotEqual(rtn.errors.bb, None) #cleanup table db.val_and_insert.drop()
Ok, time to test it again....
Watch the html report..... nice, we have validate_and_insert tested !!!
Now, go help developers adding tests, as everyone will benefit from them:
- developers adding features are istantly notified that they may have done something wrong, breaking the existing functionality
- users will upgrade knowing that there's no worries because something is tested
And all of this will happen as soon as something gets committed: how wonderful is that !!!