Parametrizing tests

pytest allows to easily parametrize test functions. For basic docs, see Parametrizing fixtures and test functions.

In the following we provide some examples using the builtin mechanisms.

Generating parameters combinations, depending on command line

Let’s say we want to execute a test with different computation parameters and the parameter range shall be determined by a command line argument. Let’s first write a simple (do-nothing) computation test:

# content of test_compute.py

def test_compute(param1):
    assert param1 < 4

Now we add a test configuration like this:

# content of conftest.py

def pytest_addoption(parser):
    parser.addoption("--all", action="store_true",
        help="run all combinations")

def pytest_generate_tests(metafunc):
    if 'param1' in metafunc.fixturenames:
        if metafunc.config.option.all:
            end = 5
        else:
            end = 2
        metafunc.parametrize("param1", range(end))

This means that we only run 2 tests if we do not pass --all:

$ py.test -q test_compute.py
..
2 passed in 0.01 seconds

We run only two computations, so we see two dots. let’s run the full monty:

$ py.test -q --all
....F
================================= FAILURES =================================
_____________________________ test_compute[4] ______________________________

param1 = 4

    def test_compute(param1):
>       assert param1 < 4
E       assert 4 < 4

test_compute.py:3: AssertionError
1 failed, 4 passed in 0.01 seconds

As expected when running the full range of param1 values we’ll get an error on the last one.

A quick port of “testscenarios”

Here is a quick port to run tests configured with test scenarios, an add-on from Robert Collins for the standard unittest framework. We only have to work a bit to construct the correct arguments for pytest’s Metafunc.parametrize():

# content of test_scenarios.py

def pytest_generate_tests(metafunc):
    idlist = []
    argvalues = []
    for scenario in metafunc.cls.scenarios:
        idlist.append(scenario[0])
        items = scenario[1].items()
        argnames = [x[0] for x in items]
        argvalues.append(([x[1] for x in items]))
    metafunc.parametrize(argnames, argvalues, ids=idlist, scope="class")

scenario1 = ('basic', {'attribute': 'value'})
scenario2 = ('advanced', {'attribute': 'value2'})

class TestSampleWithScenarios:
    scenarios = [scenario1, scenario2]

    def test_demo1(self, attribute):
        assert isinstance(attribute, str)

    def test_demo2(self, attribute):
        assert isinstance(attribute, str)

this is a fully self-contained example which you can run with:

$ py.test test_scenarios.py
=========================== test session starts ============================
platform linux -- Python 3.4.0 -- py-1.4.25 -- pytest-2.6.3
collected 4 items

test_scenarios.py ....

========================= 4 passed in 0.01 seconds =========================

If you just collect tests you’ll also nicely see ‘advanced’ and ‘basic’ as variants for the test function:

$ py.test --collect-only test_scenarios.py
=========================== test session starts ============================
platform linux -- Python 3.4.0 -- py-1.4.25 -- pytest-2.6.3
collected 4 items
<Module 'test_scenarios.py'>
  <Class 'TestSampleWithScenarios'>
    <Instance '()'>
      <Function 'test_demo1[basic]'>
      <Function 'test_demo2[basic]'>
      <Function 'test_demo1[advanced]'>
      <Function 'test_demo2[advanced]'>

=============================  in 0.01 seconds =============================

Note that we told metafunc.parametrize() that your scenario values should be considered class-scoped. With pytest-2.3 this leads to a resource-based ordering.

Deferring the setup of parametrized resources

The parametrization of test functions happens at collection time. It is a good idea to setup expensive resources like DB connections or subprocess only when the actual test is run. Here is a simple example how you can achieve that, first the actual test requiring a db object:

# content of test_backends.py

import pytest
def test_db_initialized(db):
    # a dummy test
    if db.__class__.__name__ == "DB2":
        pytest.fail("deliberately failing for demo purposes")

We can now add a test configuration that generates two invocations of the test_db_initialized function and also implements a factory that creates a database object for the actual test invocations:

# content of conftest.py
import pytest

def pytest_generate_tests(metafunc):
    if 'db' in metafunc.fixturenames:
        metafunc.parametrize("db", ['d1', 'd2'], indirect=True)

class DB1:
    "one database object"
class DB2:
    "alternative database object"

@pytest.fixture
def db(request):
    if request.param == "d1":
        return DB1()
    elif request.param == "d2":
        return DB2()
    else:
        raise ValueError("invalid internal test config")

Let’s first see how it looks like at collection time:

$ py.test test_backends.py --collect-only
=========================== test session starts ============================
platform linux -- Python 3.4.0 -- py-1.4.25 -- pytest-2.6.3
collected 2 items
<Module 'test_backends.py'>
  <Function 'test_db_initialized[d1]'>
  <Function 'test_db_initialized[d2]'>

=============================  in 0.00 seconds =============================

And then when we run the test:

$ py.test -q test_backends.py
.F
================================= FAILURES =================================
_________________________ test_db_initialized[d2] __________________________

db = <conftest.DB2 object at 0x2b24a8545390>

    def test_db_initialized(db):
        # a dummy test
        if db.__class__.__name__ == "DB2":
>           pytest.fail("deliberately failing for demo purposes")
E           Failed: deliberately failing for demo purposes

test_backends.py:6: Failed
1 failed, 1 passed in 0.02 seconds

The first invocation with db == "DB1" passed while the second with db == "DB2" failed. Our db fixture function has instantiated each of the DB values during the setup phase while the pytest_generate_tests generated two according calls to the test_db_initialized during the collection phase.

Parametrizing test methods through per-class configuration

Here is an example pytest_generate_function function implementing a parametrization scheme similar to Michael Foord’s unittest parameterizer but in a lot less code:

# content of ./test_parametrize.py
import pytest

def pytest_generate_tests(metafunc):
    # called once per each test function
    funcarglist = metafunc.cls.params[metafunc.function.__name__]
    argnames = list(funcarglist[0])
    metafunc.parametrize(argnames, [[funcargs[name] for name in argnames]
            for funcargs in funcarglist])

class TestClass:
    # a map specifying multiple argument sets for a test method
    params = {
        'test_equals': [dict(a=1, b=2), dict(a=3, b=3), ],
        'test_zerodivision': [dict(a=1, b=0), ],
    }

    def test_equals(self, a, b):
        assert a == b

    def test_zerodivision(self, a, b):
        pytest.raises(ZeroDivisionError, "a/b")

Our test generator looks up a class-level definition which specifies which argument sets to use for each test function. Let’s run it:

$ py.test -q
F..
================================= FAILURES =================================
________________________ TestClass.test_equals[1-2] ________________________

self = <test_parametrize.TestClass object at 0x2b0ed5d8cdd8>, a = 1, b = 2

    def test_equals(self, a, b):
>       assert a == b
E       assert 1 == 2

test_parametrize.py:18: AssertionError
1 failed, 2 passed in 0.02 seconds

Indirect parametrization with multiple fixtures

Here is a stripped down real-life example of using parametrized testing for testing serialization of objects between different python interpreters. We define a test_basic_objects function which is to be run with different sets of arguments for its three arguments:

  • python1: first python interpreter, run to pickle-dump an object to a file
  • python2: second interpreter, run to pickle-load an object from a file
  • obj: object to be dumped/loaded
"""
module containing a parametrized tests testing cross-python
serialization via the pickle module.
"""
import py
import pytest

pythonlist = ['python2.6', 'python2.7', 'python3.4']
@pytest.fixture(params=pythonlist)
def python1(request, tmpdir):
    picklefile = tmpdir.join("data.pickle")
    return Python(request.param, picklefile)

@pytest.fixture(params=pythonlist)
def python2(request, python1):
    return Python(request.param, python1.picklefile)

class Python:
    def __init__(self, version, picklefile):
        self.pythonpath = py.path.local.sysfind(version)
        if not self.pythonpath:
            pytest.skip("%r not found" %(version,))
        self.picklefile = picklefile
    def dumps(self, obj):
        dumpfile = self.picklefile.dirpath("dump.py")
        dumpfile.write(py.code.Source("""
            import pickle
            f = open(%r, 'wb')
            s = pickle.dump(%r, f)
            f.close()
        """ % (str(self.picklefile), obj)))
        py.process.cmdexec("%s %s" %(self.pythonpath, dumpfile))

    def load_and_is_true(self, expression):
        loadfile = self.picklefile.dirpath("load.py")
        loadfile.write(py.code.Source("""
            import pickle
            f = open(%r, 'rb')
            obj = pickle.load(f)
            f.close()
            res = eval(%r)
            if not res:
                raise SystemExit(1)
        """ % (str(self.picklefile), expression)))
        print (loadfile)
        py.process.cmdexec("%s %s" %(self.pythonpath, loadfile))

@pytest.mark.parametrize("obj", [42, {}, {1:3},])
def test_basic_objects(python1, python2, obj):
    python1.dumps(obj)
    python2.load_and_is_true("obj == %s" % obj)

Running it results in some skips if we don’t have all the python interpreters installed and otherwise runs all combinations (5 interpreters times 5 interpreters times 3 objects to serialize/deserialize):

. $ py.test -rs -q multipython.py
..................FFFFFF...
================================= FAILURES =================================
________________ test_basic_objects[python3.4-python2.6-42] ________________

python1 = <multipython.Python object at 0x2b02a1bc79b0>
python2 = <multipython.Python object at 0x2b02a1bc7630>, obj = 42

    @pytest.mark.parametrize("obj", [42, {}, {1:3},])
    def test_basic_objects(python1, python2, obj):
        python1.dumps(obj)
>       python2.load_and_is_true("obj == %s" % obj)

multipython.py:51:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
multipython.py:46: in load_and_is_true
    py.process.cmdexec("%s %s" %(self.pythonpath, loadfile))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cmd = '/home/hpk/bin/python2.6 /tmp/pytest-21/test_basic_objects_python3_4_p0/load.py'

    def cmdexec(cmd):
        """ return unicode output of executing 'cmd' in a separate process.

        raise cmdexec.Error exeception if the command failed.
        the exception will provide an 'err' attribute containing
        the error-output from the command.
        if the subprocess module does not provide a proper encoding/unicode strings
        sys.getdefaultencoding() will be used, if that does not exist, 'UTF-8'.
        """
        process = subprocess.Popen(cmd, shell=True,
                universal_newlines=True,
                stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        out, err = process.communicate()
        if sys.version_info[0] < 3: # on py3 we get unicode strings, on py2 not
            try:
                default_encoding = sys.getdefaultencoding() # jython may not have it
            except AttributeError:
                default_encoding = sys.stdout.encoding or 'UTF-8'
            out = unicode(out, process.stdout.encoding or default_encoding)
            err = unicode(err, process.stderr.encoding or default_encoding)
        status = process.poll()
        if status:
>           raise ExecutionFailed(status, status, cmd, out, err)
E           py.process.cmdexec.Error: ExecutionFailed: 1  /home/hpk/bin/python2.6 /tmp/pytest-21/test_basic_objects_python3_4_p0/load.py
E           Traceback (most recent call last):
E             File "/tmp/pytest-21/test_basic_objects_python3_4_p0/load.py", line 4, in <module>
E               obj = pickle.load(f)
E             File "/usr/lib/python2.6/pickle.py", line 1370, in load
E               return Unpickler(file).load()
E             File "/usr/lib/python2.6/pickle.py", line 858, in load
E               dispatch[key](self)
E             File "/usr/lib/python2.6/pickle.py", line 886, in load_proto
E               raise ValueError, "unsupported pickle protocol: %d" % proto
E           ValueError: unsupported pickle protocol: 3

../../../.tox/regen/lib/python3.4/site-packages/py/_process/cmdexec.py:28: Error
--------------------------- Captured stdout call ---------------------------
/tmp/pytest-21/test_basic_objects_python3_4_p0/load.py
_______________ test_basic_objects[python3.4-python2.6-obj1] _______________

python1 = <multipython.Python object at 0x2b02a1b66f28>
python2 = <multipython.Python object at 0x2b02a1b66198>, obj = {}

    @pytest.mark.parametrize("obj", [42, {}, {1:3},])
    def test_basic_objects(python1, python2, obj):
        python1.dumps(obj)
>       python2.load_and_is_true("obj == %s" % obj)

multipython.py:51:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
multipython.py:46: in load_and_is_true
    py.process.cmdexec("%s %s" %(self.pythonpath, loadfile))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cmd = '/home/hpk/bin/python2.6 /tmp/pytest-21/test_basic_objects_python3_4_p1/load.py'

    def cmdexec(cmd):
        """ return unicode output of executing 'cmd' in a separate process.

        raise cmdexec.Error exeception if the command failed.
        the exception will provide an 'err' attribute containing
        the error-output from the command.
        if the subprocess module does not provide a proper encoding/unicode strings
        sys.getdefaultencoding() will be used, if that does not exist, 'UTF-8'.
        """
        process = subprocess.Popen(cmd, shell=True,
                universal_newlines=True,
                stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        out, err = process.communicate()
        if sys.version_info[0] < 3: # on py3 we get unicode strings, on py2 not
            try:
                default_encoding = sys.getdefaultencoding() # jython may not have it
            except AttributeError:
                default_encoding = sys.stdout.encoding or 'UTF-8'
            out = unicode(out, process.stdout.encoding or default_encoding)
            err = unicode(err, process.stderr.encoding or default_encoding)
        status = process.poll()
        if status:
>           raise ExecutionFailed(status, status, cmd, out, err)
E           py.process.cmdexec.Error: ExecutionFailed: 1  /home/hpk/bin/python2.6 /tmp/pytest-21/test_basic_objects_python3_4_p1/load.py
E           Traceback (most recent call last):
E             File "/tmp/pytest-21/test_basic_objects_python3_4_p1/load.py", line 4, in <module>
E               obj = pickle.load(f)
E             File "/usr/lib/python2.6/pickle.py", line 1370, in load
E               return Unpickler(file).load()
E             File "/usr/lib/python2.6/pickle.py", line 858, in load
E               dispatch[key](self)
E             File "/usr/lib/python2.6/pickle.py", line 886, in load_proto
E               raise ValueError, "unsupported pickle protocol: %d" % proto
E           ValueError: unsupported pickle protocol: 3

../../../.tox/regen/lib/python3.4/site-packages/py/_process/cmdexec.py:28: Error
--------------------------- Captured stdout call ---------------------------
/tmp/pytest-21/test_basic_objects_python3_4_p1/load.py
_______________ test_basic_objects[python3.4-python2.6-obj2] _______________

python1 = <multipython.Python object at 0x2b02a1bcd320>
python2 = <multipython.Python object at 0x2b02a1bcd358>, obj = {1: 3}

    @pytest.mark.parametrize("obj", [42, {}, {1:3},])
    def test_basic_objects(python1, python2, obj):
        python1.dumps(obj)
>       python2.load_and_is_true("obj == %s" % obj)

multipython.py:51:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
multipython.py:46: in load_and_is_true
    py.process.cmdexec("%s %s" %(self.pythonpath, loadfile))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cmd = '/home/hpk/bin/python2.6 /tmp/pytest-21/test_basic_objects_python3_4_p2/load.py'

    def cmdexec(cmd):
        """ return unicode output of executing 'cmd' in a separate process.

        raise cmdexec.Error exeception if the command failed.
        the exception will provide an 'err' attribute containing
        the error-output from the command.
        if the subprocess module does not provide a proper encoding/unicode strings
        sys.getdefaultencoding() will be used, if that does not exist, 'UTF-8'.
        """
        process = subprocess.Popen(cmd, shell=True,
                universal_newlines=True,
                stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        out, err = process.communicate()
        if sys.version_info[0] < 3: # on py3 we get unicode strings, on py2 not
            try:
                default_encoding = sys.getdefaultencoding() # jython may not have it
            except AttributeError:
                default_encoding = sys.stdout.encoding or 'UTF-8'
            out = unicode(out, process.stdout.encoding or default_encoding)
            err = unicode(err, process.stderr.encoding or default_encoding)
        status = process.poll()
        if status:
>           raise ExecutionFailed(status, status, cmd, out, err)
E           py.process.cmdexec.Error: ExecutionFailed: 1  /home/hpk/bin/python2.6 /tmp/pytest-21/test_basic_objects_python3_4_p2/load.py
E           Traceback (most recent call last):
E             File "/tmp/pytest-21/test_basic_objects_python3_4_p2/load.py", line 4, in <module>
E               obj = pickle.load(f)
E             File "/usr/lib/python2.6/pickle.py", line 1370, in load
E               return Unpickler(file).load()
E             File "/usr/lib/python2.6/pickle.py", line 858, in load
E               dispatch[key](self)
E             File "/usr/lib/python2.6/pickle.py", line 886, in load_proto
E               raise ValueError, "unsupported pickle protocol: %d" % proto
E           ValueError: unsupported pickle protocol: 3

../../../.tox/regen/lib/python3.4/site-packages/py/_process/cmdexec.py:28: Error
--------------------------- Captured stdout call ---------------------------
/tmp/pytest-21/test_basic_objects_python3_4_p2/load.py
________________ test_basic_objects[python3.4-python2.7-42] ________________

python1 = <multipython.Python object at 0x2b02a1bc78d0>
python2 = <multipython.Python object at 0x2b02a1bc7128>, obj = 42

    @pytest.mark.parametrize("obj", [42, {}, {1:3},])
    def test_basic_objects(python1, python2, obj):
        python1.dumps(obj)
>       python2.load_and_is_true("obj == %s" % obj)

multipython.py:51:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
multipython.py:46: in load_and_is_true
    py.process.cmdexec("%s %s" %(self.pythonpath, loadfile))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cmd = '/home/hpk/bin/python2.7 /tmp/pytest-21/test_basic_objects_python3_4_p3/load.py'

    def cmdexec(cmd):
        """ return unicode output of executing 'cmd' in a separate process.

        raise cmdexec.Error exeception if the command failed.
        the exception will provide an 'err' attribute containing
        the error-output from the command.
        if the subprocess module does not provide a proper encoding/unicode strings
        sys.getdefaultencoding() will be used, if that does not exist, 'UTF-8'.
        """
        process = subprocess.Popen(cmd, shell=True,
                universal_newlines=True,
                stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        out, err = process.communicate()
        if sys.version_info[0] < 3: # on py3 we get unicode strings, on py2 not
            try:
                default_encoding = sys.getdefaultencoding() # jython may not have it
            except AttributeError:
                default_encoding = sys.stdout.encoding or 'UTF-8'
            out = unicode(out, process.stdout.encoding or default_encoding)
            err = unicode(err, process.stderr.encoding or default_encoding)
        status = process.poll()
        if status:
>           raise ExecutionFailed(status, status, cmd, out, err)
E           py.process.cmdexec.Error: ExecutionFailed: 1  /home/hpk/bin/python2.7 /tmp/pytest-21/test_basic_objects_python3_4_p3/load.py
E           Traceback (most recent call last):
E             File "/tmp/pytest-21/test_basic_objects_python3_4_p3/load.py", line 4, in <module>
E               obj = pickle.load(f)
E             File "/usr/lib/python2.7/pickle.py", line 1378, in load
E               return Unpickler(file).load()
E             File "/usr/lib/python2.7/pickle.py", line 858, in load
E               dispatch[key](self)
E             File "/usr/lib/python2.7/pickle.py", line 886, in load_proto
E               raise ValueError, "unsupported pickle protocol: %d" % proto
E           ValueError: unsupported pickle protocol: 3

../../../.tox/regen/lib/python3.4/site-packages/py/_process/cmdexec.py:28: Error
--------------------------- Captured stdout call ---------------------------
/tmp/pytest-21/test_basic_objects_python3_4_p3/load.py
_______________ test_basic_objects[python3.4-python2.7-obj1] _______________

python1 = <multipython.Python object at 0x2b02a1bd2550>
python2 = <multipython.Python object at 0x2b02a1bd23c8>, obj = {}

    @pytest.mark.parametrize("obj", [42, {}, {1:3},])
    def test_basic_objects(python1, python2, obj):
        python1.dumps(obj)
>       python2.load_and_is_true("obj == %s" % obj)

multipython.py:51:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
multipython.py:46: in load_and_is_true
    py.process.cmdexec("%s %s" %(self.pythonpath, loadfile))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cmd = '/home/hpk/bin/python2.7 /tmp/pytest-21/test_basic_objects_python3_4_p4/load.py'

    def cmdexec(cmd):
        """ return unicode output of executing 'cmd' in a separate process.

        raise cmdexec.Error exeception if the command failed.
        the exception will provide an 'err' attribute containing
        the error-output from the command.
        if the subprocess module does not provide a proper encoding/unicode strings
        sys.getdefaultencoding() will be used, if that does not exist, 'UTF-8'.
        """
        process = subprocess.Popen(cmd, shell=True,
                universal_newlines=True,
                stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        out, err = process.communicate()
        if sys.version_info[0] < 3: # on py3 we get unicode strings, on py2 not
            try:
                default_encoding = sys.getdefaultencoding() # jython may not have it
            except AttributeError:
                default_encoding = sys.stdout.encoding or 'UTF-8'
            out = unicode(out, process.stdout.encoding or default_encoding)
            err = unicode(err, process.stderr.encoding or default_encoding)
        status = process.poll()
        if status:
>           raise ExecutionFailed(status, status, cmd, out, err)
E           py.process.cmdexec.Error: ExecutionFailed: 1  /home/hpk/bin/python2.7 /tmp/pytest-21/test_basic_objects_python3_4_p4/load.py
E           Traceback (most recent call last):
E             File "/tmp/pytest-21/test_basic_objects_python3_4_p4/load.py", line 4, in <module>
E               obj = pickle.load(f)
E             File "/usr/lib/python2.7/pickle.py", line 1378, in load
E               return Unpickler(file).load()
E             File "/usr/lib/python2.7/pickle.py", line 858, in load
E               dispatch[key](self)
E             File "/usr/lib/python2.7/pickle.py", line 886, in load_proto
E               raise ValueError, "unsupported pickle protocol: %d" % proto
E           ValueError: unsupported pickle protocol: 3

../../../.tox/regen/lib/python3.4/site-packages/py/_process/cmdexec.py:28: Error
--------------------------- Captured stdout call ---------------------------
/tmp/pytest-21/test_basic_objects_python3_4_p4/load.py
_______________ test_basic_objects[python3.4-python2.7-obj2] _______________

python1 = <multipython.Python object at 0x2b02a1bcdeb8>
python2 = <multipython.Python object at 0x2b02a1bcdf98>, obj = {1: 3}

    @pytest.mark.parametrize("obj", [42, {}, {1:3},])
    def test_basic_objects(python1, python2, obj):
        python1.dumps(obj)
>       python2.load_and_is_true("obj == %s" % obj)

multipython.py:51:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
multipython.py:46: in load_and_is_true
    py.process.cmdexec("%s %s" %(self.pythonpath, loadfile))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cmd = '/home/hpk/bin/python2.7 /tmp/pytest-21/test_basic_objects_python3_4_p5/load.py'

    def cmdexec(cmd):
        """ return unicode output of executing 'cmd' in a separate process.

        raise cmdexec.Error exeception if the command failed.
        the exception will provide an 'err' attribute containing
        the error-output from the command.
        if the subprocess module does not provide a proper encoding/unicode strings
        sys.getdefaultencoding() will be used, if that does not exist, 'UTF-8'.
        """
        process = subprocess.Popen(cmd, shell=True,
                universal_newlines=True,
                stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        out, err = process.communicate()
        if sys.version_info[0] < 3: # on py3 we get unicode strings, on py2 not
            try:
                default_encoding = sys.getdefaultencoding() # jython may not have it
            except AttributeError:
                default_encoding = sys.stdout.encoding or 'UTF-8'
            out = unicode(out, process.stdout.encoding or default_encoding)
            err = unicode(err, process.stderr.encoding or default_encoding)
        status = process.poll()
        if status:
>           raise ExecutionFailed(status, status, cmd, out, err)
E           py.process.cmdexec.Error: ExecutionFailed: 1  /home/hpk/bin/python2.7 /tmp/pytest-21/test_basic_objects_python3_4_p5/load.py
E           Traceback (most recent call last):
E             File "/tmp/pytest-21/test_basic_objects_python3_4_p5/load.py", line 4, in <module>
E               obj = pickle.load(f)
E             File "/usr/lib/python2.7/pickle.py", line 1378, in load
E               return Unpickler(file).load()
E             File "/usr/lib/python2.7/pickle.py", line 858, in load
E               dispatch[key](self)
E             File "/usr/lib/python2.7/pickle.py", line 886, in load_proto
E               raise ValueError, "unsupported pickle protocol: %d" % proto
E           ValueError: unsupported pickle protocol: 3

../../../.tox/regen/lib/python3.4/site-packages/py/_process/cmdexec.py:28: Error
--------------------------- Captured stdout call ---------------------------
/tmp/pytest-21/test_basic_objects_python3_4_p5/load.py
6 failed, 21 passed in 1.36 seconds

Indirect parametrization of optional implementations/imports

If you want to compare the outcomes of several implementations of a given API, you can write test functions that receive the already imported implementations and get skipped in case the implementation is not importable/available. Let’s say we have a “base” implementation and the other (possibly optimized ones) need to provide similar results:

# content of conftest.py

import pytest

@pytest.fixture(scope="session")
def basemod(request):
    return pytest.importorskip("base")

@pytest.fixture(scope="session", params=["opt1", "opt2"])
def optmod(request):
    return pytest.importorskip(request.param)

And then a base implementation of a simple function:

# content of base.py
def func1():
    return 1

And an optimized version:

# content of opt1.py
def func1():
    return 1.0001

And finally a little test module:

# content of test_module.py

def test_func1(basemod, optmod):
    assert round(basemod.func1(), 3) == round(optmod.func1(), 3)

If you run this with reporting for skips enabled:

$ py.test -rs test_module.py
=========================== test session starts ============================
platform linux -- Python 3.4.0 -- py-1.4.25 -- pytest-2.6.3
collected 2 items

test_module.py .s
========================= short test summary info ==========================
SKIP [1] /tmp/doc-exec-226/conftest.py:10: could not import 'opt2'

=================== 1 passed, 1 skipped in 0.01 seconds ====================

You’ll see that we don’t have a opt2 module and thus the second test run of our test_func1 was skipped. A few notes:

  • the fixture functions in the conftest.py file are “session-scoped” because we don’t need to import more than once
  • if you have multiple test functions and a skipped import, you will see the [1] count increasing in the report
  • you can put @pytest.mark.parametrize style parametrization on the test functions to parametrize input/output values as well.