Testing Against External Processes with pytest-xprocess

Introduction

With the advent of micro-services or simply componentization of complex applications, more often than not it is a good idea to write at least few integration tests to make sure that different components can interact in the intended way. However, these components may run in their own process, may take some time to start-up, and may require other processes to be run first.
This article aims to give some ideas on how to solve the following problems, using pytest:

•  How to start an external process before the tests run, and make sure tests start running only once the process is fully initialized?
• Starting an external process is usually resource and time consuming, how can the whole test suite be setup to run the external process only once before all tests run, and shutdown only once all tests are completed?
• Given a system under test (SUT), which depends on a process A, which in turn depends on a process B, how can all the processes be start in the correct order before the tests run?

Starting an External Process: pytest-xprocess

pytest-xprocess is a pytest plugin for managing processes across test runs. As of today, it is still an experimental plugin, and it is already used in open-source projects like wergzeug, which is a building block of Flask.
A proof-of-concept would be simply to create a simple process, like an echo server, simulate a few seconds of starting time, and run some tests against it. It should demonstrate:

1. How to start a process before running a test
2. Wait for the process to be fully initialized
3. Run the test
4. Terminate the process

Here is the code:

The echo_server.py is a simple echo server, is receives some data, and returns it back, prefixed by the string "Echo: ". It takes three seconds to start, and it doesn't accept incoming requests before it is fully started.

The test_echo_server.py is where the action happens. The start_server fixture defines a subclass of ProcessStarter, which defines how the process should be run, and how to detect that it is fully initialized. The args property is the command line of the process to start. The first element of the array is the executable to start (in this case, the Python interpreter), and the rest of the array elements are the arguments to pass to the executable, in this case, the Python script to execute, and the port of the server. The pattern property, is a regular expression that is used to determine once the process is fully up. The regex is tested against the output of the process, whether it is stdout or stderr.

Everything is handled in the xprocess.ensure function. Once the xprocess.ensure function returns, the process is ready, so the fixture can "yield" to let the test run. Once the test has run (successfully or not), the execution path comes back into the fixture, after the "yield" statement, and terminates the process.

To run the code, one can install pytest-xprocess (pip install pytest-xprocess), and run the pytest command in a prompt to run the tests.

Setup / Teardown Strategies

Now, adding a second test would look like this:

The problem is that, the process will be started up and terminated for each test, and this is not always desirable - indeed, starting a process is generally a costly operation, and this can slow down the tests considerably, not mentioning the waste of CPU resources. The way to go around it is to change the scope of the fixture, to either "module" or "session", if the process has to be started only once for all tests in all module. For a "session" scope, it can be a good idea to move the fixture in the conftest.py file, which is the place that should contain fixtures shared across different modules.

Manage Process Dependencies

In larger projects, there can be several processes depending on each other, and some of them may required to be started first in order to proceed. This section will demonstrate how to handle this easily. As an example, we have two processes:

1. An Italic TCP server, which takes a string as input, and outputs it with <i></i> HTML bold tags
2. The Echo TCP server, which calls the italic server, and echoes back the result

In this architecture, server 1. needs to be started first, and server 2. needs to wait for server 1. to be started in order to function properly.
Using pytest fixture dependencies, the code to solve this problem looks like this:
The noticeable line of code is in the conftest.py file:

def start_echo_server(xprocess, start_italic_server):

The start_italic_server parameter, indicates that the start_echo_server fixture must be run only once the start_italic_fixture has been run. This is a very neat way of ensuring that everything is started and terminated in the right order.

Drawbacks of pytest-xprocess

Even though the pytest-xprocess plugin looks promising, there are a few drawbacks.

1. By default, the ProcessStarter can read only the 50 first lines of output of the process, to determine if it has started or not. This number can be increased by overriding the filter_lines function. Another quirk, is that xprocess will make a pause of 0.1sec if it can't read an output line, and if no output is generated by the process, it will consider that the process returned no data and still count this as a line. This means, that xprocess will timeout after around 50 lines * 0.1s = 5s. Increasing the number of lines read can increase this timeout, but this seems very weird.
2. xprocess can't be easily used if the process doesn't output anything to stdout or stderr, but in a log file instead (this is not common, but some bad coded processes do that). One can try to heavily customise the ProcessStarter class, by overriding the relevant functions, but the trick is that the "process started" strings can appear multiple times, if the process constantly appends output in the same log file.
3. xprocess just can't handle a process outputting non-ascii characters
4. There is no much support for parallelization, e.g. to start two processes at the same time, if they can be started independently

Conclusion

In this article, we have talked about how to write tests running against an external process, in a clean and concise way. The pytest-xprocess plugin is quite promising in this area, and as of today, this is maybe one of the best choices to achieve this goal, despite of the few drawbacks that hopefully will be fixed. Any other testing library, fulfilling these requirements, should be good to do the job:

1. Encapsulate the starting of a process (considering the different platforms / OS)
2. Have a mechanism to detect that the process is fully started
3. Handle the dependencies, in the starting order of the processes

Please feel free to comment about other tools that could work, especially using Python unittest, or if there is another approach that could be used.