piglit (II): How to launch a tailored piglit run

Last post I gave an introduction to piglit, an open-source test suite for OpenGL implementations. On that post, I explained how to compile the source code, how to run full piglit test suite and how to analyze the results.

However, you can tailor a piglit run to execute the specific tests that match your needs. For example I might want to check if a specific subset of tests pass because I am testing a new feature in Mesa as part of my job at Igalia.

Configure the piglit run

There are several parameters that configure the piglit run to match our needs:

  • –dry-run: do not execute the tests. Very useful to check if the parameters you give are doing what you expect.
  • –valgrind: It runs Valgrind memcheck on each test program that it’s going to execute. If it finds any error, the test fails and it saves the valgrind output into results file.
  • –all-concurrent: run all tests concurrently.
  • –no-concurrency: disable concurrent test runs.
  • –sync: sync results to disk after every test so you don’t lose information if something bad happens to your system.
  • –platform {glx,x11_egl,wayland,gbm,mixed_glx_egl}: if you compiled waffle with different windows system, this is the name of the one passed to waffle.

There is a help parameter if you want further information:

Skip tests

Sometimes you prefer to skip some tests because they are causing a GPU hang or taking a lot of time and their output is not interesting for you. The way to skip a test is using the -x parameter:

Run specific tests

You can run some specific subset of tests using a filter by test name. Remember that it’s filtering by test name but not by functionality, so you might miss some test programs that check the same functionality.

Also, you can concatenate more parameters to add/remove more tests that are going to be executed.

Run standalone tests

There is another way to run standalone tests apart from using the -t argument to piglit run. You might be interested in this way if you want to run GDB on a failing tests to debug what is going on.

If we remember how was the HTML output for a given test:

piglit-test-details

The command field specifies the program name executed for this test together with all its arguments. Let me explain what they mean in this example.

  • Some binaries receives arguments that specify the data type to test (like at this case GL_RGB16_SNORM) or any other data: number of samples, msaa, etc.
  • -fbo: it draws in an off-screen framebuffer.
  • -auto: it automatically run the test and when it finishes, it closes the window and prints if the test failed or passed.

Occasionally, you might run the test program without fbo and auto parameters because you want to see what it draws on the window to understand better the bug you are debugging.

Create your own test profile

Besides explicitly adding/removing tests from tests/all.py (or other profiles) in the piglit run command, there is other way of running a specific subset of tests: profiles.

A test profile in piglit is a script written in Python that selects the tests to execute.

There are several profiles already defined in piglit, two of them we already saw them in this post series: tests/sanity.tests and tests/all. First one is useful to check if piglit was correctly compiled together with its dependencies, while the second runs all piglit tests in one shot. Of course there are more profiles inside tests/ directory: cl.py (OpenCL tests), es3conform.py (OpenGL ES 3.0 conformance tests), gpu.py, etc.

Eventually you will write your own profile because adding/removing tests in the console command is tiresome and prone to errors.

This is an example of a profile based on tests/gpu.py that I was recently using for testing the gallium llvmpipe driver.

As you see, it picks the test lists from quick.py but with some changes: it drops all the tests related to a couple of OpenGL extensions (ARB_vertex_program and ARB_fragment_program) and it drops other seven tests because they took too much time when I was testing them with gallium llvmpipe driver on my laptop.

I recommend you to spend some time playing with profiles as it is a very powerful tool when tailoring a piglit run.

What else?

In some situations, you want to know which test produces kernel error messages in dmesg, or which test is being ran now. For both cases, piglit provides parameters for run command:

  • Verbose (-v) prints a line of output for each test before and after it runs, so you can find which takes longer or outputs errors without need to wait until piglit finishes.
  • Dmesg (–dmesg): it saves the difference of dmesg outputs before and after each test program is executed. Thanks to that, you can easily find which test produces kernel errors in the graphics device driver.

Wrapping up

After giving an introduction to piglit in my blog,  this post explains how to configure a piglit run, change the list of tests to execute and how to run a standalone test. As you see, piglit is very powerful tool that requires some time to learn how to use it appropriately.

In next post I will talk about a more advanced topic: how to create your own tests.

CC BY-SA 4.0 This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

One thought on “piglit (II): How to launch a tailored piglit run

Comments are closed.