In general, we expect every PR that fixes a bug in rustc to come accompanied by a regression test of some kind. This test should fail in master but pass after the PR. These tests are really useful for preventing us from repeating the mistakes of the past.
The first thing to decide is which kind of test to add. This will depend on the nature of the change and what you want to exercise. Here are some rough guidelines:
library/${crate}/tests
(where ${crate}
is usually core
, alloc
, or std
).rustdoc
or rustdoc-ui
test. Occasionally you'll need rustdoc-js
as well.debuginfo
test suite.codegen
or mir-opt
test suites.run-make
.The following is a basic guide for creating a UI test, which is one of the most common compiler tests. For this tutorial, we'll be adding a test for an async error message.
The first step is to create a Rust source file somewhere in the tests/ui
tree. When creating a test, do your best to find a good location and name (see Test organization for more). Since naming is the hardest part of development, everything should be downhill from here!
Let's place our async test at tests/ui/async-await/await-without-async.rs
:
// Check what happens when using await in a non-async fn. // edition:2018 async fn foo() {} fn bar() { foo().await } fn main() {}
A few things to notice about our test:
// edition:2018
comment is called a header which provides instructions to compiletest on how to build the test. Here we need to set the edition for async
to work (the default is 2015).fn main
function. This is because the default for UI tests is a bin
crate-type, and we don't want the “main not found” error in our test. Alternatively, you could add #![crate_type="lib"]
.The next step is to create the expected output from the compiler. This can be done with the --bless
option:
./x test tests/ui/async-await/await-without-async.rs --bless
This will build the compiler (if it hasn't already been built), compile the test, and place the output of the compiler in a file called tests/ui/async-await/await-without-async.stderr
.
However, this step will fail! You should see an error message, something like this:
error: /rust/tests/ui/async-await/await-without-async.rs:7: unexpected error: ‘7:10: 7:16:
await
is only allowed insideasync
functions and blocks E0728’
Every error needs to be annotated with a comment in the source with the text of the error. In this case, we can add the following comment to our test file:
fn bar() { foo().await //~^ ERROR `await` is only allowed inside `async` functions and blocks }
The //~^
squiggle caret comment tells compiletest that the error belongs to the previous line (more on this in the Error annotations section).
Save that, and run the test again:
./x test tests/ui/async-await/await-without-async.rs
It should now pass, yay!
Somewhat hand-in-hand with the previous step, you should inspect the .stderr
file that was created to see if it looks like how you expect. If you are adding a new diagnostic message, now would be a good time to also consider how readable the message looks overall, particularly for people new to Rust.
Our example tests/ui/async-await/await-without-async.stderr
file should look like this:
error[E0728]: `await` is only allowed inside `async` functions and blocks --> $DIR/await-without-async.rs:7:10 | LL | fn bar() { | --- this is not `async` LL | foo().await | ^^^^^^ only allowed inside `async` functions and blocks error: aborting due to previous error For more information about this error, try `rustc --explain E0728`.
You may notice some things look a little different than the regular compiler output. The $DIR
removes the path information which will differ between systems. The LL
values replace the line numbers. That helps avoid small changes in the source from triggering large diffs. See the Normalization section for more.
Around this stage, you may need to iterate over the last few steps a few times to tweak your test, re-bless the test, and re-review the output.
Sometimes when adding or changing a diagnostic message, this will affect other tests in the test suite. The final step before posting a PR is to check if you have affected anything else. Running the UI suite is usually a good start:
./x test tests/ui
If other tests start failing, you may need to investigate what has changed and if the new output makes sense. You may also need to re-bless the output with the --bless
flag.
The first comment of a test file should summarize the point of the test, and highlight what is important about it. If there is an issue number associated with the test, include the issue number.
This comment doesn't have to be super extensive. Just something like “Regression test for #18060: match arms were matching in the wrong order.” might already be enough.
These comments are very useful to others later on when your test breaks, since they often can highlight what the problem is. They are also useful if for some reason the tests need to be refactored, since they let others know which parts of the test were important (often a test must be rewritten because it no longer tests what is was meant to test, and then it's useful to know what it was meant to test exactly).