UI tests are a particular test suite of compiletest.
The tests in tests/ui
are a collection of general-purpose tests which primarily focus on validating the console output of the compiler, but can be used for many other purposes. For example, tests can also be configured to run the resulting program to verify its behavior.
A test consists of a Rust source file located anywhere in the tests/ui
directory. For example, tests/ui/hello.rs
is a basic hello-world test.
Compiletest will use rustc
to compile the test, and compare the output against the expected output which is stored in a .stdout
or .stderr
file located next to the test. See Output comparison for more.
Additionally, errors and warnings should be annotated with comments within the source file. See Error annotations for more.
Headers in the form of comments at the top of the file control how the test is compiled and what the expected behavior is.
Tests are expected to fail to compile, since most tests are testing compiler errors. You can change that behavior with a header, see Controlling pass/fail expectations.
By default, a test is built as an executable binary. If you need a different crate type, you can use the #![crate_type]
attribute to set it as needed.
UI tests store the expected output from the compiler in .stderr
and .stdout
files next to the test. You normally generate these files with the --bless
CLI option, and then inspect them manually to verify they contain what you expect.
The output is normalized to ignore unwanted differences, see the Normalization section. If the file is missing, then compiletest expects the corresponding output to be empty.
There can be multiple stdout/stderr files. The general form is:
test-name.
revision.
compare_mode.
extension
stderr
— compiler stderrstdout
— compiler stdoutrun.stderr
— stderr when running the testrun.stdout
— stdout when running the test64bit.stderr
— compiler stderr with stderr-per-bitwidth
header on a 64-bit target32bit.stderr
— compiler stderr with stderr-per-bitwidth
header on a 32-bit targetA simple example would be foo.stderr
next to a foo.rs
test. A more complex example would be foo.my-revision.polonius.stderr
.
There are several headers which will change how compiletest will check for output files:
stderr-per-bitwidth
— checks separate output files based on the target pointer width. Consider using the normalize-stderr
header instead (see Normalization).dont-check-compiler-stderr
— Ignores stderr from the compiler.dont-check-compiler-stdout
— Ignores stdout from the compiler.compare-output-lines-by-subset
— Checks that the output contains the contents of the stored output files by lines opposed to checking for strict equality.UI tests run with -Zdeduplicate-diagnostics=no
flag which disables rustc's built-in diagnostic deduplication mechanism. This means you may see some duplicate messages in the output. This helps illuminate situations where duplicate diagnostics are being generated.
The compiler output is normalized to eliminate output difference between platforms, mainly about filenames.
Compiletest makes the following replacements on the compiler output:
$DIR
. Example: /path/to/rust/tests/ui/error-codes
$SRC_DIR
. Example: /path/to/rust/library
$SRC_DIR
are replaced with LL:COL
. This helps ensure that changes to the layout of the standard library do not cause widespread changes to the .stderr
files. Example: $SRC_DIR/alloc/src/sync.rs:53:46
$TEST_BUILD_DIR
. This only comes up in a few rare circumstances. Example: /path/to/rust/build/x86_64-unknown-linux-gnu/test/ui
\t
.\
) are converted to forward slashes (/
) within paths (using a heuristic). This helps normalize differences with Windows-style paths.//~ ERROR some message
are removed.[HASH]
or <SYMBOL_HASH>
.Additionally, the compiler is run with the -Z ui-testing
flag which causes the compiler itself to apply some changes to the diagnostic output to make it more suitable for UI testing. For example, it will anonymize line numbers in the output (line numbers prefixing each source line are replaced with LL
). In extremely rare situations, this mode can be disabled with the header command // compile-flags: -Z ui-testing=no
.
Note: The line and column numbers for -->
lines pointing to the test are not normalized, and left as-is. This ensures that the compiler continues to point to the correct location, and keeps the stderr files readable. Ideally all line/column information would be retained, but small changes to the source causes large diffs, and more frequent merge conflicts and test errors.
Sometimes these built-in normalizations are not enough. In such cases, you may provide custom normalization rules using the header commands, e.g.
// normalize-stdout-test: "foo" -> "bar" // normalize-stderr-32bit: "fn\(\) \(32 bits\)" -> "fn\(\) \($$PTR bits\)" // normalize-stderr-64bit: "fn\(\) \(64 bits\)" -> "fn\(\) \($$PTR bits\)"
This tells the test, on 32-bit platforms, whenever the compiler writes fn() (32 bits)
to stderr, it should be normalized to read fn() ($PTR bits)
instead. Similar for 64-bit. The replacement is performed by regexes using default regex flavor provided by regex
crate.
The corresponding reference file will use the normalized output to test both 32-bit and 64-bit platforms:
... | = note: source type: fn() ($PTR bits) = note: target type: u16 (16 bits) ...
Please see ui/transmute/main.rs
and main.stderr
for a concrete usage example.
Besides normalize-stderr-32bit
and -64bit
, one may use any target information or stage supported by ignore-X
here as well (e.g. normalize-stderr-windows
or simply normalize-stderr-test
for unconditional replacement).
Error annotations specify the errors that the compiler is expected to emit. They are “attached” to the line in source where the error is located.
fn main() { boom //~ ERROR cannot find value `boom` in this scope [E0425] }
Although UI tests have a .stderr
file which contains the entire compiler output, UI tests require that errors are also annotated within the source. This redundancy helps avoid mistakes since the .stderr
files are usually auto-generated. It also helps to directly see where the error spans are expected to point to by looking at one file instead of having to compare the .stderr
file with the source. Finally, they ensure that no additional unexpected errors are generated.
They have several forms, but generally are a comment with the diagnostic level (such as ERROR
) and a substring of the expected error output. You don't have to write out the entire message, just make sure to include the important part of the message to make it self-documenting.
The error annotation needs to match with the line of the diagnostic. There are several ways to match the message with the line (see the examples below):
~
: Associates the error level and message with the current line~^
: Associates the error level and message with the previous error annotation line. Each caret (^
) that you add adds a line to this, so ~^^^
is three lines above the error annotation line.~|
: Associates the error level and message with the same line as the previous comment. This is more convenient than using multiple carets when there are multiple messages associated with the same line.The space character between //~
(or other variants) and the subsequent text is negligible (i.e. there is no semantic difference between //~ ERROR
and //~ERROR
although the former is more common in the codebase).
Here are examples of error annotations on different lines of UI test source.
Use the //~ ERROR
idiom:
fn main() { let x = (1, 2, 3); match x { (_a, _x @ ..) => {} //~ ERROR `_x @` is not allowed in a tuple _ => {} } }
Use the //~^
idiom with number of carets in the string to indicate the number of lines above. In the example below, the error line is four lines above the error annotation line so four carets are included in the annotation.
fn main() { let x = (1, 2, 3); match x { (_a, _x @ ..) => {} // <- the error is on this line _ => {} } } //~^^^^ ERROR `_x @` is not allowed in a tuple
Use the //~|
idiom to define the same error line as the error annotation line above:
struct Binder(i32, i32, i32); fn main() { let x = Binder(1, 2, 3); match x { Binder(_a, _x @ ..) => {} // <- the error is on this line _ => {} } } //~^^^^ ERROR `_x @` is not allowed in a tuple struct //~| ERROR this pattern has 1 field, but the corresponding tuple struct has 3 fields [E0023]
error-pattern
The error-pattern
header can be used for messages that don't have a specific span.
Let's think about this test:
fn main() { let a: *const [_] = &[1, 2, 3]; unsafe { let _b = (*a)[3]; } }
We want to ensure this shows “index out of bounds” but we cannot use the ERROR
annotation since the error doesn‘t have any span. Then it’s time to use the error-pattern
header:
// error-pattern: index out of bounds fn main() { let a: *const [_] = &[1, 2, 3]; unsafe { let _b = (*a)[3]; } }
But for strict testing, try to use the ERROR
annotation as much as possible.
The error levels that you can have are:
ERROR
WARN
or WARNING
NOTE
HELP
and SUGGESTION
You are allowed to not include a level, but you should include it at least for the primary message.
The SUGGESTION
level is used for specifying what the expected replacement text should be for a diagnostic suggestion.
UI tests use the -A unused
flag by default to ignore all unused warnings, as unused warnings are usually not the focus of a test. However, simple code samples often have unused warnings. If the test is specifically testing an unused warning, just add the appropriate #![warn(unused)]
attribute as needed.
When using revisions, different messages can be conditionally checked based on the current revision. This is done by placing the revision cfg name in brackets like this:
// edition:2018 // revisions: mir thir // [thir]compile-flags: -Z thir-unsafeck async unsafe fn f() {} async fn g() { f(); //~ ERROR call to unsafe function is unsafe } fn main() { f(); //[mir]~ ERROR call to unsafe function is unsafe }
In this example, the second error message is only emitted in the mir
revision. The thir
revision only emits the first error.
If the cfg causes the compiler to emit different output, then a test can have multiple .stderr
files for the different outputs. In the example above, there would be a .mir.stderr
and .thir.stderr
file with the different outputs of the different revisions.
By default, a UI test is expected to generate a compile error because most of the tests are checking for invalid input and error diagnostics. However, you can also make UI tests where compilation is expected to succeed, and you can even run the resulting program. Just add one of the following header commands:
// check-pass
— compilation should succeed but skip codegen (which is expensive and isn't supposed to fail in most cases).// build-pass
— compilation and linking should succeed but do not run the resulting binary.// run-pass
— compilation should succeed and running the resulting binary should also succeed.// check-fail
— compilation should fail (the codegen phase is skipped). This is the default for UI tests.// build-fail
— compilation should fail during the codegen phase. This will run rustc
twice, once to verify that it compiles successfully without the codegen phase, then a second time the full compile should fail.// run-fail
— compilation should succeed, but running the resulting binary should fail.For run-pass
and run-fail
tests, by default the output of the program itself is not checked. If you want to check the output of running the program, include the check-run-results
header. This will check for a .run.stderr
and .run.stdout
files to compare against the actual output of the program.
Tests with the *-pass
headers can be overridden with the --pass
command-line option:
./x test tests/ui --pass check
The --pass
option only affects UI tests. Using --pass check
can run the UI test suite much faster (roughly twice as fast on my system), though obviously not exercising as much.
The ignore-pass
header can be used to ignore the --pass
CLI flag if the test won't work properly with that override.
The known-bug
header may be used for tests that demonstrate a known bug that has not yet been fixed. Adding tests for known bugs is helpful for several reasons, including:
Do not include error annotations in a test with known-bug
. The test should still include other normal headers and stdout/stderr files.
When deciding where to place a test file, please try to find a subdirectory that best matches what you are trying to exercise. Do your best to keep things organized. Admittedly it can be difficult as some tests can overlap different categories, and the existing layout may not fit well.
For regression tests – basically, some random snippet of code that came in from the internet – we often name the test after the issue plus a short description. Ideally, the test should be added to a directory that helps identify what piece of code is being tested here (e.g., tests/ui/borrowck/issue-54597-reject-move-out-of-borrow-via-pat.rs
)
When writing a new feature, create a subdirectory to store your tests. For example, if you are implementing RFC 1234 (“Widgets”), then it might make sense to put the tests in a directory like tests/ui/rfc1234-widgets/
.
In other cases, there may already be a suitable directory. (The proper directory structure to use is actually an area of active debate.)
Over time, the tests/ui
directory has grown very fast. There is a check in tidy that will ensure none of the subdirectories has more than 1000 entries. Having too many files causes problems because it isn‘t editor/IDE friendly and the GitHub UI won’t show more than 1000 entries. However, since tests/ui
(UI test root directory) and tests/ui/issues
directories have more than 1000 entries, we set a different limit for those directories. So, please avoid putting a new test there and try to find a more relevant place.
For example, if your test is related to closures, you should put it in tests/ui/closures
. If you‘re not sure where is the best place, it’s still okay to add to tests/ui/issues/
. When you reach the limit, you could increase it by tweaking here.
UI tests can validate that diagnostic suggestions apply correctly and that the resulting changes compile correctly. This can be done with the run-rustfix
header:
// run-rustfix // check-pass #![crate_type = "lib"] pub struct not_camel_case {} //~^ WARN `not_camel_case` should have an upper camel case name //~| HELP convert the identifier to upper camel case //~| SUGGESTION NotCamelCase
Rustfix tests should have a file with the .fixed
extension which contains the source file after the suggestion has been applied.
When the test is run, compiletest first checks that the correct lint/warning is generated. Then, it applies the suggestion and compares against .fixed
(they must match). Finally, the fixed source is compiled, and this compilation is required to succeed.
Usually when creating a rustfix test you will generate the .fixed
file automatically with the x test --bless
option.
The run-rustfix
header will cause all suggestions to be applied, even if they are not MachineApplicable
. If this is a problem, then you can instead use the rustfix-only-machine-applicable
header. This should be used if there is a mixture of different suggestion levels, and some of the non-machine-applicable ones do not apply cleanly.
Compare modes can be used to run all tests with different flags from what they are normally compiled with. In some cases, this might result in different output from the compiler. To support this, different output files can be saved which contain the output based on the compare mode.
For example, when using the Polonius mode, a test foo.rs
will first look for expected output in foo.polonius.stderr
, falling back to the usual foo.stderr
if not found. This is useful as different modes can sometimes result in different diagnostics and behavior. This can help track which tests have differences between the modes, and to visually inspect those diagnostic differences.
If in the rare case you encounter a test that has different behavior, you can run something like the following to generate the alternate stderr file:
./x test tests/ui --compare-mode=polonius --bless
Currently none of the compare modes are checked in CI for UI tests.