This document walks through how to get started adding fuzz tests to Chromium.
It guides you how to use our latest fuzzing technology, called FuzzTest. This replaces earlier technology called libfuzzer. See the section at the end for reasons why you might sometimes still want to use libfuzzer.
You should fuzz any function which takes input from any untrusted source, such as the internet. If the code parses, decodes, or otherwise manipulates that input, it definitely should be fuzzed!
enable_fuzztest = true
. See below for details. Create a .cc
file.#include "third_party/fuzztest/src/fuzztest/fuzztest.h"
FUZZ_TEST
macro, which might be as simple as FUZZ_TEST(MyApiTest, ExistingFunctionWhichTakesUntrustedInput)
(though you may wish to structure things differently, see below)That's it!
This fuzzer will be built automatically, using various [sanitizers], and run on our distributed fuzzing infrastructure ClusterFuzz. If it finds bugs, they'll be reported back to you.
More detail in all the following sections.
FUZZ_TEST
targetif (fuzztest_supported) { }
blocks in your gn
file for now. We’ll remove these in future when it works on all platforms.import("//build/config/sanitizers/sanitizers.gni") import("//testing/test.gni") if (fuzztest_supported) { test("hypothetical_fuzztests") { sources = [ "hypothetical_fuzztests.cc" ] enable_fuzztest = true deps = [ ":hypothetical_component", "//third_party/fuzztest:fuzztest_gtest_main", ] } }
FUZZ_TEST
support to a targetIn the near future we'll support adding FUZZ_TEST
s alongside existing unit tests, even in the same .cc file. You will add an extra enable_fuzztest = true
line:
if (is_linux) { test("existing_unit_tests") { sources = [ "existing_unit_tests.cc" ] # add FUZZ_TESTs here enable_fuzztest = true # add this! deps = [ ":existing_component", # Other stuff ] } }
This will:
This relies on something, somewhere, calling base::LaunchUnitTests
within your executable to initialize FuzzTest. This should be the case already.
(If you have other code targets, such as source_set
s, contributing to your unit test target they may need to explicitly depend upon //third_party/fuzztest
too.)
FUZZ_TEST
s in the codeFirst, #include "third_party/fuzztest/src/fuzztest/fuzztest.h"
.
Then, it‘s normal to create a function named after the thing you’re trying to prove, with assertions to prove it.
For instance,
void MyApiCanSuccessfullyParseAnyString(std::string input) { bool success = MyApi(input); EXPECT_TRUE(success); }
Then, declare the FUZZ_TEST
macro:
FUZZ_TEST(MyApiTest, MyApiCanSuccessfullyParseAnyString);
Our fuzzing infrastructure will generate all possible strings and prove it works. Obviously, that takes infinite time, so instead our fuzzing infrastructure will carefully craft strings to explore more and more branches within MyApi
, mutating the input according to code coverage, so there's a good chance bugs will be found quickly.
Fuzzing should always be alongside traditional unit testing - never rely on it to find all the bugs! It should be a backstop to prevent unexpected security flaws sneaking past your regular testing.
In more complex cases, you'll need to tell FuzzTest about the expected domains of valid input. For example:
void MyApiAlwaysSucceedsOnPositiveIntegers(int i) { bool success = MyApi(i); EXPECT_TRUE(success); } FUZZ_TEST(MyApiTest, MyApiAlwaysSucceedsOnPositiveIntegers) .WithDomains(/*i:*/fuzztest::Positive<int>());
See the FuzzTest reference for all your options here.
Simply build and run your unit tests as normal. FUZZ_TEST
s are supported only on some platforms. If you‘re on such a platform, you’ll see your fuzz test run for one second:
[==========] Running 1 test from 1 test suite. [----------] Global test environment set-up. [----------] 1 test from ScaleFuzz [ RUN ] ApiTest.MyApiCanSuccessfullyParseAnyString [ OK ] ApiTest.MyApiCanSuccessfullyParseAnyString (1000 ms) [----------] 1 test from ScaleFuzz (1000 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test suite ran. (1000 ms total) [ PASSED ] 1 test.
On other platforms, the test will be ignored.
If you want to try actually fuzzing with FuzzTest, add the gn argument enable_fuzztest_fuzz = true
. You can then run your unit test with the extra command line argument --fuzz=
, optionally specifying a test name. You'll see lots of output as it explores your code:
[*] Corpus size: 1 | Edges covered: 73 | Fuzzing time: 1.60482ms | Total runs: 1.00e+00 | Runs/secs: 623 | Max stack usage: 0 [*] Corpus size: 2 | Edges covered: 103 | Fuzzing time: 1.844ms | Total runs: 2.00e+00 | Runs/secs: 1084 | Max stack usage: 0 [*] Corpus size: 3 | Edges covered: 111 | Fuzzing time: 2.747931ms | Total runs: 3.00e+00 | Runs/secs: 1091 | Max stack usage: 0 [*] Corpus size: 4 | Edges covered: 135 | Fuzzing time: 2.92305ms | Total runs: 4.00e+00 | Runs/secs: 1368 | Max stack usage: 0 [*] Corpus size: 5 | Edges covered: 173 | Fuzzing time: 3.35237ms | Total runs: 5.00e+00 | Runs/secs: 1491 | Max stack usage: 0 [*] Corpus size: 6 | Edges covered: 178 | Fuzzing time: 4.15666ms | Total runs: 6.00e+00 | Runs/secs: 1443 | Max stack usage: 0
(“Edges covered”) is how many different code blocks have been explored (that is, sections between branches). Over time, you'll see it explore more and more until it runs out of new edges to explore.
Nothing special is required here!
After a day or two, we should see ClusterFuzz starting to run your new fuzzer, and it should be visible on ClusterFuzz Fuzzer Stats. Look for fuzzers starting with centipede_
and your test target's name.
Thanks very much for doing your part in making Chromium more secure!
There are some situations where FuzzTests may not work. For example:
In these cases, you may be best off creating a standalone fuzzer using our older fuzzing technology, libfuzzer. There are further options beyond that, e.g. uploading “black box” fuzzers to ClusterFuzz, or even running fuzzers outside of ClusterFuzz which then upload results to ClusterFuzz for triage and diagnosis. To explore any of those options, please discuss with the fuzzing team (email security@chromium.org if you're outside Google).