Bug: 135760423

Clone this repo:
  1. 6521d19 Merge "Switch to NDK r27-beta1" into main by Treehugger Robot · 15 hours ago main master
  2. 1f078e1 Switch to NDK r27-beta1 by Stephen Hines · 2 days ago
  3. 0d3010f Merge "Switch to StrEnum to allow conversion to strings" into main by Treehugger Robot · 2 days ago
  4. 4473277 Merge "Adjust arguments to update_prebuilts.py" into main by Treehugger Robot · 2 days ago
  5. 10ceeb0 Adjust arguments to update_prebuilts.py by Chris Wailes · 2 days ago

Updating the Android Rust Toolchain

A new stable version of Rust is released approximately every 6 weeks. Our team works towards releasing a prebuilt for each release that is used by other Google developers. This document describes the process for creating those artifacts. - In Section 1 we focus on the technical steps for updating the Android Rust toolchain during the initial set-up, Beta Workflow and the stable workflow. - In Section 2 we provide more details on the types of things that can go wrong and provide some guidance on how to get past those. - In Section 3 some notes - In Section 4 for a description of the PGO pipeline and instructions on reproducing builds using the smaller rust-toolchain branch.

Note our team policy

Section 1: The-Steps

A new stable version of Rust is released approximately every 6 weeks. In order to resolve issues ahead of time we perform the release process on the Beta version of the toolchain. Testing against the Beta version of the toolchain can often hint at possible incompatibilities between the Rust toolchain and the Android code base. However, sometimes there will be language features available in a Beta that have not been released in the Stable branch. Other times issues pop up in the Stable version of the language that weren't in Beta, keeping us toolchain developers on our toes and mindful of what/when we commit changes to the code base.

Testing and deploying the Rust toolchain involves several steps: fetching the toolchain from upstream, building it locally, modifying the Android source code to be compatible with the latest release, testing the build, and uploading compiler prebuilts to Gerrit. Since the Beta source is only used for testing and preparation, only a subset of the steps are performed and the commands used differ slightly from those used during a Stable release. We enumerate the steps below and show the distinct workflows. The Beta Workflow is when the toolchain developer is targeting the Beta version of the toolchain for the purposes of diagnosing possible issues. The Stable Workflow is when the toolchain developer is targeting the Stable version of the Rust toolchain for the purpose of providing prebuilts for other developers.

Section 1.0 Prebuilds Prep

When beginning the update process there should only be a single prebuilt version checked in. If this is not the case then it is likely that a downstream team is stuck on a previous version. Check to see if the client teams have completed their migration and, if so, begin deleting the unused prebuilts.

Post on Buganizer ticket that the release process is beginning, asks for patches, and declares the prebuilt that will be deleted.

Check what Rust version each downstream team depends on and update the doc. If a team is depending on the prebuilt you are deleting then make sure they know the prebuilt will be deleted. You can do this by CC'ing someone from their team on the update buganizer ticket, emailing them, or opening a new ticket for them to update their toolchain.

Section 1.1: Setting up

The Rust toolchain has two branch manifests in AOSP, rust-toolchain and master-plus-rust. We will be using the master-plus-rust branch in this documentation for two reasons: 1) testing the prebuilts to ensure they can build Android and 2) generating profiles for Profile Guided Optimization.

Just once

The following steps need to be done once when setting up. The following commands will create a repo directory (e.g. ~/master-plus-rust), initialize it with the master-plus-rust manifest, synchronize the repository, and tell Git how to find the list of commit revisions to skip when computing blame.

$ mkdir master-plus-rust
$ cd master-plus-rust
$ repo init -u https://android.googlesource.com/platform/manifest -b master-plus-rust
$ repo sync -j16
$ git config blame.ignoreRevsFile .git-blame-ignore-revs

To allow editors like VSCode and other tools to find the project's Python code you can add the line below to the toolchains/android_rust/.env file:

PYTHONPATH=$PYTHONPATH:src/

The Android Rust toolchain uses the MyPy and YAPF tools to handle linting and autoformatting respectively. You can install these by running

sudo apt install mypy yapf

Finally, there are multiple ways to handle pre-commit hooks for the Android Rust toolchain directory. One option is to place the following script at toolchains/android_rust/.git/hooks/precommit:

#!/usr/bin/sh

yapf -i tools/*.py src/android_rust/*.py src/binary_crates/*.py
mypy .

Each time

Each time you start to work on updating the toolchain you will want to sync in the toolchain and aosp directory.

$ cd $TOOLCHAIN
$ repo sync -j16

It can also be helpful to initialize the Android build environment:

$ . build/envsetup.sh
$ lunch <build target; e.g. aosp_cf_x86_64_phone-staging-userdebug>

Commands below will assume you have performed this initialization. You may select a new lunch target at any point.

Section-1.2: Testing Workflow

It is best to start the process with a freshly-synchronized repository.

Step Test-1: Fetch latest upstream toolchain

The following includes the arguments we currently use when creating prebuilts shell $ ./toolchain/android_rust/tools/fetch_source.py <rust_version>

Below are some of the options that can be passed to the build script:

  • --beta - Used when testing the next update. This will grab the beta version of a Rust release.
  • --overwrite - Manually overwrite existing branches

Details are listed in the --help output. Note that you can check when the branches are updated on github.

If you hit issues with prohibited patterns in the commit, you can suppress those messages by running a similar command to the following one in toolchain/rustc:

  • git config --add secrets.allowed ARTIFACTS_AWS_ACCESS_KEY_ID
  • These are usually CI-related keys that are not actually secret for the upstream Rust toolchain sources.

Step Test-2: Build locally

The following includes the arguments we currently use when creating prebuilts

shell $ ./toolchain/android_rust/tools/build.py --lto thin

Below are some of the options that can be passed to the build script:

  • --build-name - Specify the name of the resulting build archive. E.g. passing test-build will result in the archive name rust-test-build.tar.gz
  • --lto {none,thin,full} - If LTO should be enabled and which mode it should operate under
  • --gc-sections - If functions/data should be split into different sections and garbage collected during linking
  • --llvm-linkage {static,shared} - If the LLVM library should be linked statically or dynamically
  • --upstream-test- Run upstream tests as part of the build process
  • PGO flags are described below in the PGO section.
  • --repatch- Don't copy the whole source. Just copy and repatch the files affected by the patches.

There are various reasons why the build might break. Check out Section 2.1 for helpful instructions and tips to build Rust.

If the build seems to be going, this will take a while; switch to another task, get some coffee, etc.

By default, this step will create rust-dev-tar.gz.

Tips: - Use the same branch name as you did for rustc if you want repo tooling to help you later. E.g. rust-update-source-1.59.0

Note, in order to include a particular CL in your update. Go to the ‘download’ tab in the CL, copy the cherry pick script, and paste that in the relevant git repo.

Step Test-3: Test

The test_compiler.py script can be used to verify a compiler prebuilt.

$ ./toolchain/android_rust/tools/test_compiler.py --prebuilt-path dist/rust-dev.tar.xz \
--target aosp_cf_x86_64_phone --all-rust --test_targets --reuse-prebuilt

Below are some of the options that can be passed to the build script:

  • Use --target aosp_cf_riscv64_phone to locally run riscv tests
  • The --image flag will cause a full device image to be built and will result in significantly longer test times. It is only necessary to pass this flag if you intend to do a boot test.
  • Use --no-reuse-prebuilt to replace the existing prebuilt with a new one, if you have built a new toolchain and want to retest with it.

There are various things that can break when testing a new prebuilt. Take a look at Section 2.2 for instructions on how to fix the build.

Step Test-4: Boot

If --image was passed to the test_compiler.py script during the previous step you can now use the generated image to boot a device. To boot a cuttlefish virtual device you would use the following command:

acloud create --local-image

Once it is complete, this command will print the IP address and port that can be used to connect to the virtual device. Export this is as ANDROID_SERIAL or use it as a command line argument to Android development tools.

Use the --local-instance flag to run the virtual device on the machine running the command. This is not recommended for Google developers (see).

Section 1.3: Release Workflow

Step Release-5: Fetch Source and Build Rust

Repeat steps Test-1 and Test-2 to fetch the Rust source you wish to release. This will usually not be a beta release so ensure that the -b flag isn‘t passed to the fetch_source.py script unless that’s really what you intended. You might also need to use the --overwrite flag if you've worked on this release before, say during the Testing Workflow

Step Release-6: Testing Build

Follow steps Test-3, and Test-4 in the Testing workflow. Upload any changes required by the new compiler version to Gerrit for review. Changes that are valid in both the old and new version of Rust should go in independently while any changes that require the new compiler will need to go into the topic used to update the prebuilts.

These changes need to go in before the compiler source is updated to ensure that our CI builds remain green.

Note here is an example of a ticket that posted all related CLs in the Description bug

Step Release-7: Upload

Place any changes to toolchain/android_rust and toolchain/rustc into a topic, and use repo to upload as you usually would:

repo upload -t -o l=Presubmit-Ready+1
# You may need to use `-o nokeycheck` too for cargo prebuilts.

This may take a while because updates to rustc can be hefty in size. Double check the response from the server to make sure the change went through.

You‘ll need to get these changes +2’d and merged before you can proceed.

Make sure the same topic has been added to both CLs with an updated Rust version number, e.g. rust-update-source-1.59.0.

Note the Commit message should include the Test strategy and buganizer ID, such as

Test: ./tools/build.py --lto thin
Bug: 232437287

Step Release-8: Wait for builds

The PGO pipeline needs to complete a green build including your changes. After the PGO pipeline has completed, the next new rust toolchain build needs to be a complete green build.

It will often be necessary to submit an empty CL to trigger a build on the rust-toolchain branch.

$ git commit --allow-empty

Step Release-9: Update prebuilts

Use the following command to fetch, unpack, and commit the prebuilts and updated references to the prebuilts:

$ ./toolchain/android_rust/tools/update_prebuilts.py -i <issue_number> <rust_version>

By default the script will fetch artifacts from the last known good build. If you wish to specify a specific build you can use the --bid <bid> flag. Use this flag when there is a broken build and you want to override the LKGB detection logic.

The script will fetch artifacts from the build at the end of the PGO pipeline. To disable that use the -no-pgo flag.

Step Release-10: Upload to Gerrit

Upload the new commits in prebuilts/rust, build/soong, toolchain/android_rust, and any of the manually updated locations to Gerrit. Do this by going into both directories and using:

$ repo upload -t --cbr .

Note that as a last argument cbr will upload the current branch and . will upload this directory. Note that the terminal might flag an issue at this step with f.. instructions, but you can just follow the instructions from the command line and retry this step.

This process should create three CLs. Open up the Gerrit links and mark one of them as presubmit-ready. Add the same topic to both CLs with an updated Rust version number, e.g. rust-update-prebuilts-1.59.0.

Step Release-11: Remove previous prebuilts

Our goal is to keep only a single prebuilt checked out under steady-state conditions. Once a new release has been stabilized we can delete any previous versions of the toolchain that are not actively being used by a downstream team. In this case “stabilized” means that each client team that is actively working on updating to the newest version has completed their work and is happy with the state of the new prebuilt.

A release can be removed using the following commands:

$ cd prebuilts/rust
$ repo start rust-remove-$OLD_RUST_VERSION
$ git rm -rf {linux-x86,linux-musl-x86,darwin-x86}/$OLD_RUST_VERSION
$ git commit
$ repo upload .

Commit messages are usually formatted as follows:

Removing Rust $OLD_RUST_VERSION prebuilts

Bug: <bug ID>
Test: m rust

Prebuilts before version 1.61.0.p1 will not include linux-musl-x86 binaries so it will be necessary to remove that target from the above command until 1.60.0 is removed.

If a team is depending on the prebuilt you plan to delete you can pin their code.

Step Release-12: Wrapping it up

Create a ticket for the next release and assign it to the next team member. Check that the CC’s and hotlist (android_rust_toolchain) on it are consistent with prior tickets.

Announce on g/android-rust-announce when the release is finished

Schedule a retrospective meeting with the team, and update notes on the release in the retrospective doc.

Document in this file any changes to the toolchain process (including argument changes) and debugging tips for issues you hit.

Section 2: How to Fix Things

While updating the Rust toolchain there are various issues that can arise. Things can break, roadblocks can get in the way, and others might need to be brought in. In this section we describe the different types of issues that can occur, examples, and instruct on how to move past them.

They are organized by topics: - Section 2.1: Itemizes various breaks that can occur when building Rust during Step Test-2 Build locally. - Section 2.2: Describes roadblocks hit when testing during Step Test-3:Test, which is a part of Step Release-8: Testing Build - Section 2.3: Proves some notes on updating the prebuilts during Step Release-9:Update Prebuilts - Section 2.4: Describes issues with uploading to Gerrit as done in both Step Release 6:Upload and Step Release 10:Upload to Gerrit

Section 2.1 : The Rust Build

Things that can break while building the Rust toolchain:

  • Patch Application
  • Directory Structure Change
  • Binary Incompatibility
  • Compilation Failure
  • New Crate
  • New flags
  • Missing tools

Patch Application

The Android project carries several patches for the Rust toolchain source. Patches make changes to a code file. In order for a patch file to be successfully applied the code that it is targeting needs to match the code file closely enough for the algorithm in the patch program to identify the relevant code. If a patch file fails to be applied then it is likely due to a change in the targeted code base. In which case, the next step is to figure out if the patch is still necessary to be applied or if that Patch file can be removed.

To know which patch file failed take a look at the terminal error message. The error message will also say what hunk # and name of the Rust file. Open up the patch file in android_rust/patches and the Rust file. At this point it'll either look like (1) patch was already applied, (2) the codebase was otherwise modified.

In the case of (1), it would be useful to verify that the patch was applied. This can be done by look at the log history for that Rust file.If you do believe that the patch was merged upstream then you just need to remove the patch from the patches/ directory, e.g.

pushd toolchain/android_rust
repo start update-rustc-$RUST_VERSION
git rm patches/rustc-000n-Already-merged.patch
git commit -m "Remove Foo patch that has landed upstream"
popd

In the case of (2), the codebase was changed in some way. Sometimes the difference might be very simple. For instance, one time the difference was just a variable name change and looking at the log history confirmed that.

To be able to successfully apply the patch the code in the patch must match the code base exactly so go ahead and modify the patch code to match the code base. If adding or removing lines of code be sure to update the number of lines that is noted in the patch file. If you are unsure if the change upholds the intent of the patch go ahead and email the patch owner, but note they will also be added as a reviewer. When generating patch files using git format-patch be sure to include extra context (-U10) as it is needed to successfully apply some patches.

After editing the patch file, upload the change to Gerrit, and ask the patch owner to review the changes to make sure the intent of the patch is still Upheld. Use the topic with “source”.

Here is an example of updating a patch file to correspond to changes in code CL.

It may also be useful to create a new patch. The following:

git format-patch -U10 HEAD~

will generate a patch file for just the previous commit.

Directory Structure Change Sometimes another library needs to be imported. We can do this by adding to the STDLIB_SOURCES definition in the tools/build.py script.

For example we got the following:

error: couldn't read prebuilts/rust/linux-x86/1.59.0/src/stdlibs/library/core/src/../../portable-simd/crates/core_simd/src/mod.rs: No such file or direct
ory (os error 2)
   --> prebuilts/rust/linux-x86/1.59.0/src/stdlibs/library/core/src/lib.rs:415:1
    |
415 | mod core_simd;
    | ^^^^^^^^^^^^^^
error: aborting due to previous error

22:37:34 ninja failed with: exit status 1

#### failed to build some targets (18 seconds) ####

and as a result we added the portable-simd library as seen in this CL.

Binary Incompatibility

TODO: text here to describe how the user will know this type of error occurred and how to fix it

Compilation Failure

TODO: text here to describe how the user will know this type of error occurred and how to fix it

New Crate

Sometimes a new crate is added and a modification needs to be made to prebuilts/rustc/Android.bp.

TODO: text here to describe how the user will know this type of error occurred and how to fix it

New Flags

The solution might be to disable a new flag that have been added to the build process. Example CL

Check the out file CMakeCache.txt to see any additional flags that may have been added during the build process. Try testing the new flags with the older version of the compiler.

Missing Tools

failed to execute command: "strip" "--strip-debug" "/usr/local/google/home/aosp/out/rustc/build/x86_64-unknown-linux-gnu/llvm/lib/libLLVM-17.so"
error: No such file or directory (os error 2)

This indicates that strip is not available on your local computer. We solved this by adding a symbolic link to another tool CL

Section 2.2: Test

Things that can break during testing:

  • Hermeticity breakage
  • Build system breakage
  • Android Source Warnings
  • Miscompilation
  • Deprecated Flag

Hermeticity Breakage

A Hermeticity breakage occurs when the build uses a tool or header that is different from what we expect. For instance, a crate including a build script allowing a different compiler/library to be used than expected. Some of these issues arise because of bugs in the upstream build system. Others can also be caused because the environment they are built-in have certain defaults set and carry unintended expectations.

TODO @Chriswailes: Describle how to approach this type of problem, given lzma-sys example.

Build System Breakage

There can be build breakage issues.

For instance, needing to change "-C passes='sancov'", to "-C passes='sancov-module'", such as in this CL.

Android Source Warnings

A common issue is that this step triggers new warnings on existing source files. If the compiler suggests a fix, apply it. Otherwise make the most reasonable looking change necessary to keep the compiler happy and rely on the original author to determine correctness during code review.

Here is an example of the workflow to modify file x.rs:

# navigate to the repo with file x.rs
repo start rust-1.59.0-fixes
# make changes to x.rs
git add x.rs
git commit

The commit message should include the testing strategy and buganizer ticket number. Here is an example commit message from one of these types of CLs.

Changes for the Rust 1.59.0 update

bug: 215232614
Test: TreeHugger and compiling with m rust
Change-Id: I1d25f5550f60ff1046a3a312f7bd210483bf9364

Note, that it is preferable to create one commit per big change to a repo, so it might be helpful to use amend when adding more changes to the code:

$ git commit --amend --no-edit

After committing the changes continue to upload them to Gerrit with

repo upload .

In Gerrit for the corresponding CL include the owners of the file as reviewers and do not set a topic.

Next we will need to periodically check-in on these CLs. If they pass presubmit, then we are waiting for the CLs to be approved/submitted by the file owners. This can take a few days and may require a friendly nudge.

If the files do not pass presubmit then the changes may not have been backwards compatible with Rust and we will need to compile it with the latest version of Rust. If that is the case on Gerrit include it in the topic rust-update-prebuilts-1.59.0, with an updated Rust version number.

We are not able to move to the final Step 12 (tagging) until these CLs created in this process has been submitted/accepted.

Clipper errors are typically pretty straightforward but sometimes it’s not clear what to change. Try checking at the change history or how the change has been implemented in other parts of the Rust codebase. For instance, builder.config.llvm_link_shared was changed to !builder.llvm_link_shared().

Miscompilation

A miscompilation may have occured when it successfully compiles but the devices fail to boot or pass CTS tests.

TODO: text here to describe how the user will know this type of error occurred and how to fix it

Deprecated flag

The following error:

warning: `-Z symbol-mangling-version` is deprecated; use `-C symbol-mangling-version`

led to changes where that variable was used with -Z was changed to -C in rust/config/global.go. CL .

Delete files

With this error

~/updates/aosp
: > ./toolchain/android_rust/tools/test_compiler.py --prebuilt-path dist/rust-dev.tar.xz --target aosp_cf_x86_6
4_phone --image
Test prebuilt directory already exists.  Deleting contents.
Extracting archive /usr/local/google/home/chiw/updates/aosp/dist/rust-dev.tar.xz
/usr/local/google/home/chiw/updates/aosp/prebuilts/build-tools/linux-x86/bin/xz: /usr/local/google/home/c
hiw/updates/aosp/dist/rust-dev.tar: File exists
Traceback (most recent call last):
  File "/usr/local/google/home/chiw/updates/aosp/./toolchain/android_rust/test_compiler.py", line 198, in
 <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/google/home/chiw/updates/aosp/./toolchain/android_rust/test_compiler.py", line 143, in
 main
    prepare_prebuilts(args.prebuilt_path)
  File "/usr/local/google/home/chiw/updates/aosp/./toolchain/android_rust/test_compiler.py", line 129, in
 prepare_prebuilts
    (target_and_version_path / "bin" / "rustc").touch()
  File "/usr/lib/python3.11/pathlib.py", line 1109, in touch
    fd = os.open(self, flags, mode)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/google/home/chiw/updates/aosp/prebuil
ts/rust/linux-x86/9.99.9/bin/rustc

Try

rm $DIST_DIR/rust-dev.tar

Section 2.3: Update prebuilts

Things that can break when updating the prebuilts:

  • File undefined
  • Fetch ID
  • Unknown manifest

File undefined

Try deleting the branch git branch -D rust-update-prebuilts-1.59.0-local

Fetching ID shell : > ./toolchain/android_rust/tools/update_prebuilts.py --chained 1.62.0 Unable to fetch LKGB build ID This means you forgot to run gcert.

unknown manifest Note that some tests might fail during presubmit because of code that live in other manifests. Make the apprioriate changes as usual, but in the right place.

Section 2.4: Upload

Things that can break while uploading files to Gerrit:

  • Geritt limitation
  • Detached head

Geritt Limitation

Help, Gerrit won't take my update!

First, try again. Sometimes Gerrit is just flaky and will take it on the second or third try.

If that‘s still not working, you are likely hitting a size limitation (for example, because rustc updated it’s LLVM revision, so the diff is bigger than usual). In this case, you will need to work with the build team to get them to use a “direct push” to skip Gerrit's hooks. Look at the initial import bug for an example conversation about importing oversized changes.

Detached head

Use this to get away from detached head:

git branch -u aosp/master

Section 2.5: Prebuilts Test failure

  • Run tests locally
  • Output Format

Run test locally

If presbuilts failed presubmit, one thing you can do is try to run test locally

source build/envsetup.sh
lunch aosp_cf_x86_64_phone-staging-userdebug
m
acloud create --local-image
export ANDROID_SERIAL=<value printed by previous command>
atest bytes_test_tests_test_bytes
// or whatever tests are failing in the prebuilts

Output Format

There could be an error parsing the results of running Rust tests.

A prior release Rust v.1.61 release notes include messages with ignored tests. We had to modify how the trade federation code analyzes the result of running tests (changed regex). CL The Gerrit CL triggered the same changes in Critique.

Other Helpful Debugging Tips

Toolchain scripts Sometimes it might be necessary to change the android_rust scripts themselves. Here are some examples:

  • audit.py: Add more .so files to the shared library allow list because they are required by other .so files
  • build.py: Useful to edit if you want to regenerate a lockfile, exclude a test
  • config.py: Change env rust flags

Other * build_error.log: Check the state of the builds and try to reproduce locally. For example if Soongs prints out/a on failure then try running m out/a to reproduce the issue locally. * upstream Check recent relevant changes to the Rust upstream code since the last update

Section 3: Notes

Troubleshooting a Broken Sysroot Build

Question: Should this be added to Section 2.1 Build Error - Missing Crate or is it a different type of error?

If the sysroot build is broken, check whether the error mentions a missing crate. If it does, there have likely been new components added to the sysroot. To address this, you will need to:

  1. Add the relevant components to STDLIB_SOURCES in toolchain/android_rust/tools/build.py.
  2. Respin the toolchain via the process above, but with a fresh commit message noting the reason for the respin. You may want to test this locally first as more than one dependency may have been added. For local testing,
    1. Build as before, using DIST_DIR=$TOOLCHAIN/dist ./toolchain/android_rust/tools/build.py
    2. Make a local commit with the contents of the tarball now in $DIST_DIR
    3. Go to prebuilts/rust in your Android tree and use git fetch local followed by a checkout to get it imported into your Android tree.
  3. Add the missing dependencies to prebuilts/rust/Android.bp. Except for publicly exported crates (which you're not adding right now), all modules in this file must be suffixed with .rust_sysroot to avoid confusion with user crates. Dependency edges should all be of rlib form unless depending on a publicly exported crate, in which case the dependency edge should match the type of the final product. As examples, libterm (exported) depends by dylib on libstd, but libterm.static (also exported) depends by rlib on libstd.static.libhashbrown.rust_sysroot is built only as an rlib, and is linked as an rlib everywhere it is used.

New Build Breaking Lint/Clippy Errors

TODO: Merge this text into the previous sections

New lints/clippys can cause build breakage and may require significant refactoring as the code base grows. To avoid blocking toolchain upgrades, explicitly allow the breaking lints/clippys when first upgrading the toolchain.

  1. Allow build breaking lints/clippys by adding them to the list in build/soong/rust/config/lints.go with the -A flag.
  2. If the new lint/clippy is beneficial enough to justify enable going forward, file a bug to track the refactor effort.

Section 4: PGO Pipeline

Android‘s Rust toolchain is built using profile guided optimization (PGO). This five-stage process records execution traces from compiling Android’s Rust codebase and uses this information to improve the performance of the Rust compiler and toolchain libraries.

Stage 1

In the first stage the Rust toolchain is built with the profile collection instrumentation. If the toolchain is built with shared LLVM linkage the resulting compiler and libraries will produce sets of profile data in the specified output directory: one for LLVM and one for Rust. If, on the other hand, the toolchain is built with static LLVM linkage the toolchain will only produce profile data for the Rust components. This is due to the fact that the statically linked LLVM objects will need to be optimized using the profile data for the executable or library they are being linked into.

Stage 2

The toolchain produced in Stage 1 is then used to compile all of the Rust code in the Android codebase for multiple architectures. This will produce one or more sets of profile data that is then merged together.

Stage 3

The profile data from Stage 2 is then used to re-compile the Rust toolchain using PGO. In addition, the LLVM libraries are instrumented to record context-sensitive profile-guided optimization (CS-PGO) information. This stage will produce a LLVM profile if the toolchain is built with shared LLVM linkage or a Rust profile if built with static linkage.

Stage 4

Android's Rust code is re-built using the CS-PGO instrumented compiler and the results are merged with the profiles generated during Stage 2.

Stage 5

The final compiler is built using the profile produced in Stage 4.

Executing the Pipeline

The PGO pipeline can be executed using the pipeline.py script. No command line arguments are necessary, though the --build-name and --dist-path arguments can be used to set the prebuilt archive name and output directory respectively. This process will take six or more hours to complete, depending on hardware.

Profile data is collected from three work sets:

  1. Compiling the Rust toolchain (Stages 1 and 3)
  2. Compiling Android's Rust code for a x86_64 target (Stages 2 and 4)
  3. Compiling Android's Rust code for an ARM target (Stages 2 and 4)

Notes About LLVM Versions

The PGO pipeline requires the Rust and C++ compilers to emit the same version of LLVM bitcode. This is not always the case, as Rust uses an internal copy of LLVM to generate code in rustc. It is this internal LLVM library that is compiled by Android's Clang/LLVM prebuilts. If the LLVM library and the code it emits use incompatible versions of LLVM IR or the runtime libraries then lld will be unable to link the toolchain.

Reproducing Android Toolchains

Starting with Rust 1.56.1 it is possible to reproduce Android‘s Rust toolchain prebuilts. To do this you will start with the prebuilt’s manifest, which is located in the prebuilts/rust/linux-x86/<version> directory. Starting with Rust 1.61.0 the manifest is also available in the toolchain/android_rust/artifacts/<version> directory.

To begin reproducing a given build navigate to a new directory and run the following command:

$ repo init -m /path/to/manifest.xml -b rust-toolchain && repo sync -c -j16

To reproduce toolchains before Rust version 1.61.0 run the following command:

$ ./toolchain/android_rust/tools/build.py --lto thin --llvm-linkage shared

To reproduce toolchains version 1.61.0 and above use the build commands listed in the toolchain/android_rust/artifacts/<version>/rust_build_command.<target>.<build id>.sh files. Be sure to replace the paths to the PGO profiles from the saved commands with paths to the profiles in the artifacts directory.

Building with Docker

The Android Rust toolchain provides a Docker image that can be used when building the toolchain. This Docker image is meant to replicate the environment that our official prebuilt releases are built in.

To use this Docker image you'll first need to build it. To do so enter the toolchain/android_rust directory and run the following command:

$ docker build -t android-rust:dev .

To then run the build in the docker image you can run the docker_build.py script and pass it the same arguments you would to build.py.

To run start an interactive Docker instance with the correct mount point for the Android source you can run this command (substituting in the correct path to your Android root):

$ sudo docker run --mount type=bind,source=/path/to/android/root,target=/android -it android-rust:dev /bin/bash

Section 5: Other Tasks

Generating the Rust module list

The Rust toolchain includes a list of the known Rust targets in Android. This is currently only used when processing Soong traces for performance information and many of the targets can be discovered automatically so an out of date file is not a major issue. To increase the accuracy of any parsed traces you can update the module list using the following commands:

$ m queryview
$ bazel query --config=queryview 'kind("rust_(?!default).*", //...)' --output label_kind > toolchain/android_rust/android-rust-modules.txt