Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Edit the "Compiler Source Code" chapter #1307

Merged
merged 8 commits into from
May 16, 2022
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 25 additions & 23 deletions src/compiler-src.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,13 @@
<!-- toc -->

Now that we have [seen what the compiler does](./overview.md), let's take a
look at the structure of the contents of the rust-lang/rust repo.
look at the structure of the [`rust-lang/rust`] repository, where the rustc
source code lives.

[`rust-lang/rust`]: https://github.com/rust-lang/rust

> You may find it helpful to read the ["Overview of the compiler"](./overview.md)
> chapter, which introduces how the compiler works, before this one.

## Workspace structure

Expand All @@ -16,29 +22,17 @@ The repository consists of three main directories:

- `compiler/` contains the source code for `rustc`. It consists of many crates
that together make up the compiler.

pierwill marked this conversation as resolved.
Show resolved Hide resolved
- `library/` contains the standard libraries (`core`, `alloc`, `std`,
`proc_macro`, `test`), as well as the Rust runtime (`backtrace`, `rtstartup`,
`lang_start`).

- `src/` contains the source code for rustdoc, clippy, cargo, the build system,
language docs, etc.

## Standard library
pierwill marked this conversation as resolved.
Show resolved Hide resolved

The standard library crates are all in `library/`. They have intuitive names
like `std`, `core`, `alloc`, etc. There is also `proc_macro`, `test`, and
other runtime libraries.

This code is fairly similar to most other Rust crates except that it must be
built in a special way because it can use unstable features.
compiler tests, language docs, etc.
pierwill marked this conversation as resolved.
Show resolved Hide resolved

## Compiler

> You may find it helpful to read [The Overview Chapter](./overview.md) first,
> which gives an overview of how the compiler works. The crates mentioned in
> this section implement the compiler, and are underneath `compiler/`

The compiler is implemented in the various `compiler/` crates.
The `compiler/` crates all have names starting with `rustc_*`. These are a
collection of around 50 interdependent crates ranging in size from tiny to
huge. There is also the `rustc` crate which is the actual binary (i.e. the
Expand Down Expand Up @@ -87,7 +81,7 @@ explanation of these crates here.

### Big picture

The dependency structure is influenced strongly by two main factors:
The dependency structure is influenced by two main factors:

1. Organization. The compiler is a _huge_ codebase; it would be an impossibly
large crate. In part, the dependency structure reflects the code structure
Expand All @@ -101,12 +95,11 @@ At the very bottom of the dependency tree are a handful of crates that are used
by the whole compiler (e.g. [`rustc_span`]). The very early parts of the
compilation process (e.g. parsing and the AST) depend on only these.

Pretty soon after the AST is constructed, the compiler's [query system][query]
gets set up. The query system is set up in a clever way using function
After the AST is constructed and other early analysis is done, the compiler's [query system][query]
gets set up. The query system is set up in a clever way using function
pointers. This allows us to break dependencies between crates, allowing more
parallel compilation.

However, since the query system is defined in [`rustc_middle`], nearly all
The query system is defined in [`rustc_middle`], so nearly all
subsequent parts of the compiler depend on this crate. It is a really large
crate, leading to long compile times. Some efforts have been made to move stuff
out of it with limited success. Another unfortunate side effect is that sometimes
Expand All @@ -116,7 +109,7 @@ linting functionality is scattered across earlier parts of the crate,

[`rustc_lint`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lint/index.html

More generally, in an ideal world, it seems like there would be fewer, more
Ideally there would be fewer, more
cohesive crates, with incremental and parallel compilation making sure compile
times stay reasonable. However, our incremental and parallel compilation haven't
gotten good enough for that yet, so breaking things into separate crates has
Expand Down Expand Up @@ -180,6 +173,15 @@ from `src/tools/`, such as [`tidy`] or [`compiletest`].

[bootstch]: ./building/bootstrapping.md

## Standard library

The standard library crates are all in `library/`. They have intuitive names
like `std`, `core`, `alloc`, etc. There is also `proc_macro`, `test`, and
other runtime libraries.

This code is fairly similar to most other Rust crates except that it must be
built in a special way because it can use unstable features.

## Other

There are a lot of other things in the `rust-lang/rust` repo that are related
Expand Down