From 919becb360ca10785a86ab6ff225ed1aefaa4c5d Mon Sep 17 00:00:00 2001 From: pierwill Date: Thu, 17 Feb 2022 17:04:56 -0600 Subject: [PATCH 1/8] Edit the "Compiler Source Code" chapter --- src/compiler-src.md | 35 ++++++++++++++--------------------- 1 file changed, 14 insertions(+), 21 deletions(-) diff --git a/src/compiler-src.md b/src/compiler-src.md index 94add9aac..23cbcf53b 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -3,7 +3,10 @@ Now that we have [seen what the compiler does](./overview.md), let's take a -look at the structure of the contents of the rust-lang/rust repo. +look at the structure of the [`rust-lang/rust`] repository, where the rustc +source code lives. + +[`rust-lang/rust`]: https://github.com/rust-lang/rust ## Workspace structure @@ -16,22 +19,11 @@ The repository consists of three main directories: - `compiler/` contains the source code for `rustc`. It consists of many crates that together make up the compiler. - - `library/` contains the standard libraries (`core`, `alloc`, `std`, `proc_macro`, `test`), as well as the Rust runtime (`backtrace`, `rtstartup`, `lang_start`). - - `src/` contains the source code for rustdoc, clippy, cargo, the build system, - language docs, etc. - -## Standard library - -The standard library crates are all in `library/`. They have intuitive names -like `std`, `core`, `alloc`, etc. There is also `proc_macro`, `test`, and -other runtime libraries. - -This code is fairly similar to most other Rust crates except that it must be -built in a special way because it can use unstable features. + compiler tests, language docs, etc. ## Compiler @@ -87,7 +79,7 @@ explanation of these crates here. ### Big picture -The dependency structure is influenced strongly by two main factors: +The dependency structure is influenced 1. Organization. The compiler is a _huge_ codebase; it would be an impossibly large crate. In part, the dependency structure reflects the code structure @@ -101,12 +93,9 @@ At the very bottom of the dependency tree are a handful of crates that are used by the whole compiler (e.g. [`rustc_span`]). The very early parts of the compilation process (e.g. parsing and the AST) depend on only these. -Pretty soon after the AST is constructed, the compiler's [query system][query] -gets set up. The query system is set up in a clever way using function -pointers. This allows us to break dependencies between crates, allowing more -parallel compilation. - -However, since the query system is defined in [`rustc_middle`], nearly all +After the AST is constructed, the compiler's [query system][query] +gets set up.[^query-setup] +The query system is defined in [`rustc_middle`], so nearly all subsequent parts of the compiler depend on this crate. It is a really large crate, leading to long compile times. Some efforts have been made to move stuff out of it with limited success. Another unfortunate side effect is that sometimes @@ -116,7 +105,7 @@ linting functionality is scattered across earlier parts of the crate, [`rustc_lint`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lint/index.html -More generally, in an ideal world, it seems like there would be fewer, more +Ideally there would be fewer, more cohesive crates, with incremental and parallel compilation making sure compile times stay reasonable. However, our incremental and parallel compilation haven't gotten good enough for that yet, so breaking things into separate crates has @@ -134,6 +123,10 @@ compilation to completion. [orgch]: ./overview.md +[^query-setup]: The query system is set up in a clever way using function +pointers. This allows us to break dependencies between crates, allowing more +parallel compilation. + ## rustdoc The bulk of `rustdoc` is in [`librustdoc`]. However, the `rustdoc` binary From 987342a9bcb73801194070f5b6f5c1620bece566 Mon Sep 17 00:00:00 2001 From: pierwill Date: Thu, 17 Feb 2022 17:08:02 -0600 Subject: [PATCH 2/8] fix --- src/compiler-src.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/compiler-src.md b/src/compiler-src.md index 23cbcf53b..41ed913f5 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -79,7 +79,7 @@ explanation of these crates here. ### Big picture -The dependency structure is influenced +The dependency structure is influenced by two main factors: 1. Organization. The compiler is a _huge_ codebase; it would be an impossibly large crate. In part, the dependency structure reflects the code structure From 07a0415eb6d5eb50a2c51059e3717617809ece08 Mon Sep 17 00:00:00 2001 From: pierwill Date: Thu, 17 Feb 2022 17:11:37 -0600 Subject: [PATCH 3/8] Edit and move note --- src/compiler-src.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/compiler-src.md b/src/compiler-src.md index 41ed913f5..5c58c6050 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -8,6 +8,9 @@ source code lives. [`rust-lang/rust`]: https://github.com/rust-lang/rust +> You may find it helpful to read the ["Overview of the compiler"](./overview.md) +> chapter, which introduces how the compiler works, before this one. + ## Workspace structure The `rust-lang/rust` repository consists of a single large cargo workspace @@ -27,10 +30,7 @@ The repository consists of three main directories: ## Compiler -> You may find it helpful to read [The Overview Chapter](./overview.md) first, -> which gives an overview of how the compiler works. The crates mentioned in -> this section implement the compiler, and are underneath `compiler/` - +The compiler is implemented in the various `compiler/` crates. The `compiler/` crates all have names starting with `rustc_*`. These are a collection of around 50 interdependent crates ranging in size from tiny to huge. There is also the `rustc` crate which is the actual binary (i.e. the From 496a7b378a57b20de48e6f7c51d5a289ef58cf15 Mon Sep 17 00:00:00 2001 From: pierwill Date: Thu, 17 Feb 2022 17:13:24 -0600 Subject: [PATCH 4/8] Replace spaces in list --- src/compiler-src.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/compiler-src.md b/src/compiler-src.md index 5c58c6050..56465f84f 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -22,9 +22,11 @@ The repository consists of three main directories: - `compiler/` contains the source code for `rustc`. It consists of many crates that together make up the compiler. + - `library/` contains the standard libraries (`core`, `alloc`, `std`, `proc_macro`, `test`), as well as the Rust runtime (`backtrace`, `rtstartup`, `lang_start`). + - `src/` contains the source code for rustdoc, clippy, cargo, the build system, compiler tests, language docs, etc. From 18c00a630a145515e5daf853eaca0d3ada39407b Mon Sep 17 00:00:00 2001 From: pierwill Date: Thu, 17 Feb 2022 17:14:44 -0600 Subject: [PATCH 5/8] Move "Stand library" section --- src/compiler-src.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/compiler-src.md b/src/compiler-src.md index 56465f84f..3593b479a 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -175,6 +175,15 @@ from `src/tools/`, such as [`tidy`] or [`compiletest`]. [bootstch]: ./building/bootstrapping.md +## Standard library + +The standard library crates are all in `library/`. They have intuitive names +like `std`, `core`, `alloc`, etc. There is also `proc_macro`, `test`, and +other runtime libraries. + +This code is fairly similar to most other Rust crates except that it must be +built in a special way because it can use unstable features. + ## Other There are a lot of other things in the `rust-lang/rust` repo that are related From d835a10188015efae08fd537500aa530e3b6cdcf Mon Sep 17 00:00:00 2001 From: pierwill Date: Sat, 19 Feb 2022 19:38:00 -0600 Subject: [PATCH 6/8] Replace query content --- src/compiler-src.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/src/compiler-src.md b/src/compiler-src.md index 3593b479a..9ec6f5877 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -96,7 +96,9 @@ by the whole compiler (e.g. [`rustc_span`]). The very early parts of the compilation process (e.g. parsing and the AST) depend on only these. After the AST is constructed, the compiler's [query system][query] -gets set up.[^query-setup] +gets set up.The query system is set up in a clever way using function +pointers. This allows us to break dependencies between crates, allowing more +parallel compilation. The query system is defined in [`rustc_middle`], so nearly all subsequent parts of the compiler depend on this crate. It is a really large crate, leading to long compile times. Some efforts have been made to move stuff @@ -125,10 +127,6 @@ compilation to completion. [orgch]: ./overview.md -[^query-setup]: The query system is set up in a clever way using function -pointers. This allows us to break dependencies between crates, allowing more -parallel compilation. - ## rustdoc The bulk of `rustdoc` is in [`librustdoc`]. However, the `rustdoc` binary From 53cdbaeaf40ea5d316741779bc792b90ed3447e0 Mon Sep 17 00:00:00 2001 From: pierwill <19642016+pierwill@users.noreply.github.com> Date: Mon, 21 Feb 2022 22:55:26 -0600 Subject: [PATCH 7/8] Update src/compiler-src.md Co-authored-by: Noah Lev --- src/compiler-src.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/compiler-src.md b/src/compiler-src.md index 9ec6f5877..b07dfe566 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -96,7 +96,7 @@ by the whole compiler (e.g. [`rustc_span`]). The very early parts of the compilation process (e.g. parsing and the AST) depend on only these. After the AST is constructed, the compiler's [query system][query] -gets set up.The query system is set up in a clever way using function +gets set up. The query system is set up in a clever way using function pointers. This allows us to break dependencies between crates, allowing more parallel compilation. The query system is defined in [`rustc_middle`], so nearly all From edab943122486d47ae9a2baa537c228fa77dcbfe Mon Sep 17 00:00:00 2001 From: Yuki Okushi Date: Mon, 16 May 2022 20:58:11 +0900 Subject: [PATCH 8/8] Update src/compiler-src.md Co-authored-by: Noah Lev --- src/compiler-src.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/compiler-src.md b/src/compiler-src.md index b07dfe566..ad6968bb4 100644 --- a/src/compiler-src.md +++ b/src/compiler-src.md @@ -95,7 +95,7 @@ At the very bottom of the dependency tree are a handful of crates that are used by the whole compiler (e.g. [`rustc_span`]). The very early parts of the compilation process (e.g. parsing and the AST) depend on only these. -After the AST is constructed, the compiler's [query system][query] +After the AST is constructed and other early analysis is done, the compiler's [query system][query] gets set up. The query system is set up in a clever way using function pointers. This allows us to break dependencies between crates, allowing more parallel compilation.