rustdoc: run on plain Markdown files & replace the use of pandoc for guides/tutorial/manual etc. #12747
Conversation
|
This is mainly just to get a review on the implementation, it still can't generate a table of contents (working on that now). |
| @@ -266,6 +265,15 @@ pub fn find_testable_code(doc: &str, tests: &mut ::test::Collector) { | |||
| }) | |||
| } | |||
| } | |||
| extern fn header(_ob: *buf, text: *buf, level: libc::c_int, opaque: *libc::c_void) { | |||
| unsafe { | |||
| vec::raw::buf_as_slice((*text).data, (*text).size as uint, |text| { | |||
There was a problem hiding this comment.
This may be null (at least sundown checks for it), so it's probably worth checking for null up front.
I also think that str::raw gives some better usage than vec => str::from_utf8 (I only realized this much later after writing this code).
There was a problem hiding this comment.
I don't understand "some better usage"? Using this avoids allocations, unlike most functions in str::raw (I think)?
There was a problem hiding this comment.
Ah I just found that there were functions that didn't require a closure/extra level of indentation, and it goes straight to a string from a buffer. Not really necessary though, I wouldn't be worried about allocations at all at locations like this.
|
Ah, I see, this is a work in progress. This seems fine to me, I'd love to drop the pandoc/node deps! |
|
Oh, erm... I forgot to add the new |
|
Added. The patch probably makes more sense now... |
|
|
||
| fn load_string(input: &Path) -> io::IoResult<Option<~str>> { | ||
| let mut f = try!(io::File::open(input)); | ||
| let d = try!(f.read_to_end()); |
There was a problem hiding this comment.
You could even combine this:
Ok(str::from_utf8_owned(try!(File::open(input).read_to_end())))(not that it's a whole lot clearer, more just combining the two try! statements.
|
Aha, this does indeed make more sense! This looks pretty awesome, could we migrate to using this now? |
|
I can do the Makefile changes and cleanups if you don't want to do it. |
|
|
||
| let err = write!( | ||
| &mut out, | ||
| r#"<!doctype html> |
There was a problem hiding this comment.
nit: DOCTYPE should be uppercase (doesn't really matter)
|
Thanks for the offer, but I'll do the makefile stuff. |
|
Pushed table of contents creation. Generated docs: http://huonw.github.io/rust/build/doc/index.html (I haven't yet adjusted the stylesheet other than the syntax highlighting, so some edge cases look peculiar). I met a snag with the makefiles/tutorials/guides: if we convert them into the format that sundown understands, we're moving away from what pandoc understands, and so we lose the ability to go directly to pdf and epub from Markdown. (i'll address review and rebase later... going to sleep now.) |
|
We could probably write a Python script that would transcript the files back to pandoc. What are the differences exactly? |
|
This continues to look awesome, nice work! I wouldn't be too worried about losing the pdf/epub formats. I would think that a pdf could be generated by printing the HTML page (with an appropriate stylesheet) |
Basically just |
|
I personally do not think a PDF tutorial and manual is that important, though I know others disagree. I would be ok with dropping them (perhaps temporarily) for the sake of progress, but also with the conversion script and retaining the optional pandoc dep. |
|
@huonw Those are equivalent. In my |
|
@adrientetar It's not quite equivalent... it seems that sundown doesn't actually support multiple languages (so my And, writing the multiple langauges/classes as I'm editing the make files so that (On the C note above: this change loses us the C syntax highlighting in the FFI guide.) |
|
Also, btw, it seems sundown has some support for creating tables of contents; however I think the Rust version I've written here is better... at least; it's obvious how to use it, is more adjustable and gives the same output as pandoc (unlike the sundown one, afaict). I'm happy to try to use the C one if someone particularly hates the duplication, but ❤️ pure Rust from me. |
|
So, anyway, I've been having the dumb a lot in this PR: I just discovered that sundown does support the /me writes "try thinking before typing" in big letters across his computer screen |
|
Cool then. Maybe this notation is required when you have multiple class names. |
|
Yeah, that's exactly it. |
|
r? Now switches HTML rendering over to rustdoc with a rather large makefile rewrite. |
|
I think you forgot to change the code highlighting in |
| # RUSTDOC_DEPS_xyz are extra dependencies for the rustdoc invocation | ||
| # on xyz | ||
| # | ||
| # L10N_LANGS are the languages for which the crates have been |
|
r? |
|
This looks fantastic, nice work! r=me with minor things here and there addressed. The only big one which I think needs to be fixed is making sure |
|
Comments reviewed; but the last commit is debatable and worth a review itself... |
This theoretically gives rustdoc the ability to render our guides, tutorial and manual (not in practice, since the files themselves need to be adjusted slightly to use Sundown-compatible functionality). Fixes rust-lang#11392.
markdown files.
This means that
# Foo
## Bar
# Baz
### Qux
## Quz
Gets a TOC like
1 Foo
1.1 Bar
2 Baz
2.0.1 Qux
2.1 Quz
This functionality is only used when rendering a single markdown file,
never on an individual module, although it could very feasibly be
extended to allow modules to opt-in to a table of contents (std::fmt
comes to mind).
This avoids having to include JS in the guide/tutorial/manual pages just to get the headers being links. The on-hover behaviour showing the little section marker § is preserved, because that gives a useful hint that the heading is a link.
The manual, tutorial and guides need the feature gates quite often, unfortunately, so this is the low-cost path to migrating to use rustdoc. This is only activated for pure-Markdown files. Preferably this would be avoided: rust-lang#12773
The changes are basically just because rustdoc runs tests/rendering on more snippets by default (i.e. everything without a `notrust` tag), and not anything significant.
This converts it to be very similar to crates.mk, with a single list of the documentation items creating all the necessary bits and pieces. Changes include: - rustdoc is used to render HTML & test standalone docs - documentation building now obeys NO_REBUILD=1 - testing standalone docs now obeys NO_REBUILD=1 - L10N is slightly less broken (in particular, it shares dependencies and code with the rest of the code) - PDFs can be built for all documentation items, not just tutorial and manual - removes the obsolete & unused extract-tests.py script - adjust the CSS for standalone docs to use the rustdoc syntax highlighting
E.g. this stops check-...-doc rules for `rustdoc.md` and `librustdoc` from stamping on each other, so that they are correctly built and tested. (Previously only the rustdoc crate was tested.)
This is meant to be compiling a crate, but the crate_id attribute seems to be upsetting it if the attribute is actually on the crate. I.e. this makes this test compile by putting the crate_id attribute on a function and so it's ignored. Such a hack. :(
parsing limitations.
Sundown parses
```
~~~
as a valid codeblock (i.e. mismatching delimiters), which made using
rustdoc on its own documentation impossible (since it used nested
codeblocks to demonstrate how testable codesnippets worked).
This modifies those snippets so that they're delimited by indentation,
but this then means they're tested by `rustdoc --test` & rendered as
Rust code (because there's no way to add `notrust` to
indentation-delimited code blocks). A comment is added to stop the
compiler reading the text too closely, but this unfortunately has to be
visible in the final docs, since that's the text on which the
highlighting happens.
This restores the old behaviour (as compared to building PDF versions of all standalone docs), because some of the guides use unicode characters, which seems to make pdftex unhappy.
This gives Rustdoc the ability to render our guides, tutorial and manual and modifies the those documents (minor modifications) and makefiles to achieve this. See commits for more details (especially the makefile rewrite).
- remove `node.js` dep., it has no effect as of #12747 (1) - switch between LaTeX compilers, some cleanups - CSS: fixup the print stylesheet, refactor highlighting code (2) (1): `prep.js` outputs its own HTML directives, which `pandoc` cannot recognize when converting the document into LaTeX (this is why the PDF docs have never been highlighted as of now). Note that if we were to add the `.rust` class to snippets, we could probably use pandoc's native highlighting capatibilities i.e. Kate ([here is](http://adrientetar.github.io/rust-tuts/tutorial/tutorial.pdf) an example of that). (2): the only real highlighting change is for lifetimes which are now brown instead of red, the rest is just refactor of twos shades of red that look the same. Also I made numbers highlighting for src in rustdoc a tint more clear so that it is less bothering. @alexcrichton, @huonw Closes #9873. Closes #12788.
…r=jonas-schievink fix: Update 1.63 proc macro ABI to match rustc This updates us to the ABI used by rustc 1.63.0-beta.5, which will likely be the ABI of the next stable Rust release. It should also work on nightly (for now, but future changes won't be supported until the rustc version is bumped). cc rust-lang/rust-analyzer#12600
…xendoo Type safe CLI implementation for clippy-dev Use the derive feature of `clap` to generate CLI of clippy-dev. Adding new commands will be easier in the future and we get better compile time checking through exhaustive matching. --- I think I tested everything locally. But I would appreciate if the reviewer could go over it again, so that everything keeps working. changelog: none
This gives Rustdoc the ability to render our guides, tutorial and manual and modifies the those documents (minor modifications) and makefiles to achieve this.
See commits for more details (especially the makefile rewrite).