Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RLlib; offline RL; docs] Docs do-over (new API stack): Offline RL. #48818

Merged
merged 83 commits into from
Jan 3, 2025
Merged

[RLlib; offline RL; docs] Docs do-over (new API stack): Offline RL. #48818

merged 83 commits into from
Jan 3, 2025

Conversation

simonsays1980
Copy link
Collaborator

@simonsays1980 simonsays1980 commented Nov 20, 2024 •
edited
Loading

Why are these changes needed?

With the new Offline RL API soon to be released the corresponding docs need an overhaul. This PR includes all these changes with:

  • An introduction to the new API and its key principles
  • Examples of how to record expert data, train on expert data, convert external data and old SampleBatch data.
  • Hints and notes about data formats, underlying procedures, etc.
  • Scaling and tuning guides (to come) and descriptions about internals

These docs should enable old and new RLlib users to implement offline RL on their data fast and smoothly. Furthermore, it should apply to real industry use cases which need to digest peta-bytes of offline data to train policies for highly-complex tasks.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@simonsays1980
Started rewriting Offline RL documentation and added examples, descri…
1513511 
…ptions, hints, notes, and testcode.

Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
@simonsays1980 simonsays1980 marked this pull request as ready for review November 20, 2024 13:08
@simonsays1980 simonsays1980 added rllib RLlib related issues rllib-docs-or-examples Issues related to RLlib documentation or rllib/examples labels Nov 20, 2024
@sven1977 sven1977 changed the title [RLlib; Offline RL] - Overhaul Offline RL Docs [RLlib; Offline RL] Documentation do-over: Offline RL. Nov 20, 2024
@sven1977
Copy link
Contributor

One warning, which breaks the readthedocs build. I think it's this one:

/home/docs/checkouts/readthedocs.org/user_builds/anyscale-ray/checkouts/48818/doc/source/rllib/rllib-offline.rst:666: WARNING: Enumerated list ends without a blank line; unexpected unindent.

@simonsays1980
Fixed indent error.
f9ab4f6 
Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
@simonsays1980
Removed :hide: syntax from testsetup b/c it was producing an error.
d3cac1f 
Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
sven1977
sven1977 reviewed Nov 21, 2024
View reviewed changes
doc/source/rllib/rllib-offline.rst Outdated
Comment on lines 422 to 426
# Make sure you use the new API stack.
.api_stack(
enable_rl_module_and_learner=True,
enable_env_runner_and_connector_v2=True,
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Make sure you use the new API stack.
.api_stack(
enable_rl_module_and_learner=True,
enable_env_runner_and_connector_v2=True,
)

@simonsays1980
Added more sections to new offline docs. In addition completed some m…
e274aa6 
…issing docstrings in 'AlgorithmConfig'.

Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
simonsays1980 and others added 22 commits December 16, 2024 14:58
@simonsays1980
Fixed reference error for offline API reference. The old stack is sti…
4ddd6a2 
…ll in there.

Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
@simonsays1980
Added offline.rst to index.rst b/c it was not referenced anywhere, ye…
bccf8b5 
…t, which produces an error in the docs tests.§

Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
@simonsays1980
Fixed some more reference bugs due to missing automsummaries.
19738f8 
Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
@simonsays1980
CHanged autodoc for SCHEMA by adding special docstring for data attri…
1ae112b 
…bute.

Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
@simonsays1980
Merge branch 'master' into offline-rl-docs-overhaul
f204881
@sven1977
Merge branch 'master' of https://github.com/ray-project/ray into offl…
f16103a 
…ine-rl-docs-overhaul
@sven1977
wip
72d2a96
@sven1977
wip
de13891
@angelinalg @simonsays1980
[docstring] adding link to runtime_env API ref to instances of ray_re…
4f0d4ef 
…mote_args (ray-project#48955)

Per request from @GokuMohandas, add link to runtime_env API ref for all
commands with the `ray_remote_args` parameter to improve usability.

## Related issue number

<!-- For example: "Closes ray-project#1234" -->

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [ ] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: angelinalg <angelina@anyscale.com>
Signed-off-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
@win5923 @simonsays1980
[Doc] Fix incorrect URL in persist-kuberay-custom-resource-logs.md (r…
dd023aa 
…ay-project#49323)

Signed-off-by: win5923 <ken89@kimo.com>
@dentiny @simonsays1980
[core] Minor update on task manager (ray-project#49272)
6184aef 
Signed-off-by: hjiang <hjiang@anyscale.com>
@dentiny @simonsays1980
[core] Fix cpp inclusion check (ray-project#49301)
bad9714 
Signed-off-by: hjiang <dentinyhao@gmail.com>
@dentiny @simonsays1980
[core] Remove unused function arg (ray-project#49331)
edb35ef 
Signed-off-by: dentiny <dentinyhao@gmail.com>
@simonsays1980
Removed recording examples from CI tests b/c it times out.
13fa5df 
Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
@simonsays1980
Merge branch 'master' into offline-rl-docs-overhaul
4d55d74
@simonsays1980
Added a line to code-block as this was failing in doc tests.
d24ddee 
Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
@simonsays1980
Merge branch 'master' into offline-rl-docs-overhaul
6cb25f1
@simonsays1980
Merged master and resolved conflicts.
c5afabe 
Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
@simonsays1980
Removed another test code block as test was timing out.
45ccce5 
Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
@simonsays1980
Fixed an import error in example and ran vale.
7278092 
Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
@simonsays1980
Changed 'BCTorchRLModule into DefaultBCTorchRLModule, which was causi…
10d8059 
…ng a bug.

Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
@simonsays1980
Merge branch 'master' into offline-rl-docs-overhaul
c046b87
@sven1977 sven1977 enabled auto-merge (squash) January 2, 2025 12:02
@github-actions github-actions bot disabled auto-merge January 3, 2025 11:51
@sven1977 sven1977 merged commit 59a8cb9 into ray-project:master Jan 3, 2025
5 checks passed
@sven1977 sven1977 changed the title [RLlib; Offline RL] Documentation do-over: Offline RL. [RLlib; offline RL; docs] Docs do-over (new API stack): Offline RL. Jan 3, 2025
roshankathawate pushed a commit to roshankathawate/ray that referenced this pull request Jan 7, 2025
@simonsays1980
[RLlib; Offline RL] Documentation do-over: Offline RL. (ray-project#4…
7b54cb7 
…8818)
roshankathawate pushed a commit to roshankathawate/ray that referenced this pull request Jan 9, 2025
@simonsays1980 @roshankathawate
[RLlib; Offline RL] Documentation do-over: Offline RL. (ray-project#4…
4418da5 
…8818)

Signed-off-by: Roshan Kathawate <roshankathawate@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sven1977 sven1977 sven1977 approved these changes

@angelinalg angelinalg angelinalg approved these changes

@maxpumperla maxpumperla Awaiting requested review from maxpumperla maxpumperla is a code owner

@hongpeng-guo hongpeng-guo Awaiting requested review from hongpeng-guo

@justinvyu justinvyu Awaiting requested review from justinvyu

@matthewdeng matthewdeng Awaiting requested review from matthewdeng

@raulchen raulchen Awaiting requested review from raulchen

@woshiyyya woshiyyya Awaiting requested review from woshiyyya

@pcmoritz pcmoritz Awaiting requested review from pcmoritz

@kevin85421 kevin85421 Awaiting requested review from kevin85421

Assignees
No one assigned
Labels
go add ONLY when ready to merge, run all tests rllib RLlib related issues rllib-docs-or-examples Issues related to RLlib documentation or rllib/examples
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.