[AIR] Tracking issues in AIR examples tested on Windows #27851
Assignees
Comments
|
It might make sense to split these up into groups: fail to run to completion, emits internal deprecation warnings, emits other warnings, fails to produce the correct answer. Some examples may appear in more than one group. |
|
In an effort to make the notes more useful, I've grouped the issues I encountered above together so that we can more easily track them and check them off the list as they are addressed. They're grouped into internal warnings, errors, other warnings, and other todo items. I've labeled problems that I think are specific to a single example with the name of the affected example. @mattip Hopefully this is useful, but if you still think it would be good to group the examples by the problems encountered, I can do that too. Internal warnings[All four are fixed by #28315]
Details
Details
Details
Details
Details
Details
Details
Details
Detailsand
DetailsOther warnings
Details
Details
Details
Details
Details
Details
Errors - fails to run
DetailsAffects the following examples:
DetailsOther todo items
DetailsExtra
DetailsExample information to include near the top of the notebook # import comet_ml at the top of your file
from comet_ml import Experiment
import os
comet_project = "ray_air_example"
# Create an experiment with your api key
experiment = Experiment(
api_key=os.environ['COMET_API_KEY'],
project_name=comet_project,
workspace="your_user_name",
)
# Rest of the example goes here
experiment.end()
Details
import wandb
wandb_project = "ray_air_example"
entity = "peytondmurray"
wandb.init(project=wandb_project, entity=entity)
Detailsfrom ray.ml.preprocessors import Chain, OrdinalEncoder, SimpleImputerchanges to from ray.data.preprocessors import Chain, OrdinalEncoder, SimpleImputerand from ray.ml.checkpoint import Checkpoint
from ray.ml.predictors.integrations.xgboost import XGBoostPredictorchanges to from ray.air.checkpoint import Checkpoint
from ray.train.xgboost import XGBoostPredictor
Details
|
|
The "Bad file descriptor" in |
|
Does the rllib initialization error in |
|
Thanks! This was quite an effort! |
|
@mattip I just tried running the I also tried creating an environment from scratch by doing pip install -ve . .[rllib] .[air] .[tune] .[data] gym tensorflow pygame( Here's the output of Outdated packages
Edit: it looks like at least for |
This was referenced Aug 27, 2022
7 tasks
7 tasks
7 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened + What you expected to happen
I've spent some time testing the AIR examples on Windows. This issue is intended to track issues encountered while running through the examples manually. Each example has it's own collapsible section - let me know if a better format for these notes would be preferred.
1.
torch_image_example2.
convert_existing_pytorch_code_to_ray_airNo additional notes, example worked as intended.
3.
tfx_tabular_train_to_serveDuring the fit several warnings were generated before the fit failed.
Deprecation warnings:
placement_group_parameterobject_store_memoryplacement_groupplacement_group_bundle_indexplacement_group_capture_child_taskspandasdeprecation warning:4.
huggingface_text_classificationresult = trainer.fit()the fit fails with the following error:5.
sklearn_exampleNo additional notes, example worked as intended.
6.
xgboost_exampleray.worker.get_resource_idsdeprecation warningray.worker.get_resource_idsbeing called;python/ray/__init__.pyprints the stack for several of these deprecation warnings - seeray/python/ray/__init__.py
Line 201 in ea47d97
7.
analyze_tuning_resultsDeprecationWarningaboutray.worker.get_resource_idsas inxgboost_exampletraceback.print_stackas inxgboost_example8.
lightgbm_example2*num_workers + 1. Not sure if this is intended.DeprecationWarningaboutray.worker.get_resource_idsas inxgboost_exampletraceback.print_stackas inxgboost_example9.
torch_incremental_learningDuring the first training step:
parallelismargument was directly specified in the example. However, a warning about aparallelismargument was generated for some internal call:pandaswarnings generated about setting with copy:ray.serve.apideprecation warningray.serve.deploymentdeprecation warningAlso, the same errors about setting on a copy of a dataframe and read-only numpy arrays errors also appeared in the second training step.
10.
rl_serving_exampleflatbuffersmaintainers have not made a release to handle this, see New Python release to avoid Deprecation Warning google/flatbuffers#6957. Likely nothing to be done here, as new versions offlatbuffershave not been released to PyPI:kerasuses deprecatedpillowfunctions. This is fixed by a recent commit tokeras: Fix usage of deprecated Pillow interpolation methods keras-team/keras#16746, so will likely go away in the future when this dependency is updated:rllibdeprecation warnings due to invalid escape sequences at the start of fit:and
Deprecation warnings - same as in
tfx_tabular_train_to_serveexample:placement_group_parameterobject_store_memoryplacement_groupplacement_group_bundle_indexplacement_group_capture_child_tasksDeprecation warnings coming from
gymrelated to usingCartPole-v0:rllibappears to be trying to initialize an object with a shape given by atensorflow.python.framework.tensor_shape.Dimension, when it should use integers:11.
rl_online_examplerllib.agents.marwildeprecation warning:Deprecation warnings - same as in
tfx_tabular_train_to_serveexample:placement_group_parameterobject_store_memoryplacement_groupplacement_group_bundle_indexplacement_group_capture_child_tasksSame
flatbuffers,keras, andCartPole-v0warnings as inrl_serving_example:flatbuffersuse ofimpkerascalling deprecatedpillowfunctionsCartPole-v0out of dateFails to train due to the same
Dimensionerror as inrl_serving_example:12.
rl_offline_exampleSame errors as in the
rl_online_example:Same
flatbuffers,keras, andCartPole-v0warnings as inrl_online_example:flatbuffersuse ofimpkerascalling deprecatedpillowfunctionsCartPole-v0out of dateplacement_group_parameterobject_store_memoryplacement_groupplacement_group_bundle_indexplacement_group_capture_child_tasksrllib.agents.marwilFit fails because of same
tensorflow.python.framework.tensor_shape.Dimensionerror as inrl_online_example13.
upload_to_comet_mlDeprecationWarningaboutray.worker.get_resource_idsas inxgboost_exampletraceback.print_stackas inxgboost_exampleCOMET_API_KEYenvironment variable. Definingcomet_projectat the top of the notebook also avoids having it undefined further down. For example:experiment.end()at the end of the notebook14.
upload_to_wandbDeprecationWarningaboutray.worker.get_resource_idsas inxgboost_exampletraceback.print_stackas inxgboost_exampleWANDB_API_KEYenvironment variable to the your API keywandb.init, taking care to set the entity - the API docs say it's optional, but I kept getting an error until I set it:15.
feast_examplecomet_mldeprecation warning about imp module (?):Deprecation warnings:
placement_group_parameterobject_store_memoryplacement_groupplacement_group_bundle_indexplacement_group_capture_child_tasksSame
DeprecationWarningaboutray.worker.get_resource_idsas inxgboost_exampleSame call to
traceback.print_stackas inxgboost_exampleSome broken Ray imports:
needs to be changed to
and
needs to be changed to
Versions / Dependencies
platform.platform()Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.31The latest development version of
ray(410fe1b) was installed frommasterviapip install -ve .. All other python dependencies for the examples were installed throughpip, as needed. No GPU was used in testing the examples.Environment
Reproduction script
Here are the notebooks for the examples I ran. Some of them may vary slightly from the examples; most of the time I bumped up the number of cpus. In cases where the example couldn't run without being fixed, I made the necessary changes to make the example run, if possible.
ipynbs.zip
Issue Severity
Low: It annoys or frustrates me.
The text was updated successfully, but these errors were encountered: