Skip to content
This repository has been archived by the owner on Sep 18, 2024. It is now read-only.

Make pylint happy #1649

Merged
merged 10 commits into from
Oct 30, 2019
Merged

Make pylint happy #1649

merged 10 commits into from
Oct 30, 2019

Conversation

liuzhe-lz
Copy link
Contributor

@liuzhe-lz liuzhe-lz commented Oct 23, 2019

PPO tuner is not touched because I can't understand reported functions. @QuanluZhang
I have not installed all tuner dependencies and disabled import-error locally, so it may be incomplete.

@@ -191,22 +191,23 @@ def _add_index(in_x, parameter):
return {NodeType.INDEX: pos, NodeType.VALUE: item}
else:
return parameter
return None # FIXME: what to return here?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xuehui1991 Please check this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not change the logic in this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay. Though it's still smelly to me.
I'll change FIXME into a note.

@@ -576,6 +570,7 @@ def to_real_keras_layer(layer):
return layers.Flatten()
if is_layer(layer, "GlobalAveragePooling"):
return layers.GlobalAveragePooling2D()
return None # FIXME: should here return or raise?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xuehui1991 And this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do not change the logic.

@@ -608,6 +603,7 @@ def is_layer(layer, layer_type):
return isinstance(layer, (StubFlatten,))
elif layer_type == "GlobalAveragePooling":
return isinstance(layer, StubGlobalPooling)
return None # FIXME: return or raise?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And this

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return None. do not change the logic, thank you.

@liuzhe-lz liuzhe-lz requested a review from leckie-chn October 24, 2019 07:00
@@ -46,21 +45,20 @@ def trial_end(self, trial_job_id, success):
trial_job_id: identifier of the trial (str).
success: True if the trial successfully completed; False if failed or terminated.
"""
pass
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@leckie-chn Is this recommended style?
This will cause W0107: Unnecessary pass statement (unnecessary-pass)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently in our pylintrc file W1017 is not enabled. Are you sure you are using the right version of pylintrc file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm using the rc file included in this PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@leckie-chn
Copy link
Contributor

@liuzhe-lz Just to clarify, which modules is this PR fixing?

@leckie-chn
Copy link
Contributor

With the output on nni SDK with current settings:

python3 -m pylint src/sdk/pynni/nni
************* Module nni.bohb_advisor.config_generator
src/sdk/pynni/nni/bohb_advisor/config_generator.py:258:12: E1137: 'return_array' does not support item assignment (unsupported-assignment-operation)************* Module nni.compression.tensorflow.builtin_quantizers
src/sdk/pynni/nni/compression/tensorflow/builtin_quantizers.py:30:4: W0235: Useless super delegation in method '__init__' (useless-super-delegation)src/sdk/pynni/nni/compression/tensorflow/builtin_quantizers.py:55:4: W0235: Useless super delegation in method '__init__' (useless-super-delegation)************* Module nni.compression.tensorflow
src/sdk/pynni/nni/compression/tensorflow/__init__.py:2:0: W0401: Wildcard import builtin_pruners (wildcard-import)
src/sdk/pynni/nni/compression/tensorflow/__init__.py:3:0: W0401: Wildcard import builtin_quantizers (wildcard-import)
************* Module nni.compression.torch.builtin_quantizers
src/sdk/pynni/nni/compression/torch/builtin_quantizers.py:30:4: W0235: Useless super delegation in method '__init__' (useless-super-delegation)
src/sdk/pynni/nni/compression/torch/builtin_quantizers.py:56:4: W0235: Useless super delegation in method '__init__' (useless-super-delegation)
************* Module nni.compression.torch
src/sdk/pynni/nni/compression/torch/__init__.py:2:0: W0401: Wildcard import builtin_pruners (wildcard-import)
src/sdk/pynni/nni/compression/torch/__init__.py:3:0: W0401: Wildcard import builtin_quantizers (wildcard-import)
************* Module nni.curvefitting_assessor.curvefunctions
src/sdk/pynni/nni/curvefitting_assessor/curvefunctions.py:103:4: E1111: Assigning to function call which doesn't return (assignment-from-no-return)
************* Module nni.curvefitting_assessor.model_factory
src/sdk/pynni/nni/curvefitting_assessor/model_factory.py:311:12: E1111: Assigning to function call which doesn't return (assignment-from-no-return)
************* Module nni.evolution_tuner.test_evolution_tuner
src/sdk/pynni/nni/evolution_tuner/test_evolution_tuner.py:54:32: C0326: Exactly one space required after comma
                "_type":"choice","_value":["Adam", "SGD"]
                                ^ (bad-whitespace)
src/sdk/pynni/nni/evolution_tuner/test_evolution_tuner.py:26:0: C0411: standard import "from unittest import TestCase, main" should be placed before "import numpy as np" (wrong-import-order)
************* Module nni.gp_tuner.target_space
src/sdk/pynni/nni/gp_tuner/target_space.py:142:61: E1102: self.dim is not callable (not-callable)
************* Module nni.hyperopt_tuner.hyperopt_tuner
src/sdk/pynni/nni/hyperopt_tuner/hyperopt_tuner.py:59:20: E1111: Assigning to function call which doesn't return (assignment-from-no-return)
************* Module nni.networkmorphism_tuner.graph
src/sdk/pynni/nni/networkmorphism_tuner/graph.py:801:8: E0401: Unable to import 'keras' (import-error)
src/sdk/pynni/nni/networkmorphism_tuner/graph.py:905:4: E0401: Unable to import 'onnx' (import-error)
************* Module nni.networkmorphism_tuner.layers
src/sdk/pynni/nni/networkmorphism_tuner/layers.py:33:4: W0235: Useless super delegation in method '__init__' (useless-super-delegation)
src/sdk/pynni/nni/networkmorphism_tuner/layers.py:412:4: W0235: Useless super delegation in method '__init__' (useless-super-delegation)
src/sdk/pynni/nni/networkmorphism_tuner/layers.py:471:4: W0235: Useless super delegation in method '__init__' (useless-super-delegation)
src/sdk/pynni/nni/networkmorphism_tuner/layers.py:528:4: E0401: Unable to import 'keras' (import-error)
src/sdk/pynni/nni/networkmorphism_tuner/layers.py:544:4: E0401: Unable to import 'keras' (import-error)
************* Module nni.networkmorphism_tuner.test_networkmorphism_tuner
src/sdk/pynni/nni/networkmorphism_tuner/test_networkmorphism_tuner.py:195:27: C0326: Exactly one space required after comma
        tuner.add_model(0.8,  0)
                           ^ (bad-whitespace)
src/sdk/pynni/nni/networkmorphism_tuner/test_networkmorphism_tuner.py:196:27: C0326: Exactly one space required after comma
        tuner.add_model(0.9,  1)
                           ^ (bad-whitespace)
************* Module nni.ppo_tuner.distri
src/sdk/pynni/nni/ppo_tuner/distri.py:153:15: E1120: No value for argument 'mask_npinf' in constructor call (no-value-for-parameter)
src/sdk/pynni/nni/ppo_tuner/distri.py:153:15: E1120: No value for argument 'nsteps' in constructor call (no-value-for-parameter)
src/sdk/pynni/nni/ppo_tuner/distri.py:153:15: E1120: No value for argument 'size' in constructor call (no-value-for-parameter)
src/sdk/pynni/nni/ppo_tuner/distri.py:153:15: E1120: No value for argument 'is_act_model' in constructor call (no-value-for-parameter)
************* Module nni.ppo_tuner.ppo_tuner
src/sdk/pynni/nni/ppo_tuner/ppo_tuner.py:540:0: C0330: Wrong continued indentation (remove 2 spaces).
                                        mb_dones, last_values, self.inf_batch_size)
                                      | ^ (bad-continuation)
src/sdk/pynni/nni/ppo_tuner/ppo_tuner.py:239:12: E1137: 'mb_advs' does not support item assignment (unsupported-assignment-operation)
src/sdk/pynni/nni/ppo_tuner/ppo_tuner.py:584:27: W1202: Use % formatting in logging functions and pass the % parameters as arguments (logging-format-interpolation)
src/sdk/pynni/nni/ppo_tuner/ppo_tuner.py:585:80: C1801: Do not use `len(SEQUENCE)` to determine if a sequence is empty (len-as-condition)
src/sdk/pynni/nni/ppo_tuner/ppo_tuner.py:25:0: W0611: Unused import os (unused-import)
src/sdk/pynni/nni/ppo_tuner/ppo_tuner.py:29:0: W0611: Unused import json_tricks (unused-import)
************* Module nni.ppo_tuner.util
src/sdk/pynni/nni/ppo_tuner/util.py:59:4: W0612: Unused variable 'nbatch' (unused-variable)
************* Module nni.smac_tuner.smac_tuner
src/sdk/pynni/nni/smac_tuner/smac_tuner.py:282:20: E1111: Assigning to function call which doesn't return (assignment-from-no-return)

------------------------------------------------------------------
Your code has been rated at 9.85/10 (previous run: 9.39/10, +0.46)

@liuzhe-lz
Copy link
Contributor Author

liuzhe-lz commented Oct 28, 2019

@leckie-chn For your pylintrc output:

  • PPO tuner is not covered in this PR
  • Unit tests are not required to pass pylint (though recommended maybe)
  • I disabled useless-super-delegation until we find a way to write derived constructor docstring
  • unsupported-assignment-operation and assignment-from-no-return are because numpy not marked as generated-members in rc file
  • wildcard-import is disabled for __init__.py in latest pylint, see here
  • The last problem in GP tuner is strangely ignored by latest pylint (fixed below)

@leckie-chn
Copy link
Contributor

leckie-chn commented Oct 28, 2019

@leckie-chn For your pylintrc output:

  • Unit tests should not be required to pass pylint

@liuzhe-lz

nni/pylintrc

Line 45 in 041c3f0

ignore-patterns=test.py
Here we have a rule for disabling check for Unit tests.

@liuzhe-lz
Copy link
Contributor Author

liuzhe-lz commented Oct 28, 2019

@leckie-chn Suggest use ignore-patterns=test.*.py (.* is wildcard).

@leckie-chn
Copy link
Contributor

@leckie-chn For your pylintrc output:

  • PPO tuner is not covered in this PR
  • Unit tests are not required to pass pylint (though recommended maybe)
  • I disabled useless-super-delegation until we find a way to write derived constructor docstring
  • unsupported-assignment-operation and assignment-from-no-return are because numpy not marked as generated-members in rc file
  • wildcard-import is disabled for __init__.py in latest pylint, see here
  • The last problem in GP tuner is strangely ignored by latest pylint (fixed below)

output w/ pylint 2.4.3, removing all UTs and PPO modules:

************* Module nni.bohb_advisor.config_generator
src/sdk/pynni/nni/bohb_advisor/config_generator.py:312:69: E1136: Value 'train_configs.shape' is unsubscriptable (unsubscriptable-object)
src/sdk/pynni/nni/bohb_advisor/config_generator.py:313:72: E1136: Value 'train_configs.shape' is unsubscriptable (unsubscriptable-object)
src/sdk/pynni/nni/bohb_advisor/config_generator.py:321:11: E1136: Value 'train_data_good.shape' is unsubscriptable (unsubscriptable-object)
src/sdk/pynni/nni/bohb_advisor/config_generator.py:321:39: E1136: Value 'train_data_good.shape' is unsubscriptable (unsubscriptable-object)
src/sdk/pynni/nni/bohb_advisor/config_generator.py:323:11: E1136: Value 'train_data_bad.shape' is unsubscriptable (unsubscriptable-object)
src/sdk/pynni/nni/bohb_advisor/config_generator.py:323:38: E1136: Value 'train_data_bad.shape' is unsubscriptable (unsubscriptable-object)
************* Module nni.compression.tensorflow.builtin_quantizers
src/sdk/pynni/nni/compression/tensorflow/builtin_quantizers.py:30:4: W0235: Useless super delegation in method '__init__' (useless-super-delegation)src/sdk/pynni/nni/compression/tensorflow/builtin_quantizers.py:55:4: W0235: Useless super delegation in method '__init__' (useless-super-delegation)************* Module nni.compression.torch.builtin_quantizers
src/sdk/pynni/nni/compression/torch/builtin_quantizers.py:30:4: W0235: Useless super delegation in method '__init__' (useless-super-delegation)     src/sdk/pynni/nni/compression/torch/builtin_quantizers.py:56:4: W0235: Useless super delegation in method '__init__' (useless-super-delegation)     ************* Module nni.metis_tuner.metis_tuner
src/sdk/pynni/nni/metis_tuner/metis_tuner.py:358:43: E1133: Non-iterable value results_outliers is used in an iterating context (not-an-iterable)   ************* Module nni.networkmorphism_tuner.graph
src/sdk/pynni/nni/networkmorphism_tuner/graph.py:801:8: E0401: Unable to import 'keras' (import-error)
src/sdk/pynni/nni/networkmorphism_tuner/graph.py:905:4: E0401: Unable to import 'onnx' (import-error)
************* Module nni.networkmorphism_tuner.layers
src/sdk/pynni/nni/networkmorphism_tuner/layers.py:33:4: W0235: Useless super delegation in method '__init__' (useless-super-delegation)
src/sdk/pynni/nni/networkmorphism_tuner/layers.py:412:4: W0235: Useless super delegation in method '__init__' (useless-super-delegation)
src/sdk/pynni/nni/networkmorphism_tuner/layers.py:471:4: W0235: Useless super delegation in method '__init__' (useless-super-delegation)
src/sdk/pynni/nni/networkmorphism_tuner/layers.py:528:4: E0401: Unable to import 'keras' (import-error)
src/sdk/pynni/nni/networkmorphism_tuner/layers.py:544:4: E0401: Unable to import 'keras' (import-error)

@leckie-chn leckie-chn mentioned this pull request Oct 29, 2019
@liuzhe-lz
Copy link
Contributor Author

liuzhe-lz commented Oct 29, 2019

@leckie-chn

  • See our old pylintrc file. generated-members=numpy can stop pylint from analyzing numpy types.
  • I believe it's a good practice to refine parent class' docstring. And I don't want to make good things verbose. If a class has a useless __init__ without docstring, it will be later detected by missing-docstring.
  • It looks like an iterable to me. results_outliers is returned by gp_oulier_detection.outlierDetection_threaded, which is defined in nni/metis_tuner/Regression_GP/OutlierDetection.py. The function returns either a list or None, and it's checked is not None before reported line. So this should be a pylint bug.

@liuzhe-lz liuzhe-lz merged commit 7c4b8c0 into microsoft:master Oct 30, 2019
@liuzhe-lz liuzhe-lz deleted the pylint branch November 2, 2019 09:00
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants