Remove dependency to setoptconf

[sbrunner]

Description

Prototype to remove setuptconf, do you like it, should I go ahead with that?

Related Issue

fix #442

[Pierre-Sassoulas]

Removing deps is always nice.

Not sure if adding pydantic to the deps tree is advisable as we already have a lot of dependencies to consider (and there's also the user depencency in prod) but ultimately you choose :)

[sbrunner]

Pydentic is a very popular library to add types on configuration, it's true that it can interfere with user dependency, I will use a wider version range to avoid that, If you are against that, I can also use TypedDict, but this will not check the input this is only for type checking.

[sbrunner]

Currently, this is not working, but what do you think about using pydentic for data validation, if we use it, I plane to also use it to validate the profiles.

@carlio ?

[carlio]

@sbrunner I just discovered tyro and minterface which might help replacing setoptconf. Just ideas for you. Also clipstick and others linked from the tyro README

[sbrunner]

With clipstick I didn't succeed to create command like papaerless path1 path2, I will try tyro

[sbrunner]

With tyro I will get help like that

usage: prospector [-h] [OPTIONS]

The configuration for Prospector.

╭─ options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ -h, --help                                                                                                                                                 │
│     show this help message and exit                                                                                                                        │
│ --zero-exit, --no-zero-exit                                                                                                                                │
│     Prospector will exit with a code of 1 (one) if any messages are found. This makes automation easier; if there are any problems at all, the exit code   │
│     is non-zero. However this behavior is not always desirable, so if this flag is set, prospector will exit with a code of 0 if it ran successfully, and  │
│     non-zero if it failed to run. (default: False)                                                                                                         │
│ --autodetect, --no-autodetect                                                                                                                              │
│     Turn off auto-detection of frameworks and libraries used. By default, autodetection will be used. To specify manually, see the --uses option.          │
│     (default: True)                                                                                                                                        │
│ --uses [STR [STR ...]]                                                                                                                                     │
│     A list of one or more libraries or frameworks that the project uses. Possible values are: django, celery, flask. This will be autodetected by default, │
│     but if autodetection doesn't work, manually specify them using this flag. (default: )                                                                  │
│ --blending, --no-blending                                                                                                                                  │
│     Turn off blending of messages. Prospector will merge together messages from different tools if they represent the same error. Use this option to see   │
│     all unmerged messages. (default: True)                                                                                                                 │
│ --doc-warnings {None,True,False}                                                                                                                           │
│     Include warnings about documentation. (default: None)                                                                                                  │
│ --test-warnings {None,True,False}                                                                                                                          │
│     Also check test modules and packages. (default: None)                                                                                                  │
│ --no-style-warnings {None,True,False}                                                                                                                      │
│     Don't create any warnings about style. This disables the PEP8 tool and similar checks for formatting. (default: None)                                  │
│ --member-warnings {None,True,False}                                                                                                                        │
│     Attempt to warn when code tries to access an attribute of a class or member of a module which does not exist. This is disabled by default as it tends  │
│     to be quite inaccurate. (default: None)                                                                                                                │
│ --full-pep8 {None,True,False}                                                                                                                              │
│     Enables every PEP8 warning, so that all PEP8 style violation will be reported. (default: None)                                                         │
│ --max-line-length {None}|INT                                                                                                                               │
│     The maximum line length allowed. This will be set by the strictness if no value is explicitly specified (default: None)                                │
│ --messages-only, --no-messages-only                                                                                                                        │
│     Only output message information (don't output summary information about the checks) (default: False)                                                   │
│ --summary-only, --no-summary-only                                                                                                                          │
│     Only output summary information about the checks (don't output message information) (default: False)                                                   │
│ --quiet, --no-quiet                                                                                                                                        │
│     Run but do not output anything to stdout. Useful to suppress output in scripts without sending to a file (via -o) (default: False)                     │
│ --output-format {None}|{[STR [STR ...]]}                                                                                                                   │
│     The output format. Valid values are: {}. This will output to stdout by default, however a target file can be used instead by adding                    │
│     :path-to-output-file, eg, -o json:output.json (default: None)                                                                                          │
│ --absolute-paths, --no-absolute-paths                                                                                                                      │
│     Whether to output absolute paths when referencing files in messages. By default, paths will be relative to the project path (default: False)           │
│ --tools {None}|{[STR [STR ...]]}                                                                                                                           │
│     A list of tools to run. This lets you set exactly which tools to run. To add extra tools to the defaults, see --with-tool. (default: None)             │
│ --with-tools [STR [STR ...]]                                                                                                                               │
│     A list of tools to run in addition to the default tools. To specify all tools explicitly, use the --tool argument. Possible values are {}. (default: ) │
│ --without-tools [STR [STR ...]]                                                                                                                            │
│     A list of tools that should not be run. Useful to turn off only a single tool from the defaults. To specify all tools explicitly, use the --tool       │
│     argument. (default: )                                                                                                                                  │
│ --profiles [STR [STR ...]]                                                                                                                                 │
│     The list of profiles to load. A profile is a certain 'type' of behavior for prospector, and is represented by a YAML configuration file. Either a full │
│     path to the YAML file describing the profile must be provided, or it must be on the profile path (see --profile-path) (default: )                      │
│ --profile-path [STR [STR ...]]                                                                                                                             │
│     Additional paths to search for profile files. By default this is the path that prospector will check, and a directory called '.prospector' in the path │
│     that prospector will check. (default: )                                                                                                                │
│ --strictness {None,veryhigh,high,medium,low,verylow,none,from_profile}                                                                                     │
│     How strict the checker should be. This affects how harshly the checker will enforce coding guidelines. The default value is "medium", possible values  │
│     are "veryhigh", "high", "medium", "low" and "verylow". (default: None)                                                                                 │
│ --show-profile, --no-show-profile                                                                                                                          │
│     Include the computed profile in the summary. This will show what prospector has decided the overall profile is once all profiles have been combined    │
│     and inherited from. This will produce a large output in most cases so is only useful when trying to debug why prospector is not behaving like you      │
│     expect. (default: False)                                                                                                                               │
│ --no-external-config, --no-no-external-config                                                                                                              │
│     Determines how prospector should behave when configuration already exists for a tool. By default, prospector will use existing configuration. This     │
│     flag will cause prospector to ignore existing configuration and use its own settings for every tool. Note that prospector will always use its own      │
│     config for tools which do not have custom configuration. (default: False)                                                                              │
│ --legacy-tool-names, --no-legacy-tool-names                                                                                                                │
│     Output deprecated names for tools (pep8, pep257) instead of updated names (pycodestyle, pydocstyle) (default: False)                                   │
│ --pylint-config-file {None}|STR                                                                                                                            │
│     The path to a pylintrc file to use to configure pylint. Prospector will find .pylintrc files in the root of the project, but you can use this option   │
│     to specify manually where it is. (default: None)                                                                                                       │
│ --path {None}|STR                                                                                                                                          │
│     The path to a Python project to inspect. Defaults to PWD if not specified. Note: This command line argument is deprecated and will be removed in a     │
│     future update. Please use the positional PATH argument instead. (default: None)                                                                        │
│ --ignore-patterns [STR [STR ...]]                                                                                                                          │
│     A list of paths to ignore, as a list of regular expressions. Files and folders will be ignored if their full path contains any of these patterns.      │
│     (default: )                                                                                                                                            │
│ --ignore-paths [STR [STR ...]]                                                                                                                             │
│     A list of file or directory names to ignore. If the complete name matches any of the items in this list, the file or directory (and all                │
│     subdirectories) will be ignored. (default: )                                                                                                           │
│ --die-on-tool-error, --no-die-on-tool-error                                                                                                                │
│     If a tool fails to run, prospector will try to carry on. Use this flag to cause prospector to die and raise the exception the tool generated. Mostly   │
│     useful for development on prospector. (default: False)                                                                                                 │
│ --include-tool-stdout, --no-include-tool-stdout                                                                                                            │
│     There are various places where tools will output warnings to stdout/stderr, which breaks parsing of JSON output. Therefore while tols is running, this │
│     is suppressed. For developing, it is sometimes useful to see this. This flag will cause stdout/stderr from a tool to be shown as a normal message      │
│     amongst other warnings. See also --direct-tool-stdout (default: False)                                                                                 │
│ --direct-tool-stdout, --no-direct-tool-stdout                                                                                                              │
│     There are various places where tools will output warnings to stdout/stderr, which breaks parsing of JSON output. Therefore while tols is running, this │
│     is suppressed. For developing, it is sometimes useful to see this. This flag will cause stdout/stderr from a tool to be shown as a normal message      │
│     amongst other warnings. See also --direct-tool-stdout (default: False)                                                                                 │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

This is nice, but the boolean and list are not threaded as expected :-/

[sbrunner]

Shouldn't I continue to improve the couple Pydentic + argparse?

[carlio]

Shouldn't I continue to improve the couple Pydentic + argparse?

Use whichever works best, I don't mind :) I just saw those and thought they might help, but I guess not.

[carlio]

Tyro uses pydantic anyway though you might be able to patch the help formatter if that's the only concern

[sbrunner]

The issue with tyro is not the help it that he expected that we use, e.g.:
--profile profile1 ptofile2
instead of
--profile profile1 --profile ptofile2
This is a braking changes for, in my point of view, no added value.

[sbrunner]

For me, the current solution is the best I tested :-)

[sbrunner]

Do You think that I can merge this pull request, from now I don't see any better solution

[carlio]

I've been thinking about this a lot and I think that adding Pydantic would be a bad idea.

Prospector is installed on top of existing environments and has to be careful to make sure that any dependencies of prospector have a wide range so that it can be compatible with whatever constraints users have. This adds lots of legacy code and compatability code. Adding more stuff to prospector, especially something as popular as pydantic, will add more surface area for version conflicts for users. That leads to bug reports and so on and so on.

setoptconf is really good at what it does. The desire to remove it was because it's not maintained.

However, similarly to the poetry-semver use in requirements-detector, attempts to replace it with other things were a pain and actually, vendoring it was a good solution.

Therefore I'd suggest that instead of rewriting or replacing setoptconf, instead, simply copying the code from the temporary replacement into the main prospector repository/package.

It works already. Trying to replace it might mean breaking changes.

@Pierre-Sassoulas @sbrunner what do you think? I find it to be an easy and practical solution which would minimise conflicts for users.

[Pierre-Sassoulas]

I agree, this is especially important for a linter that need to be installed alongside the code depencencies (because of pylint that always will). Vendoring seems reasonable but I would look at reducing deps to the bare minimum too. (The dependency lock with pylint-django / requirements-detector / pylint / prospector was complicated to navigate for python 3.12 support).

[sbrunner]

For me, the problem of the current code is that he isn't typed, it's the real more value of adding Pydentic.
I also planned to introduce Pydentic model int the profile to also by typed.

Prospector is installed on top of existing environments and has to be careful to make sure that any dependencies of prospector have a wide range so that it can be compatible with whatever constraints users have. This adds lots of legacy code and compatability code. Adding more stuff to prospector, especially something as popular as pydantic, will add more surface area for version conflicts for users. That leads to bug reports and so on and so on.

I understand this, and it's a relay important point, but I think It will not be difficult to support a wide range of Pydentic versions.

But if you are not convinced, we can keep it like it and not be typed.

[carlio]

I've removed the ability for brand new accounts or non-contributors to review changes as the @Bjackson4837 account appears to be a bot - recently joined, no other contributions except reviewing this PR 3 times

[carlio]

@sbrunner The headache of having Django as a dependency of pylint-django or not makes me want to avoid adding extra things to prospector. It just leads to massive amounts of discussion and issues.

I don't think the value added from typing will make up from the hassle that would come from maintaining the fallout and issues raised by conflicting with user's own environments.

[Pierre-Sassoulas]

It's possible to have typing without pydantic, and pylint already depends on typing_extensions so we can have "fancy" typing even if we must support python 3.9 still. (I never used pydantic, so I don't know, maybe it's so much better that it would be worth the trouble)

[sbrunner]

@carlio OK, I understand that you don't want to renew a bad experience!

@Pierre-Sassoulas I don't understand what you have in mind?

For some other projects, I created two projects

• https://github.com/sbrunner/jsonschema-gentypes/
• https://github.com/sbrunner/jsonschema2md/

to be able from a JSON schema to generate types and docs by using pre-commit.
e.g.:

• https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/schema.json
• https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/configuration.py
• https://github.com/camptocamp/tilecloud-chain/blob/master/tilecloud_chain/CONFIG.md

Do you think that's a good solution to typed and validate the profile?

For the config we can try to use a typed dict be at first look it don't look to be evident.

[Pierre-Sassoulas]

I was thinking about typed dict yes. Not sure if I have the whole context though.