add unique_dir option to config.project (for script tools/train.py)

[jpcbertoldo]

Description

It's a tiny functionality that makes sure that the script tools/train.py will always create a new directory to dump the results in.

If config.project.unique_dir == True, then project_path = project_path / datetime.now().strftime("%Y-%m-%d_%H-%M-%S").

It should not break the current script because it defaults to False even if the option is not present in the .yaml, falling back to the previous behavior.

Changes

• Bug fix (non-breaking change which fixes an issue)
• Refactor (non-breaking change which refactors the code base)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Checklist

• My code follows the pre-commit style and check guidelines of this project.
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing tests pass locally with my changes

[samet-akcay]

Would users like this minor feature? If yes, it might be an idea to document this somewhere so it would not be a hidden gem

[samet-akcay]

I guess this change would also be consistent with the behaviour in the new CLI.

[jpcbertoldo]

Would users like this minor feature? If yes, it might be an idea to document this somewhere so it would not be a hidden gem

Of course :)

I'm just writting sketchy (i.e. only the necessary code) PRs first to get some approval before writting documentation and tests.

@samet-akcay does it look good to proceed? (in particular, do you think this needs unittest?)

[djdameln]

It would be better to place this after line 158, so that the category folder comes before the unique folder.

[jpcbertoldo]

Ah yess. Thanks.

Btw, another idea : would it make sense to name it (in addition to the datetime) a hash of the config file?

Like this you know that two folders were run with exactly the same options.

[djdameln]

As long as it doesn't lead to excessively long and unreadable folder names I'm fine with adding it. We could also store a copy of the config in the folder to make it easier to find back a specific run.

[jpcbertoldo]

So, a few updates:

1. I am computing a hash of the config object

The hash does not consider all the keys in the config in order to ignore things that only concern the run itself (like directories) and should not influence the outcomes of the optimization ("math-related stuff").

Its digest 8-byte/16-charachter long.

2. the hash is saved in a text file in the root of the project dir

3. category will be taken in consideration as you asked @djdameln

4. i used the opportunity to save the config file in the project dir as well (it makes debugging easier), keeping the original and normalized version.

[ashwinvaidya17]

Not sure if I am in favour of hashing as it adds more complexity to the code. I would suggest users to use comet or wandb as they automatically upload the configuration. This makes comparison and filtering easier.

[jpcbertoldo]

It was supposed to be a tiny easy PR to pass quickly 😬. I do use wandb and i see what you mean but i see some added value in adding the hash but i don't have a strong opinion on it so Im good with whatever is decided.

@samet-akcay @djdameln do you have hard opinions on this?

[samet-akcay]

@jpcbertoldo, my concern is the verbosity that will be added to the name of the directory, which already increases with the addition of date-time. In addition, this will be obsolete when moved to the new cli and deprecate this one. Perhaps, we could only stick to the addition of date and time.

[jpcbertoldo]

Done : )

Where should the docs be updated?

[samet-akcay]

Done : )

Where should the docs be updated?

Looks like the documentation does not have a specific section for the configuration. Not sure how exactly to document this.

Perhaps, documenting this could be skipped this time as this CLI will be deprecated

[djdameln]

Thanks!

[jpcbertoldo]

I think the failling check here is not related to my changes.
I only changes anomalib/config/config.py, cf screenshot.

[samet-akcay]

@jpcbertoldo, this looks like an issue from previous commits. Have you rebased your branch?

[jpcbertoldo]

i couldnt rebase ☝️ so i updated with normal merge

[jpcbertoldo]

Same unrelated problem with coverage report.

I think everything else is fine.

Can we merge?