Rework plugin config to be file-based

[erik-megarad]

Background

Currently the way to configure which plugins to use is to change ALLOWLISTED_PLUGINS / DENYLISTED_PLUGINS in .env.
The only way to configure a plugin is through the same .env file.

This change adds a plugin config file, plugins_config.yaml which allows plugins to be enabled and disabled (replacing the old allowlist/denylist). It also allows arbitrary config per plugin, such as API keys or model names or whatever that plugin needs. Now, plugins don't need their own env variables.

I made significant effort for this to be backwards-compatible. On start-up, if the plugins settings file isn't found, it will create a config file derived on the old ALLOWLISTED_PLUGINS / DENYLISTED_PLUGINS env variables.

Configuration for old plugins won't need to be changed immediately, they can still use whatever env variables they wanted. The plugin author can decide whether to use env variables or the plugins config file. We should encourage using the plugins config file, though.

Changes

• Remove use of allowlist / denylist for plugins, updating those checks to use the plugin config enabled value
• Add plugin config classes
• Load plugin config from file, creating one if necessary

Documentation

• In-code comments
• Updated options.md with new/removed vars
• Updated plugins.md with the config file details

Test Plan

• Rewrote a lot of tests
• Tested manually

PR Quality Checklist

• My pull request is atomic and focuses on a single change.
• I have thoroughly tested my changes with multiple different prompts.
• I have considered potential risks and mitigations for my changes.
• I have documented my changes clearly and comprehensively.
• I have not snuck in any "extra" small tweaks changes.
• I have run the following commands against my code to ensure it passes our linters:

  black .
  isort .
  mypy
  autoflake --remove-all-unused-imports --recursive --ignore-init-module-imports --ignore-pass-after-docstring autogpt tests --in-place

[github-actions]

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

[netlify]

✅ Deploy Preview for auto-gpt-docs ready!

Name	Link
🔨 Latest commit	0da03cf
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs/deploys/64891aac59747e00082cbfca
😎 Deploy Preview	https://deploy-preview-4673--auto-gpt-docs.netlify.app/
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[github-actions]

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

[github-actions]

This PR exceeds the recommended size of 500 lines. Please make sure you are NOT addressing multiple issues with one PR.

[github-actions]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[Auto-GPT-Bot]

You changed AutoGPT's behaviour. The cassettes have been updated and will be merged to the submodule when this Pull Request gets merged.

[codecov]

Codecov Report

Patch coverage: 85.54% and project coverage change: +0.33 🎉

Comparison is base (3df8c1b) 70.43% compared to head (0da03cf) 70.77%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4673      +/-   ##
==========================================
+ Coverage   70.43%   70.77%   +0.33%     
==========================================
  Files          70       72       +2     
  Lines        3437     3504      +67     
  Branches      547      556       +9     
==========================================
+ Hits         2421     2480      +59     
- Misses        848      856       +8     
  Partials      168      168              

Impacted Files	Coverage Δ
autogpt/plugins/plugins_config.py	78.84% <78.84%> (ø)
autogpt/plugins/plugin_config.py	87.50% <87.50%> (ø)
autogpt/config/config.py	72.04% <100.00%> (+1.46%)	⬆️
autogpt/plugins/__init__.py	80.66% <100.00%> (ø)

... and 3 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.