start fixtures, types, challenge creation, mock run (stable

[SilenNaihin]

Creating all of the starter code other than the server boilerplate.

• in agbenchmark/define_task_types.py define challenge creation schema
• view it in action in agbenchmark/challenges/retrieval/r1_test.py
• follow the chain of inheritance to the challenge specific class at agbenchmark/retrieval/Retrieval.py
• all the way down to the base challenge class agbenchmark/Challenge.py which all challenges inherit
• agbenchmark/start_benchmark for click
• agbenchmark/conftest for fixtures (including getting the config file, getting workspace path and cleaning workspace after test, and mock test/server call)

[ScarletPan]

Hi @SilenNaihin , it's not recommended to include challenge data within the Python file, as this can make it difficult to scale the benchmark and allow others to contribute challenges using a simple JSON format. The data folder, samples for each challenges and README.md should be kept, as they provide guidance for each type of challenge.

[ScarletPan]

The config.json file and mocks folder should also be excluded from the agbenchmark folder, as it will be modified every time you run the program.

Error message shows up [Error 2] No such file or directory: /xxx/agbenchmark/mocks/workspace/file_to_check.txt

[AbTrax]

Other than what pan said, I can't see anything else.

[SilenNaihin]

Thank for the comments @ScarletPan @AbTrax, writing out for reference

not recommended to include challenge data within the Python file

good point, I moved it to a r1_data.json file that lives within a directory for that specific challenge (ease of access per challenge)

data folder, samples for each challenges and readme should be kept

this has become the mocks folder now. added back the read me to the challenges folder, good point there

config.json file and mocks should also be excluded from the agbenchmark folder

typically yes, but since this is a pip installable package in order to interact with it through poetry these must be in the agbenchmark folder - that is functionally the root of the project

error

did you follow the instructions outlined in readme to run it?

1. clone the repo auto-gpt-benchmarks
2. pip install poetry
3. poetry shell
4. poetry install
5. agbenchmark start