New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Testing StatefulReplayStrategy #327

Open

FFFiend wants to merge 45 commits into OpenAdaptAI:main from Owais-Enhancement-Org:stateful-test

Collaborator

FFFiend commented Jun 28, 2023

What kind of change does this PR introduce?
Adds a test file to examine the quality and accuracy of action event generation by GPT-4.

Summary
Simple pytest function for comparing expected vs actual values of action event keys. Code is currently in boilerplate stage.

Responds to #242, we would like to enhance as well as extend this strategy.

Checklist

My code follows the style guidelines of OpenAdapt
I have perfomed a self-review of my code
If applicable, I have added tests to prove my fix is functional/effective
I have linted my code locally prior to submission
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation (e.g. README.md, requirements.txt)
New and existing unit tests pass locally with my changes

How can your code be run and tested?
Not yet meant to be run. Will update with time.

FFFiend and others added 14 commits

June 23, 2023 01:24


          added layout replaystrat mixin (partially complete)

e4b7490


          fixed return value

3ea5cf1


          removed extra space

ed74db9


          removed unintended methods

b9453c8


          reordered imports

0d3000d


          ordered imports

009dd8d


          changed type contract of path list to str

f8f2c90


          Merge branch 'MLDSAI:main' into main

4e59ccc


          fixed formatting

d17e35c


          Merge branch 'MLDSAI:main' into main

bdd2ad5


          added boilerplate test code

e735c4d


          testing stateful output

53f7266


          added todo tag on main testing function

03f409c


          added generic output function

7c76dd6

Collaborator Author

FFFiend commented Jun 28, 2023 •

edited

Loading

Wrote a function to verify that, given a previously seen action and window event sequence, the model is able to generate the exact same action event sequence when given the old windowevent sequence as a parameter.

current TODOs:

Create minimal versions of each: Recording, WindowEvent, etc.. and obtain diffs


          fixed completion str

e844d68

FFFiend mentioned this pull request

[Bug]: Error while parsing completion str in get_next_action_event in StatefulReplayStrategy #329

Open

FFFiend added 7 commits

June 28, 2023 16:04


          Update test_stateful.py

173e856


          added no_diff (non general) test and completion method

e995fd2


          Merge branch 'OpenAdaptAI:main' into stateful-test

de40a9d


          added generalized test methods

bb151ae


          reverted string slice on prompt change

8c4ced8


          Update stateful.py

fffbb5e


          Update stateful.py

f7e7ab1

FFFiend marked this pull request as ready for review

June 30, 2023 23:21

Collaborator Author

FFFiend commented Jun 30, 2023

Added create event and action dict methods for synthetic input generation. Current TODO:

Write a couple tests to evaluate generation quality from GPT-4. Move to a GGML Open Source LLM when results are satisfactory.

FFFiend added 4 commits

June 30, 2023 19:28


          added default value for meta param in create_window_event method

4d6d570


          added default param for element state dict in create_action_event method


          Merge branch 'OpenAdaptAI:main' into stateful-test

d8b47cc


          added single and multi-click tests

56940f0

LaPetiteSouris mentioned this pull request

WIP feat(model_evaluation): Add script to evaluate models #420

Closed

7 tasks

OpenAdaptAI deleted a comment from cr-gpt bot

OpenAdaptAI deleted a comment from cr-gpt bot

OpenAdaptAI deleted a comment from cr-gpt bot

OpenAdaptAI deleted a comment from cr-gpt bot

abrichr reviewed

View reviewed changes

tests/openadapt/test_stateful.py Outdated

+                  mouse_button_name: str = None,
+                  mouse_pressed: bool = None,
+                  key_name: str = None,
+                  element_state: dict[Any, Any] = None,

Member

abrichr Jul 28, 2023

Should this be dict[str, Any]?

abrichr reviewed

View reviewed changes

tests/openadapt/test_stateful.py Outdated

+                      width=WIN_WIDTH,
+                      height=WIN_HEIGHT,
+                      window_id=WINDOW_ID,
+                      meta={},

Member

abrichr Jul 28, 2023

What do you think about removing meta={},?

abrichr reviewed

View reviewed changes

tests/openadapt/test_stateful.py Outdated

+                      width=WIN_WIDTH,
+                      height=WIN_HEIGHT,
+                      window_id=WINDOW_ID,
+                      meta={},

Member

abrichr Jul 28, 2023

What do you think about removing meta={},?

abrichr reviewed

View reviewed changes

tests/openadapt/test_stateful.py Outdated

+                      width=WIN_WIDTH,
+                      height=WIN_HEIGHT,
+                      window_id=WINDOW_ID,
+                      meta={},

Member

abrichr Jul 28, 2023

What do you think about removing meta={},?

abrichr reviewed

View reviewed changes

tests/openadapt/test_stateful.py Outdated

+                          mouse_y=REF_Y,
+                          mouse_button_name="left",
+                          mouse_pressed=True,
+                          element_state={},

Member

abrichr Jul 28, 2023

What do you think about removing element_state={},?

abrichr reviewed

View reviewed changes

tests/openadapt/test_stateful.py Outdated

+                          mouse_y=REF_Y + i,
+                          mouse_button_name="left",
+                          mouse_pressed=True,
+                          element_state={},

Member

abrichr Jul 28, 2023

What do you think about removing element_state={},?

abrichr reviewed

View reviewed changes

tests/openadapt/test_stateful.py Outdated

+                          mouse_y=REF_Y + i,
+                          mouse_button_name="left",
+                          mouse_pressed=True,
+                          element_state={},

Member

abrichr Jul 28, 2023

What do you think about removing element_state={},?

abrichr reviewed

View reviewed changes

tests/openadapt/test_stateful.py Outdated

+                      width=WIN_WIDTH,
+                      height=WIN_HEIGHT,
+                      window_id=WINDOW_ID,
+                      meta={},

Member

abrichr Jul 28, 2023

What do you think about removing meta={},?

abrichr reviewed

View reviewed changes

tests/openadapt/test_stateful.py Outdated

+                          mouse_y=NEW_Y,
+                          mouse_button_name="left",
+                          mouse_pressed=True,
+                          element_state={},

Member

abrichr Jul 28, 2023

What do you think about removing element_state={},?

abrichr reviewed

View reviewed changes

tests/openadapt/test_stateful.py Outdated

+                          mouse_y=NEW_Y + i,
+                          mouse_button_name="left",
+                          mouse_pressed=True,
+                          element_state={},

Member

abrichr Jul 28, 2023

What do you think about removing element_state={},?

abrichr reviewed

View reviewed changes

tests/openadapt/test_stateful.py Outdated

+                          mouse_y=NEW_Y + i,
+                          mouse_button_name="left",
+                          mouse_pressed=True,
+                          element_state={},

Member

abrichr Jul 28, 2023

What do you think about removing element_state={},?

abrichr reviewed

View reviewed changes

tests/openadapt/test_stateful.py Outdated

+                      MULTI_ACTION_WIN_WIDTH,
+                      MULTI_ACTION_WIN_HEIGHT,
+                      MULTI_ACTION_WINDOW_ID,
+                      {},

Member

abrichr Jul 28, 2023

What do you think about removing {},?

abrichr reviewed

View reviewed changes

tests/openadapt/test_stateful.py Outdated

+                      MULTI_ACTION_WIN_WIDTH,
+                      MULTI_ACTION_WIN_HEIGHT,
+                      MULTI_ACTION_WINDOW_ID,
+                      {},

Member

abrichr Jul 28, 2023

What do you think about removing {},?

FFFiend mentioned this pull request

Exploring Minimizing Window State Content while Retaining Maximum Info #442

Open

LaPetiteSouris mentioned this pull request

feat(evaluators): add evaluator model to evaluate a model #444

Open

7 tasks

LaPetiteSouris reviewed

View reviewed changes

tests/openadapt/test_stateful.py

+                  "refuse. Copy the given format exactly. Your response should be "
+                  "valid Python3 code. Do not respond with any other text. "
+              )

Contributor

LaPetiteSouris Jul 29, 2023

@FFFiend to better organize the code and to enforce strict validation on action and window, as well as because I need these pieces of code to generate generic/simple action for model evaluation, I picked these parts and move to a dedicated model as in my PR #444

Contributor

LaPetiteSouris Jul 29, 2023 •

edited

Loading

Thanks for the idea to use single action for evaluation. This is truly great.

Member

abrichr Jul 31, 2023

@FFFiend what do you think about moving this to a .j2 file?

abrichr reviewed

View reviewed changes

tests/openadapt/test_stateful.py

+                  test_action_dict = gpt_completion(
+                      reference_window_dict, reference_action_dicts, active_window_dict
+                  )
+                  test_dict = eval(

Member

abrichr Jul 30, 2023

What do you think about using json.loads instead of eval?

FFFiend mentioned this pull request

FineTuning #453

Draft

7 tasks

FFFiend and others added 6 commits

August 8, 2023 00:26


          Merge branch 'main' into stateful-test

71c940d


          removed meta and element_state empty dicts

6f8ccf8


          fixed params

029ef42


          renamed func

64be692


          fixed logger output

5c89ed8


          added blank line

6e2601a

Collaborator Author

FFFiend commented Aug 7, 2023

So one of the tests is failing now apparently, when it previously wasn't. The model is outputting more events than necessary, see here:

expected_action_dict=[{'name': 'click', 'mouse_x': 138, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 139, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 90, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 139, 'mouse_y': 90, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 140, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 91, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 140, 'mouse_y': 91, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 141, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 92, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 141, 'mouse_y': 92, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 142, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 93, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 142, 'mouse_y': 93, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 143, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 94, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 143, 'mouse_y': 94, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 144, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 95, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 144, 'mouse_y': 95, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 145, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 96, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 145, 'mouse_y': 96, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 146, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 97, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 146, 'mouse_y': 97, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 147, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 98, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 147, 'mouse_y': 98, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 148, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 99, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 148, 'mouse_y': 99, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 149, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 100, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 149, 'mouse_y': 100, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}], len(expected_action_dict)=36

and

test_dict=[{'name': 'click', 'mouse_x': 138, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 139, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 90, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 139, 'mouse_y': 90, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 140, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 91, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 140, 'mouse_y': 91, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 141, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 92, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 141, 'mouse_y': 92, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 142, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 93, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 142, 'mouse_y': 93, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 143, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 94, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 143, 'mouse_y': 94, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 144, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 95, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 144, 'mouse_y': 95, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 145, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 96, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 145, 'mouse_y': 96, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 146, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 97, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 146, 'mouse_y': 97, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 147, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 98, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 147, 'mouse_y': 98, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 148, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 99, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 148, 'mouse_y': 99, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 149, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 100, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 149, 'mouse_y': 100, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 150, 'mouse_y': 89, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 138, 'mouse_y': 101, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}, {'name': 'click', 'mouse_x': 150, 'mouse_y': 101, 'mouse_button_name': 'left', 'mouse_pressed': True, 'element_state': {}}], len(test_dict)=39

Clearly test_dict has 3 more actions than expected_dict

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet