Question About Abstract State and Correct Elements

[jim850223]

Dear Authors,

Thank you for your excellent work on SYNAPSE and for providing the corresponding paper.

I have a question regarding Section 3.2 of the paper, where you mentioned that to abstract the state, "we set k to 3 and 5 for the previous and current observations, respectively." My question is related to how you handle the inclusion of correct elements in the abstract state, especially in the context of Mind2Web.

As stated in the Mind2Web dataset, "each step of the task is evaluated independently with the ground truth action history provided." If the correct element is not included in the previous abstract state, do you append (concatenate) the correct element to the previous abstract state to ensure it is available for the next step? Alternatively, if you don't append the correct element, does this mean that the agent still records the action as taken even if the correct element was not part of the abstract state?

I appreciate your insights and look forward to your response.

Thank you again for your contributions!

[ltzheng]

See here for the code for getting the top k observations.

[jim850223]

Thank you for the answering, I got it right now.
However, it raised another question:
Given that pos_candidates can potentially contain multiple valid candidates, the current approach of only comparing the first candidate may overlook valid matches if the predicted element is not the first pos_candidate. Is this problem concered in the system? And if it is, how do you solve it?

[ltzheng]

Can you provide an example? I think in Mind2Web there is only one correct element.

[jim850223]

In C.1 Evaluation of the appendix of Mind2Web, it says:

"One complication that arises during evaluation on real-world websites is that multiple elements on a webpage may induce the same effect. For instance, a button might house a text span within it, both of which, when clicked, yield identical results. To enhance the robustness of our evaluation, we employ heuristics to detect elements equivalent to the ground truth. We first examine the ancestors of the labeled element to identify potential higher-level elements acceptable for the current action. We employ a straightforward heuristic that locates the nearest clickable element to the ground truth, including itself. After identifying the top-level acceptable element, we include all its visible descendants that are located within its post-rendering bounding box as acceptable as well. Manual checking on 100 instances where the heuristic identifies a top-level element other than the ground truth confirms the validity of the approach. For both training and evaluation stages, all acceptable elements are considered positive."

[ltzheng]

But after we do state abstraction, the clean observations for LLMs only contain one positive element.