Continual Editing

[Wonder1905]

Hi, thanks again for this excellent framework and papers, In you paper youve talked about "Continual Editing", is there a piece of code that handles this experiments? if not do you have any idea how to implement it using the current code, with minimum amount of changes?
Please share your thoughts.

Thank again!

[pengzju]

Thank you for your attention, if you would like to edit sequentially. You can set keep_original_weight to False so that the parameters will not be rolled back after each edit.

• But even so, the evaluation is still based on a single edit before proceeding to the next step of editing.
• If you want to evaluate after editing 500 times for example, you can move the code here outside the loop

[Wonder1905]

Thanks for your quick reply, about this line:
"But even so, the evaluation is still based on a single edit before proceeding to the next step of editing"
How did you evaluate you continual editing on the paper? after each step or at the end? just so that we will be aligned

[pengzju]

In our paper, the sequential editing used is to edit one item and test the performance of the other. For example, after editing the second item, use the current checkpoint to test the performance of the second item, and the cycle repeats. "This aligns with our original paper." The pseudocode looks like this:

for i, request in tqdm(enumerate(requests)):
    edited_model, weights_copy = edit_func(model, tok, request, hparams)
    edit_evaluation(all_metrics, request, edited_model, hparams, tok, i)

    with torch.no_grad():
        for k, v in weights_copy.items():
            nethook.get_parameter(model, k)[...] = v.to(f'cuda:{hparams.device}')

However, a more scientific test method is to edit N items and use this checkpoint to test the performance of 1-N. You can appreciate the difference. If you go this route, the pseudocode looks like this:

for i, request in tqdm(enumerate(requests)):
    edited_model, weights_copy = edit_func(model, tok, request, hparams)
for i, request in tqdm(enumerate(requests)):
    edit_evaluation(all_metrics, request, edited_model, hparams, tok, i)

[pengzju]

If you have any questions, please feel free to leave a message. Thank you again for your attention. 😊

[zxlzr]

Hi, do you have any further issues?

[Wonder1905]

No, thanks for the detailed comment!