problem about the result

[Yuan0320]

I used paper_defaults.jsonnet to reproduce the result of 0.40 in the paper, but with another configuration defaults.jsonnet ran for many times, the maximum reached 0.41 instead of 0.47. I am a little confused, can you help me approach the problem, thank you!

[benbogin]

Hi, training the model as described should give 47-48 for the sql_match metric, did you run it exactly as described?
Can you list the actions you have done to train, and share the final metrics file you get?

[Yuan0320]

metrics.json.zip
Hi! The above is the best result after I run defaults.jsonnet many times.
I did it as you described, first install and configure, and then use the AllenNLP command to train. It can run successfully and I can used paper_defaults.jsonnet to get the result 0.40 in the paper.

[Yuan0320]

Hi!
"train_data_path": dataset_path + "train_spider.json",
And does this mean that we only used train_spider.json in the official Spider training dataset, but not train_others.json?(there are two official Spider training dataset, train_spider.json and train_others.json)
Thanks!

[benbogin]

Sorry, I am unsure of why you get different results. When the repository was first out, starting from a clean environment and running the default configuration, I got 47-48 percent accuracy, and other reported the same. Perhaps there were some changes to packages since them? Are you using the same versions of packages?

One thing you could do is to try the follow-up repository - https://github.com/benbogin/spider-schema-gnn-global
It is based on the same code, with a few changes. Running it without the re-ranker should get you around 49.3 accuracy.

Regarding your last question - yes, we only trained on train_spider.json (the rest of the dataset did not improve our results, although you can re-test this if you'd like)

[Yuan0320]

I checked the version of the packages and it is the same as yours. How many times did you run it to achieve 0.47? I see issue #5 and its result is only 0.43.Thanks!

[benbogin]

I ran it once or twice to reproduce the results from pull request #13, and got about the same results, but since then I've worked mostly with the follow-up repository. Let me know if you have trouble reproducing the results there as well

[Yuan0320]

Oh,sorry! I can reproduce the 0.47 result now, which is my problem.Thanks!

[benbogin]

Glad to hear. What was the issue?

[Yuan0320]

So far I don't know where the problem is. I re-downloaded your project and ran it, and found that a result of 47 was successfully obtained. It's my problem, sorry!