A question for the grid search

[ChairManMeow-SY]

Hi Zhengzhong,

I noticed in your evaluate script (demo_eval_BVQA_feats_all_combined.py), a grid search is used to obtain a best [gamma, c] for each iteration.

This may bring a problem: for each iteration (the 8-2 division) the best gamma c will be different.

If we have 100 params pairs, and 100 iterations. Then we should have a 100x100 searching matrix, naming K. Usually I take the median for each row (i.e. each iteration with a fixed param pair), then the max value to select the best param pair.

It takes the np.max(np.median(K, 1)) as the final result, here the np denotes numpy.

The current script essentially takes the np.median(np.max(K, 0)).

I don't think these two ways will make the same results ---- the latter one may offer a better score but it can not tell us what's the best params in the whole dataset.

Please correct me if I misunderstand anything....

[ChairManMeow-SY]

Read the code again... sorry I see the param k-fold procedure.

It's absolutely right to select the param with y_param_valid set.

---

Finally just curious, which is better, np.max(np.median(K,1)) or the processing in your script? 😂

[vztu]

It depends on the model and dataset. You can just try and see which one is better 😂