Model selection

[tatiana-pashkova]

I wasn't sure if I should put this as a separate discussion issue or add it to the list of topics for Friday, but decided to go with a separate issue because we already covered the topic today.
So, my questions are:

1. Do you do model selection for random effects only or for fixed effects as well? Today in class we saw only random effects but Gries (2021) shows model selection starting from random effects and then moving to fixed effects.
2. Is there an existing source on which values to pay attention to when doing model selection? We mentioned LR test, AIC and BIC (ordered from the least to the most conservative), but I don't think I managed to write down all the rules - the AICs should be 5+ units different to consider the models different, and what about BICs? And before we look at AIC and BIC, we should pay attention to the chi-square value and the degrees of freedom, checking that χ² < 2*(χ²-dof)? I'm pretty sure I got something confused, so I was wondering if we can either write down the order of steps (I have 2 models, which values do I check and how) or have a link where this process is already described.

[kliegl]

1. There is not really a need for model selection in the fixed effects when you are analyzing a factorial experiment -- because main effects and interactions are uncorrelated in a balanced design. Missing data destroys the balancing but as long as they are in the usual range, this will not make much of a difference. If you have very many factors (like in the mrk17 example) and no hypothesis about higher-order interactions, it is also defensible to remove interactions top down. There is a very convenient formula syntax for this. In the example: F*P*Q*lQ*lT is the full factorial. You can write this also as: (F+P+Q+lQ+lT)^5. To keep all but the five-factor interaction you write: (F+P+Q+lQ+lT)^4; to drop all four-factor and the five-factor interactions (P+Q+lQ+lT)^3. You always compare with LRT, AIC, BIC and stop when you start to lose substantial information. Important: You must include using RegressionFormulae to have this syntax available. This is a very good exercise with the mrk17 data. Please try it out using the m_prm as baseline and let's see whether we come to the same solution! Keep the RES constant!
   When you have covariates in the model, model selection always involves both random and fixed effects, because (1) if you have to decide on the order of the trend (usually linear, quadratic, or cubic; note: these are mutiplicative "interactions" of the variable with itself) and (2) because interactions of factors with covariates and interactions between covariates are usually correlated, often strongly so. You find examples of this strategy in all scripts associated with publications in the OSF repositories (e.g., in all papers related to physical fitness of children or in all papers with corpus analyses of eye movements about reading at PMR2 http://read.psych.uni-potsdam.de/). I can walk you through such a covariate-based model comparison today.
   In the covariate case, I think it is better to start with the fixed-effect part and perhaps a minimal RES. As this model is anti-conservative for fixed effects, it may keep terms that will have zero in their confidence interval. However, in principle, it can happen that including a VC or CP will cause the reverse (e.g., when a VC or CP pulls much variance out of the residual). Usually, VCs and CPs "compete" with fixed effects for the same variance. So adding them will increase the uncertainty (i.e., the confidence interval)who should get what. Model selection always carries a risk of missing an important source of variance. There is judgement involved; it is not an algorithm! Always think: Replication.

2. AIC and BIC is 5 units according to Burnham and Anderson (1998) https://link.springer.com/book/10.1007/978-1-4757-2917-7. Not everybody agrees with this. There could also be more recent publications; I have not checked in a long time. Suggestions? My personal and vague rules of thumb: Use LRT, χ² < 2*(χ²-dof), when you have an explicit strong hypothesis motivated by theory and the usual size of data , AIC when of theoretical relevance, BIC when in an exploratory mode, no particular expectation, trying to get the model in a stable range of support by the available data.