October 12, 2012
Dear Methodology Center,
I was recently performing a latent class analysis (LCA) and, as is fairly common, I had trouble interpreting the fit statistics. The BIC indicated a 3-class model; the AIC indicated a 5-class model. How do I interpret model fit when the penalized-likelihood information criteria do not point me to a single model?
Stymied by Fit Statistics
Many LCA users have questions like this. As you stated, AIC and BIC are both penalized-likelihood criteria. They are often used for comparing non-nested models, which ordinary statistical tests cannot do, and help with the fundamental question, “How many classes should there be in the model I select?”
Despite the differences in their theoretical derivation and motivation, their primary difference in practice is the size of the penalty; BIC penalizes model complexity more heavily. The only way they should disagree is when AIC indicates a model with more latent classes than BIC. In practice, AIC always has a chance of choosing too big a model, regardless of n. BIC has very little chance of choosing many classes if n is sufficient, but it has a larger chance than AIC, for any given n, of choosing too few classes.
So what’s the bottom line? In general, it might be best to use AIC and BIC together in model selection. For example, in selecting the number of latent classes in a model, if BIC points to a three-class model and AIC points to a five-class model, it makes sense to select from models with 3, 4, and 5 latent classes. AIC is better in situations when not identifying all of the classes would be considered more misleading than choosing too many classes, and BIC is better in situations where identifying too many classes would be as misleading as, or more misleading than, choosing too few classes.
For details on this, my colleagues and I recently produced a technical report titled, “Sensitivity and Specificity of Information Criteria.”