Section 15 Factor analysis
If we think that there might be some underlying variables behind our observed variables, we can use Factor Analysis (FA) to discover and measure these hidden variables. This section only covers exploratory FA. There is also a more advanced technique, confirmatory FA. This involves structural equation modelling where we use factors in regression models to predict other factors.
See Crawley (2013) section 25.2 and Navarro and Foxcroft (2018) section 15.1.
15.1 FA and PCA
Both FA and PCA are variation maximization techniques. In PCA we are attempting to maximize the total variation in data. In FA we instead try to maximize some underlying shared variation between variables. As such, factors in FA are the underlying latent variables that load the observed variables that we have (and not the factors in the same sense as in previous methods). Factors are equivalent to PCs in PCA. A common example of factors are personality traits: we can not measure these directly but we can ask subjects various questions and then use factor analysis to summarize the responses into factors that represent these traits. The distinction between PCs and factors leads to some differences between FA and PCA:
- PCA aims to explain total variation while FA aims to explain covariances,
- PCs are uniquely defined while factors depend on estimation and rotation method,
- In PCA each PC is a linear combination of variables but in FA the variables are expressed as linear combinations of factors.
How do we decide which method to use? If our goal would be to summarize variables into a smaller set of variables, then we should use PCA. If the goal is to determine some underlying variables, then we should use FA.
15.2 Orthogonal factor model
In FA framework, variables are loaded by latent variables, i.e. factors. Each variable in a factor model can thus be represented as a linear combination of factor loadings as follows:
\[x_j = q_{jl} F_l + U_j +\mu_j,\]
where
- \(x_j\) is a variable \(j\),
- \(q_{jl}\) the loading of the \(j\)th variable on the \(l\)th common factor,
- \(F_l\) the \(l\)th common factor,
- \(U_j\) the \(j\)th specific factor and
- \(\mu_j\) the mean of \(x_j\).
The idea behind FA is to explain the common variance of variables by extracting factors from the correlations between variables. There are three distinct methods for extracting factors. We are not going to cover the differences but these are residual minimization, likelihood maximization and principal axis method. In case of the latter method, eigenvalues and eigenvectors can also be found, but it can be argued that these two concepts should not be used in the FA framework.
In Jamovi: Factor > Exploratory Factor Analysis.
15.3 Rotations
To better distinguish factors that best separate our variables, we can apply a rotation on factors. The result of rotation is an adjusted loadings matrix, where factors are more distinct and thus more intuitive to interpret. While rotation introduces some subjectivity as there are no rules about the choice of rotation, it often eases description of factors. There are many rotations but the three most common are described below. Rotations that retain orthogonality result in more distinct factors but restrict factors from correlating even if they actually do.
15.3.1 Varimax
This rotation involves maximizing variances of squared loadings within factors. As a result, each factor has few variables with high loadings. Factors remain orthogonal, thus uncorrelated.
In Jamovi: Factor > Exploratory Factor Analysis | Method | Rotation: Varimax.
15.3.2 Oblimin
This rotation also involves maximization of squared loading variances within factors. The difference from Varimax rotation is that factors are not necessarily orthogonal but oblique and can thus be correlated. The resulting factors may be more similar to each other than in case of Varimax but the variances of loadings within factors are likely to be higher.
In Jamovi: Factor > Exploratory Factor Analysis | Method | Rotation: Oblimin.
15.3.3 Quartimax
This rotation aims to maximize variances of factor loadings within variables (not factors as with the two other rotations). The result is a factor structure in which each variable has high loadings in as few factors as possible. In other words, each variable is loaded by only few factors.
In Jamovi: Factor > Exploratory Factor Analysis | Method | Rotation: Quartimax.
15.4 Assumptions
While PCA does not have very strict assumptions, non-sphericity and particularly sampling adequacy are important for FA. These two assumptions can be described and tested for as explained below.
15.4.1 Sphericity
Variables need to be correlated to a certain extent for us to able to summarize them into a smaller set of factors. If data cloud on scatter plot is spherical (spherical data), the correlations in data are random and factor analysis is thus not accurate. Whether or not correlation matrix is statistically significantly different from identity matrix can be tested with Bartlett’s test.
\(H_0:\) Correlation matrix is equal to identity matrix \(H_1:\) Correlation matrix is different from identity matrix
If the test returns \(p \ge 0.05\), then we should not trust the results of factor analysis.
In Jamovi: Factor > Exploratory Factor Analysis | Assumption Checks | Bartlett's test of sphericity.
15.4.2 Sampling adequacy
In addition to correlations it is necessary that each factor is not related to only few variables. We can use Kaiser-Meyer-Olkin (KMO) measure to determine if sum of partial correlations is higher than sum of correlations or not. The value of the measure is interpreted as follows regarding the suitability of FA.
- 0 … 0.6 - Not suitable
- 6 … 0.7 - Suitable
- 0.7 … 0.9 - Adequate
- 0.9 … 1 - Excellent
In Jamovi: Factor > Exploratory Factor Analysis | Assumption Checks | KMO measure of sampling adequacy.
15.5 Number of components
The methods that can be used to determine an optimal number of components are the same as those for PCA, because PCs are equivalent to factors. For FA a further consideration can be the factor loadings or uniqueness. If some variable has a very high uniqueness, adding factors might increase the respective communality.
In Jamovi: Factor > Exploratory Factor Analysis | Number of Components.
15.6 Interpretation
15.6.1 Loadings
The interpretation of loadings in FA is similar to PCA, with the exception that factors now have an effect on variables. In FA only loadings that are higher than 0.3 or 0.5 are usually considered and used for interpretation of factors.
In Jamovi: Factor > Exploratory Factor Analysis | Factor Loadings | Hide loadings below: 0.5.
15.6.2 Uniqueness and communality
Uniqueness is a measure of proportion of variation explained only by a particular variable and not by factors. Communality is the inverse of uniqueness (\(1-Uniquenss\)), i.e. the variation of a variable that is shared with other variables.
15.6.3 Complexity
Complexity can be calculated for each variable and it expresses the number of factors which load each variable.
FA is a controversial method in statistics because we can never be sure if the factors actually exist and were correctly estimated. This is due to the different options for choosing the number of factors, methods of estimating the model, rotation techniques and subjectivity of interpreting factors.