Experience gained in the prototype field test led to the recommended goal of shortening and simplifying the skills questionnaire to reduce rater burden, to increase the response rate of incumbent raters, and to produce data that would be more accurate and more user friendly to end-users. A panel of experts was again formed to revise the questionnaire. The panel shortened and simplified the questionnaire primarily by modifying the rating scale, clarifying the text of all constructs, and removing those constructs for which there were insufficient theoretical or research grounds. In addition, there were theoretical grounds for forming new and more general construct composites. Fifteen constructs in the prototype skills questionnaire were combined to form four new construct composites. Two of the new skill construct composites resulted from combining two constructs each, one from combining three constructs, and one from combining eight constructs. The four new skill construct composites replaced their constituent original constructs in the revised questionnaire.

The final instrument – a product of qualitative and quantitative evidence – included 35 skill constructs. The next section details the quantitative evidentiary findings.

The evidence for the creation, combination, acceptance or deletion of the initial prototype skill constructs came from factor analysis of ratings made by both incumbents and analysts (i.e., four sets of factor analyses were conducted for each of the four new construct composites). The factor analyses had three goals. The first goal was to demonstrate that a single factor underlies each of the new constructs. The second goal was to demonstrate that each of the constructs being combined has a majority of its variance accounted for by the factor underlying the new construct composite. The third goal was to examine whether the constructs needed to be differentially weighted before being averaged to form the composite scores.

To determine if a single factor underlies a construct composite, the average of the eigenvalues of the second factor across its four analyses were examined. Given that a value around 1.00 is generally considered the lower bound for factors that are meaningful, an average below .75 was selected as the criterion for demonstrating that only a single meaningful factor underlies one of the new construct composites.

To determine if a construct has a majority of its variance in common with the factor underlying a new construct composite, an average communality of .50 across the four analyses was selected as the criterion. This is a straightforward demonstration that a majority of a construct’s variance is accounted for by the factor underlying the composite.

To determine if the constructs could be averaged without differential weighting in forming composite scores, the criterion selected was that the smallest loading should be at least 75% as large as the largest loading within the composite construct. This is a stricter criterion than the one commonly used in forming factor composite scores. The common criterion for averaging the scores of variables loading on a factor is simply that each has a substantial loading on the factor, given the variables have equivalent or equated standard deviations (Nie, Hull, Jenkins, Steinbrenner, & Bent,1975).

The results of the four factor analyses for each of the four new construct composites were similar. The average of the eigenvalues of the second factor across the four analyses for each new construct composite ranged from .34 to .63 (mean .47). The average of the communalities for the constructs ranged from .56 to .83 (mean .70). That is, at least 56% of the variance of any given construct is accounted for by the factor underlying the new construct composite.

These results provide ample evidence that the new composites which were formed on theoretical and conceptual grounds also have an empirical basis. There is little reason to believe that a second factor underlies any of the four composites. A majority of the variance in each of the constituent constructs is accounted for by the factor underlying them.