The research design for generating skill profiles for 739 different careers involved obtaining rating profiles from two sources: a) an SME (Subject Matter Expert) study in which skill ratings were derived from expert judgments of occupational analysts and industrial/organizational psychology graduate students, and b) ongoing field research to develop rating profiles directly from respondents in various careers (a.k.a. career “incumbents”). Both the incumbent and the SME have advantages. For example, incumbents are able to provide first-hand, job-level information, whereas analysts are able to discern relatives of skills across all occupations.

These skill profiles were instrumental in the development of the CareerMatch™ system, allowing users to compare their own skill profiles to that of each career in the database and determine which careers best matched their personal skill set.

25 occupational analysts and 32 industrial/organizational psychology graduate students, assembled in groups of five or six, independently rated 739 careers for each skill construct using the prototype questionnaire discussed in the previous chapter. Collectively, they produced 6,605 sets of skills ratings. Outlier ratings were first examined and discussed before being included or discarded. After a thorough review of any statistical anomies, profiles were then developed from the mean scores of the 6,605 sets of skill ratings, and submitted for further review by statisticians. Interrater reliabilities were calculated for the mean of the analyst rating on each construct. The reliabilities for each construct mean were all above .60 with an overall mean reliability of .87, indicating strong internal consistency.

Data collection from incumbent raters is an extensive and ongoing research effort conducted by the Research Triangle Institute. This data collection program was developed as a continuous activity in order to maintain the most and up-to-date information available. About 200 career skill profiles are updated each year from field surveys of thousands of respondents. Each updated skill profile replaces an existing incumbent rating and/or SME rating. To date, 480 career skill profiles have been produced from incumbent raters.

The incumbent studies data collection program uses a two-stage sample design, including a statistical sample of businesses and a sample of workers in selected occupations within the sampled businesses. These ‘incumbent workers’ are asked to complete the skills questionnaire.

In the initial field tests using prototype versions of skills instrument, the response rates for incumbent workers were low.