This chapter presents a broad overview of the procedures implemented to develop the Interest Inventory. The information is provided at a fairly basic level of technical detail to give the wide range of users of the test an introduction to its technical underpinnings.

Goals of the Project. Four primary goals served as the basis for the development process of the Interest Inventory test:

1. Develop an instrument with strong technical characteristics that would provide users with accurate and useful information.
   
   
2. Develop a fair and unbiased instrument that would serve the needs of users from a variety of ethnic, cultural, and socioeconomic backgrounds. Every effort was made to ensure that the instrument would be helpful to users with different backgrounds.
   
   
3. Develop an instrument that included items representing the entire world of work. This would help ensure that the instrument would provide useful information to individuals with different work-related goals and interests.
   
   
4. Develop an instrument that could be used as a self-assessment tool that individuals could self-administer and self-interpret. This self-assessment instrument would help empower users to take control of their career exploration efforts. Of course, the instrument also could be used by counselors with users in a one-on-one or group setting.

Review of Existing Instruments. Before development, existing DOL interest instruments (e.g., USES Interest Inventory, USES Interest Checklist, Job Search Inventory) were reviewed to determine if they could contribute items on the Holland scale to the new Interest Profiler. From a pool of 453 existing items, 281 were retained, and an additional 288 new items were generated based on the content of existing items. These 569 items were included in a pilot test with individuals from a variety of backgrounds (e.g., age, education, race, socioeconomic, gender) to examine endorsement rates. Items with low endorsement rates were dropped, as well as items with duplicate content, resulting in a pool of 532 items.

Taxonomy Development. A taxonomy was created to provide structure to the process of developing and selecting items for the Interest Profiler, helping ensure that a variety of items representing the world of work would be included in the final instrument. Once the taxonomy was developed, the pool of existing items would be placed into the structure. Then, areas within the taxonomy that did not have enough items would be identified, indicating that new items would have to be developed.

The six RIASEC constructs served as the over-arching structure of the taxonomy. Within each RIASEC construct, work content areas were identified. These areas were derived from the 66 work groups contained in the Guide for Occupational Exploration (GOE; U.S. Department of Labor, 1979a), which is very representative of the world of work. Each of the 66 areas was assigned to one of the six RIASEC categories, based on expert judgments. Additionally, with each RIASEC construct, five levels of training requirements were identified to help ensure that items were developed that represent the variety of complexity in the world of work. The training levels were a modified form of the Specific Vocational Preparation Scale (SVP; U.S. Department of Labor, 1991b), which focuses on the amount of time required to learn the techniques, acquire the information, and develop the skills to perform a job. Lastly, based on a literature review and on the purpose of the Interest Profiler, description of work activities was selected as the type of item to be developed.

Item development targets for the taxonomy were set at a minimum of 100 items per RIASEC construct, with equal representation for each work content area. These numbers were set fairly high because it was estimated that a large number of items would fail to pass a variety of later development phases (e.g., item screening and item tryout).

Placement of Items. Each of the 532 items in the pool, derived from the initial pilot study, was placed within the taxonomy. A team of four judges was trained in Holland’s (1985a) vocational personality theory and the taxonomy. Judges reviewed the items and independently assigned them to one of the work content areas within a RIASEC construct. Assignment disagreements were flagged, discussed, and resolved. After the placement of the items was complete, the coverage of the taxonomy was examined. Areas that did not have enough items to meet the taxonomy targets were identified for new item development.

Development of New Items. A team of four item writers was trained in Holland’s (1997) vocational personality theory and the taxonomy. Each item writer was requested to write new items that met the following criteria: (1) filled in areas of the taxonomy that did not meet minimum goals; (2) were descriptions of work activities; (3) increased the representation of training-level requirements found within the RIASEC construct; (4) were inoffensive to individuals and subgroups; (5) contained vocabulary comprehensible to individuals with an eighth grade reading level; (6) would elicit an endorsement rate that falls between 10 percent and 90 percent; (7) would likely reduce spurious gender and race/ethnic endorsement rate differences; and (8) would be familiar to individuals from a variety of settings, including a.) entry level and career transition counseling settings, b.) urban, rural, and suburban settings, and c.) nationwide regional settings. All new items were reviewed and edited by the team of item writers.

Pilot Study. A total of 272 new items was developed, resulting in a pool of 804 items (532 items from the earlier phase, plus 272 new items). These items were included in a small pilot test with individuals from a variety of backgrounds (e.g., age, education, employment status). Items with extreme means, large gender differences, or large race/ethnic differences were removed. Priority was placed on eliminating items with similar or duplicate content. After this phase, a pool of 776 items existed.

The pool of 776 items underwent a comprehensive screening process designed to remove items that failed to meet the rigorous standards for inclusion in the Interest Profiler. Each item was required to pass the seven screens presented below to be included in the next phase of the instrument development.

Retranslation. This screen was conducted to ensure that items truly represented their intended RIASEC construct. Five expert judges in Holland’s (1985a) vocational personality theory received a pool of items with no indication of the construct or work content area each item was intended to represent. Judges independently assigned each item to a RIASEC construct. Following the assignment task, judges discussed assignment differences, recommended item alterations, and finalized all ratings. Items were retained for which at least four of the five judges agreed on assignment.

Sensitivity. The purpose of this screening was to ensure that items would not be offensive to particular segments of the potential user population. A panel of six individuals representing diverse race/ethnic and gender groups was convened. The protocol for the screen was derived from guidelines developed by the Educational Testing Service (1987), along with a review of the sensitivity procedures used in the development of the O*NET Ability Profiler (Mellon, Daggett, MacManus, & Moritsch, 1996). Panel members reviewed each item for possible bias against or offensiveness to racial, ethnic, or gender groups. The panel concluded with a list of suggested item revisions and deletions that were incorporated within the item pool.

Comprehensibility. The estimated range of education for potential users of the Interest Profiler begins at the junior high school level; thus, items must be comprehensible to these users. An eighth grade reading level was selected as the goal for the items. The Living Word Vocabulary (Dale & O’Rourke, 1981) was used to determine the grade level appropriateness of the vocabulary present in each item. Two sets of inspectors independently identified the grade level assigned to all words present in the pool of items.

All items with words exceeding an eighth grade level were identified. For each of these items, one of the following actions was taken: (1) inappropriate grade-level words were replaced with synonyms with a lower grade-level designation; (2) items were entirely rewritten; or ( 3) alterations of the items were overruled by a team of four judges.

Familiarity. The work activities described by the items within the final version of the instrument should be recognizable (i.e., familiar) to the entire range of potential users. Eight focus groups were conducted in four regions of the country to determine if the items were indeed recognizable by different segments of the potential user base. The groups were drawn from employment service offices, community colleges, and technical/trade schools located in urban, suburban, and rural sites. A total of 254 individuals from a variety of backgrounds (e.g., age, education, employment status) participated. Each participant rated the familiarity of the work activities. Items that individuals were not able to recognize were eliminated.

Training Requirement. This screening was conducted to ensure that items represented the broad range of training and educational requirements specified by the taxonomy. Occupational analysts with expert knowledge of the Specific Vocational Preparation scale (SVP; U.S. Department of Labor,1991b) were trained on the use of the Modified SVP scale. Each analyst rated the amount of vocational training required to perform the work activity described by the items. The rating of training level was used to remove items from areas of the taxonomy that were over-represented. The goal was to maximize the variety of training levels represented by items within each RIASEC construct.

Duplication. The purpose of this screening was to eliminate items with identical or nearly identical content. A team of inspectors reviewed the pool of items to ensure that nearly identical work activities were not present. For example, “type a memo” and “type a letter” would be considered nearly identical, and only one would be retained.

Copyright. To avoid copyright infringement, potential Interest Profiler items were compared to items widely used in existing interest instruments. Items were compared with those in the (1) Interest-Finder (U.S. Department of Defense, 1995), (2) Self-Directed Search (Holland, 1985b), (3) Strong Interest Inventory (Hansen & Campbell, 1985), and (4) UNIACT-R (American College Testing Program, 1995). Two inspectors independently identified duplicate and near duplicate items. Agreement between the inspectors was extremely high, with the few discrepancies being resolved by the team of inspectors. All items that represented potential copyright infringements were removed.

A total of 226 items failed to pass the seven-stage screening process, resulting in a pool of 500 items. A large scale study was conducted to gather information on the psychometric characteristics of the 500 items left in the tryout pool. This information would serve to identify those items most likely to yield a test with high reliability, low gender and race/ethnic biases, and strong evidence of construct validity. In addition, the Interest-Finder (Defense Manpower Data Center, 1995) was administered to allow for a comparison between the newly created Interest Profiler and an established interest instrument. The Interest-Finder is a vocational interest assessment instrument developed by the Defense Manpower Data Center for use in the ASVAB Career Exploration Program.

Sampling Plan. The sampling plan for this study attempted to target groups of users most likely to use the Interest Profiler upon its completion. Groups identified included: (1) unemployed workers, (2) junior college and technical/trade school students, (3) high school students, (4) college students, and (5) workers in transition (employed workers looking for different jobs/careers). The sampling plan also called for a high proportion of minority participants, an equivalent number of participants from each gender, and participants drawn from a variety of regions across the United States.

Participants. Data collection sites included employment service offices, high schools, junior colleges, technical/trade schools, universities, and other government agencies located in six states across the country (Florida, Michigan, New York, North Carolina, Texas, and Utah). A total of 1,123 participants provided useable responses. The sample consisted of approximately equal numbers of males and females, a high degree of ethnic diversity, a broad distribution of age groups, and represented a variety of education and employment situations.

Procedures. Participants were administered, in a counterbalanced fashion, the Interest Profiler and the Interest-Finder. The Interest Profiler consisted of 500 tryout items. In addition, each participant completed a brief demographics questionnaire, along with a comment sheet eliciting feedback regarding the Interest Profiler.

Item Analyses. A set of general item-level screens were conducted to eliminate items with extremely low and high endorsement rates, items with large differences in endorsements between males and females, items with large differences among racial/ethnic groups, and items that did not correlate highly with their intended scale. An item pool of 461 items was retained after these screens.

Scale Analyses. The purpose of this stage of the analyses was to select from the pool of items a total of 180 items that would create six internally consistent scales which would: (1) demonstrate strong conformity to the hexagonal model of Holland’s (1985a) theory of vocational interests; (2) contain maximum training level and occupation representation; and
(3) minimize gender and race/ethnic endorsement rate differences.

Based on their correlations with the six RIASEC scales, items were rank ordered in terms of their conformity to the structure of the hexagonal model. The structure specifies that an item should correlate most highly with its target scale (i.e., the scale it was intended to measure), next strongest with its adjacent scale, less strongly with its alternative scales, and least strongly with its opposite scale (for an overview of the Holland Model, please refer to the earlier chapter, Introduction & Theory).

Items were eliminated if they correlated less with their target scale than with another scale. Then, four judges with psychometric backgrounds, as well as training in Holland’s (1985a) vocational theory and the Interest Profiler taxonomy, independently made qualitative selection judgments based on the following information: (1) item-to-scale correlations, (2) gender and race/ethnic endorsement rate differences, (3) training level requirement ratings, and (4) work content area assignments. Judges discussed their respective selections and agreed on an initial selection of 30 items per scale.

Finally, starting with the initial 30-item scales, different item combinations within scales were examined to maximize the empirical relationships of items within scales, as well as to minimize the relationship of each item with non-target scale totals. For example, an item was replaced if its removal significantly increased the scale’s internal reliability.

The current Interest Inventory began with the 180 items developed under the Interest Profiler project, and was further refined to include a final pool of 150 items on six 25-item scales. Initial user feedback had suggested that the length of the original instrument (90 questions, 180 items) might prove problematic as user attentiveness began to fade toward the end of the test. Reliability & Item Analysis revealed that the reliability of the test would not be significantly compromised by dropping an additional 30 items to create a final 75-question test (with 150 total items). Items were reviewed for how well they discriminated the population into Holland's model and how much variance they exhibited. Weaker items were removed. Confirmatory factor analysis (test of the factor loadings and intercorrelations of items) also helped verify which of the original 180 items proved of most predictive of Holland’s hexagonal model.

The final instrument included six scales of 25 items. An extremely wide representation of work activities was achieved. One or more items were present in approximately 80% of the work content areas of the original taxonomy.

Pilot User Study. In a pilot user study, the Interest Inventory and Interest Profiler were administered to 232 individuals from a variety of backgrounds (e.g., age and employment status). All six scales in the Interest Inventory demonstrated a high degree of internal reliability, with coefficient alphas ranging from .93 to .96. Although differences exist between the Interest Inventory and the Interest Profiler, examination of the instruments’ scale intercorrelations reveals a very high correlation between corresponding scales, ranging from .73 (Artistic) to .48 (Enterprising), with a median value of .62. Overall, the correlational relationships between the two instruments provide evidence of both convergent and discriminant validity.