Award Date

12-2010

Degree Type

Dissertation

Degree Name

Doctor of Philosophy in Learning and Technology

Department

Educational Psychology

First Committee Member

Kendall Hartley, Chair

Second Committee Member

Michael Nussbaum

Third Committee Member

Kent J. Crippen

Graduate Faculty Representative

Sterling J. Saddler

Number of Pages

169

Abstract

This research attempted to bridge the gap between cognitive psychology and educational measurement (Mislevy, 2008; Leighton & Gierl, 2007; Nichols, 1994; Messick, 1989; Snow & Lohman, 1989) by using cognitive theories from working memory (Baddeley, 1986; Miyake & Shah, 1999; Grimley & Banner, 2008), multimedia learning (Mayer, 2001), and cognitive load (Chandler and Sweller, 1991, 1992; Cerpa, Chandler, & Sweller., 1996) to identify potential design weaknesses of traditional select-and-fill-in (SAFI) concept map assessment and then to guide the design of the new and improved multiple-choice concept map (MCCM) assessment method. The primary objective of this study was to evaluate the effects of (1) the type of the list of concepts or relations and (2) the spatial placement of the selection list on the examinee's overall mental effort. Using a 2x2 Factorial MANCOVA, with prior knowledge as the covariate, the participants were compared on the length of time to complete the assessment and on examinee's rating of personal mental effort exerted during the assessment. A Simple Planned Comparison was conducted to evaluate estimated mean differences between the primary group of interest (MCCM-integrated Group 4) and each of the other three groups. Of additional interest were differences among the four different groups on the examinees' attitudes and impressions toward their respective assessment method; these data provided more insight into the quality of design for each treatment group.

The type of List/Map had a significant main effect on the overall mental effort/strain or the linear combination of time on task and mental effort rating scores. More specifically, however, the test of between-subjects effects indicated that the type of list/map had a statistically significant effect on the time to complete the assessment, but not on the examinees' mental effort ratings. In other words, participants who received the MCCM assessments (which used the MCL) completed the assessment task significantly faster but did not rate higher or lower on personal mental effort than the participants who received the traditional SAFI concept map assessments.

The spatial placement of the selection list (integrated and non-integrated) did not have a statistically significant main effect on the linear combination of time on task and mental effort rating scores. The test of between-subjects effects indicated that the spatial placement of the selection list (integrated versus non-integrated) did not have a statistically significant effect on the time to complete the assessment or on the examinees' exerted mental effort. In other words, participants who received who received the integrated lists did not complete the assessment task faster and did not rate themselves lower on personal mental effort than the participants who received non-integrated selection list.

The estimated mean time on task of the primary group of interest (MCCM-integrated Group 4) was significantly shorter than that of both SAFI groups (integrated and non-integrated), but was not significantly shorter when compared to the time on task of the MCCM-non-integrated Group 3. There were no significant differences on mental effort ratings between MCCM-integrated Group 4 and either of the other three groups.

The participant responses to the Attitudes and Impressions Questionnaire indicated a general preference for the MCCM over the more traditional SAFI map and, in some cases, a slight preference of MCCM-integrated over MCCM-non-integrated.

The multiple-choice concept map (MCCM) is a new interactive computer-based assessment method that bridges well-documented research findings from cognitive science and psychometrics and has the potential to be an external knowledge representation with much-desired characteristics of being valid, efficient, and explicit (Mislevy, Behrens, Bennett, DeMark, Frezzo, Levy, Robinson, Rutstein, Stanley, Winters, & Shute, 2007) when assessing students' knowledge structures or schema-based knowledge (Pellegrino, Chudowsky, & Glasser, editors, 2001; NAEP, 2008).

Keywords

Assessment; Cognitive psychology; Concept; Examinations—Design and construction; Map; Measurement; Method; Multiple-choice examinations; Tool

Disciplines

Cognitive Psychology | Educational Assessment, Evaluation, and Research | Educational Psychology

File Format

pdf

Degree Grantor

University of Nevada, Las Vegas

Language

English

Rights

IN COPYRIGHT. For more information about this rights statement, please visit http://rightsstatements.org/vocab/InC/1.0/

Download


Share

COinS