If you work in assessment, you know that the answer to this question is far from simple. Early this month our director of Psychometrics, Dr. Jennifer Dunn, gave a talk at TEDxPiscataquaRiver in Portsmouth, NH. Titled “Using Data to Motivate Change,” Dr. Dunn described her own introduction to psychometrics, and how the challenge of measuring the intangible—what a person knows—captured her interest. Here are some of the main points of her talk.
The three “pillars” of good testing
- Reliability and validity underpin the quality of the measure
- Define the story and gather evidence to support the design
- Use the results to articulate a call to action
The complexity of measuring a construct well
For example, if a student answers 2 + 2 = ? correctly, does this tell us that
the student can add? No.
the student can add single-digit numbers? No.
the student can add to 4? No.
So what can we confidently conclude? That the student can add 2 + 2!
Understanding the “right” story behind the data
It’s tempting to use a single example—or anecdote—to sway a decision. But we need to make sure to ask: Is this anecdote a compelling exception or the rule? We need to make sure that the anecdotes we use to bring data to a human scale illustrate what the results really tell us. If we understand the stories the data tell us, we can use those stories to influence positive changes.