Measurement Tools

Strategies for Direct and Indirect Assessment of Student Learning

Mary J. Allen, mallen@csub.edu SACS-COC Summer Institute July 28, 2008

See http://assessment.aas.duke.edu/documents/DirectandIndirectAssessmentMethods.pdf for the entire 12 page document

 

Two Basic Ways to Assess Student Learning:

1. Direct – The assessment is based on an analysis of student behaviors or products in which they demonstrate how well they have mastered learning outcomes.

2. Indirect – The assessment is based on an analysis of reported perceptions about student mastery of learning outcomes.

 

Properties of Good Assessment Techniques

• Valid—directly reflects the learning outcome being assessed

• Reliable—especially inter-rater reliability when subjective judgments are made

• Actionable—results help faculty identify what students are learning well and what requires more attention

• Efficient and cost-effective in time and money

• Engaging to students and other respondents—so they’ll demonstrate the extent of their learning

• Interesting to faculty and other stakeholders—they care about results and are willing to act on them

• Triangulation—multiple lines of evidence point to the same conclusion

 

Strategies for Direct Assessment of Student Learning

1. Published Tests

2. Locally-Developed Tests

3. Embedded Assignments and Course Activities

  • ·         Community-service learning and other fieldwork activities
  • ·         Culminating projects, such as papers in capstone courses
  • ·         Exams or parts of exams
  • ·         Group projects
  • ·         Homework assignments
  • ·         In-class presentations
  • ·         Student recitals and exhibitions
  • ·         Comprehensive exams, theses, dissertations, and defense interviews.

4. Portfolios

  • ·         Showcase vs. Developmental Portfolios: best work vs. evidence of growth
  • ·         Workload and storage demands for large programs can be overwhelming!

5. Collective Portfolios

 

 

Strategies for Indirect Assessment of Student Learning

 

  1. 1.       Surveys
  2. 2.      Interviews
  3. 3.      Focus Groups

 

Qualitative vs. Quantitative Measures

(from Cal State Stanislaus Assessment Workbook)

Not all assessment data must be quantitative; in fact, a mixture of quantitative and qualitative measures may offer the most effective means of measuring student learning outcomes and program goals. Of course, the assessment measure chosen should be fashioned to the unique needs of the program.

QUALITATIVE ASSESSMENT – “Assessment findings that are verbal descriptions of what was discovered, rather than numerical findings.” - Allen (2004) p.171

Some Examples Include:

Exit Interviews

Focus Groups

Writing Samples

Open-ended questions on surveys and interviews

Employer interviews

 

QUANTITATIVE ASSESSMENT – “Assessment findings are summarized with a number that indicates the extent of learning.” – Allen (2004) p.171

Some Examples Include:

Written and Oral Exams

Research Papers

Senior Projects

Exam Scores

Course Grades

 

Why don't student opinions or satisfaction count as assessment?
Why can't we just ask students if they thought the program was effective? If students are satisfied with their learning and the program quality, isn't that evidence of effectiveness?

The purpose of assessment is to provide information about the student learning and development that occurs as a result of a program. Student's opinions and satisfaction ratings are considered indirect measures of student learning. They provide some information, but it is insufficient to make inferences about the program. In order to feel comfortable basing decisions about a program on assessment results, the assessment needs to use a direct measure of student learning. Direct measures means that scores reflect the student's actual level of knowledge or development.

 

From http://www.jmu.edu/assessment/pass/assmntresources/what_is_assessment.htm with a little bit of editing.

Why can't we use course grades for assessment?
Course grades are assigned to individual students to indicate the extent to which a student has met the instructor's expectations for a given set of course requirements. Assessment results are intended to reflect the extent to which all students achieve the outcomes of a program. Clearly, grades and assessment differ in that one deals with individuals and courses and the other deals with groups and programs. Why does this difference matter?

How grades are assigned varies across courses and course sections. This means we can't compare grades across courses to make inferences about the program as a whole.

Often, criteria unrelated to program outcomes are considered when assigning grades such as attendance, participation, or getting work turned in on time. Some graded assignments may help students prepare for an outcome but not be part of the outcome. This means that grades are not a pure measure of student learning.

Grades are assigned based on one instructor's judgment only. This means that there is a lot of subjectivity in grades. Assessment involves multiple-raters and checks on reliability and validity of scores.

Grades don't tell us what information the student knows and doesn't know. Assessments are intended to provide additional and more descriptive information so that we know more about what each assessment score means about a student's ability-level.