Mary J. Allen, mallen@csub.edu SACS-COC Summer Institute July 28, 2008. For the entire document, see https://assessment.trinity.duke.edu/documents/DirectandIndirectAssessmentMethods.pdf
Two Basic Ways to Assess Student Learning
- Direct – The assessment is based on an analysis of student behaviors or products in which they demonstrate how well they have mastered learning outcomes.
- Indirect – The assessment is based on an analysis of reported perceptions about student mastery of learning outcomes.
Properties of Good Assessment Techniques
- Valid – directly reflects the learning outcome being assessed
- Reliable – especially inter-rater reliability when subjective judgments are made
- Actionable – results help faculty identify what students are learning well and what requires more attention
- Efficient and cost-effective in time and money
- Engaging to students and other respondents – so they’ll demonstrate the extent of their learning
- Interesting to faculty and other stakeholders – they care about results and are willing to act on them
- Triangulation – multiple lines of evidence point to the same conclusion
Strategies for Direct Assessment of Student Learning
- Published Tests
- Locally-Developed Tests
- Embedded Assignments and Course Activities
- Community-service learning and other fieldwork activities
- Culminating projects, such as papers in capstone courses
- Exams or parts of exams
- Group projects
- Homework assignments
- In-class presentations
- Student recitals and exhibitions
- Comprehensive exams, theses, dissertations, and defense interviews
- Portfolios
- Showcase vs. Developmental Portfolios: best work vs. evidence of growth
- Workload and storage demands for large programs can be overwhelming
- Collective Portfolios
Strategies for Indirect Assessment of Student Learning
- Surveys
- Interviews
- Focus Groups
Qualitative vs. Quantitative Measures
(Reference: Cal State Stanislaus Assessment Workbook)
Not all assessment data must be quantitative; in fact, a mixture of quantitative and qualitative measures may offer the most effective means of measuring student learning outcomes and program goals. Of course, the assessment measure chosen should be fashioned to the unique needs of the program.
Qualitative Assessment
Some examples:
- Exit Interviews
- Focus Groups
- Writing Samples
- Open-ended questions on surveys and interviews
- Employer interviews
Quantitative Assessment
Some examples:
- Written and Oral Exams
- Research Papers
- Senior Projects
- Exam Scores
- Course Grades
Why don’t student opinions or satisfaction count as assessment?
Why can’t we just ask students if they thought the program was effective? If students are satisfied with their learning and the program quality, isn’t that evidence of effectiveness?
The purpose of assessment is to provide information about the student learning and development that occurs as a result of a program. Student’s opinions and satisfaction ratings are considered indirect measures of student learning. They provide some information, but it is insufficient to make inferences about the program. In order to feel comfortable basing decisions about a program on assessment results, the assessment needs to use a direct measure of student learning. Direct measures means that scores reflect the student’s actual level of knowledge or development.>
Why can’t we use course grades for assessment?
(Reference: Adapted from a James Madison University resource no longer available online. See also Assessment and Grades. As noted in this latter resource, Suskie [2004] and Walvoord and Anderson [1998] provide suggestions about “how to tie grades to explicit learning goals and standards.”)
Course grades are assigned to individual students to indicate the extent to which a student has met the instructor’s expectations for a given set of course requirements. Assessment results are intended to reflect the extent to which all students achieve the outcomes of a program. Clearly, grades and assessment differ in that one deals with individuals and courses and the other deals with groups and programs. Why does this difference matter?
How grades are assigned varies across courses and course sections. This means we can’t compare grades across courses to make inferences about the program as a whole.
Often, criteria unrelated to program outcomes are considered when assigning grades – e.g., attendance, participation, or getting work turned in on time. Some graded assignments may help students prepare for an outcome but not be part of the outcome. This means that grades are most often not a pure measure of student learning.
Grades are assigned based on one instructor’s judgment only. This means that there is a lot of subjectivity in grades. Assessment involves multiple raters and checks on reliability and validity of scores.
Grades don’t tell us what information the student knows and doesn’t know. Assessments are intended to provide additional and more descriptive information so that we know more about what each assessment score means about a student’s ability level.