EPIC Newsletter:
February 2020
Welcome to this month’s EPIC Newsletter. For this issue, our focus is on course assessment strategies, tips, and techniques. If you are looking to shake-up your course design by gamifying your grading system or would like to know more about inclusive strategies to assess student learning, please read on!
EPIC, by the way, will be hosting a special faculty only edition of our TILT Discovery workshop in collaboration with Associate Dean, Diversity and Inclusion, Professor Jenny Sharpe. If you would like to know more about transparent teaching and designing transparent assignments—a teaching approach proven to help the success of historically underserved students in the classroom—please join us for the TILT Discovery workshop on Monday, March 2, 2020 at 12:00-2:00pm in 186 Powell, facilitated by me, Lisa, and CAT’s Associate Director of Faculty Engagement, Beth Goodhue. To attend, you must RSVP via our Events page. Space is very limited and the RSVP page will close when capacity is reaced.
As always, let us know what you think about the newsletter or send us suggestions for future features at epic@humnet.ucla.edu.
On behalf of the EPIC team, thank you for joining us again this month!
Lisa Felipe
EPIC Program Director
|
|
|
An Interview with Professor Kie Zuraw and Adam Royer on Skills Grading
|
|
|
|
Dr. Kie Ross Zuraw, Professor and Director of Graduate Studies for Applied Lingustics, Department of Linguistics
|
|
|
Adam Royer, PhD Candidate and Teaching Fellow, Department of Linguistics
|
|
In their recent publication, "Gotta catch 'em all: Skills grading in undergraduate linguistics," (2019) Professor Kie Zuraw and co-authors Adam Royer, Ann M. Aly, and Isabelle Lin share how an innovative approach to grading piloted in two undergraduate linguistics courses, Phonetics and Phonology I, led to increased student learning, less stress for the students, the promotion of a growth mindset, less time spent grading by the instructors, and a simpler method of measuring learning outcomes for new accreditation requirements, among other positive results. How was this achieved? Professor Zuraw and her then-teaching assistants invented a grading system that was based on measuring the acquisition and mastery of a specific set of skills. Unlike a traditional grading system, in which student grades tend to reflect "some combination of mastery, compliance, and luck" (Zuraw e406), Zuraw's skills-based system simply requires students to demonstrate mastery of fifty core skills and, if they want to shoot for an A, to show an additional advanced proficiency in fifteen of those skills. Students are given multiple opportunities to solve problems or answer questions for each skill category in homework, quizzes, and exams. Students are not penalized for wrong answers, and once they demonstrate mastery of a specific skill, future mistakes in that category will not count against them. Therefore, progress throughout the course is measured through the aggregation of the required skills. Students can track their progress through an anonymized gradebook chart, and also see how their own progress compares to the rest of the class. This aspect lends itself to gamification—future versions of each course might include further elements of game design such as the incorporation of achievement badges or "leveling-up" to unlock opportunities to demonstrate more advanced proficiency (e419).
Two of the article's authors and teaching assistants for the aforementioned undergraduate courses, Ann M. Aly and Isabelle Lin, have since graduated from UCLA. Professor Zuraw wrote in an email to EPIC: "Ann M. Aly earned her PhD in linguistics from UCLA in 2017 and is a Senior UX Researcher at Agile Six Applications. Isabelle Lin earned her PhD in linguistics from UCLA in 2019 and is an artist and illustrator based in Paris."
EPIC was able to meet with the other two authors, Professor Kie Zuraw and Adam Royer, for an interview about skills grading and their experiences with this novel form of assessment. See some of the highlights from our conversation below.
|
|
What inspired you to invent the skills-based grading system?
ZURAW: This was something that happened to me over one summer. I had taught Phonology I for the first time the previous year and I got frustrated with myself when I was preparing the review for the final exam. I was listing out for the students, "here are all the skills that you need to have" for the final exam. And I realized, wait a minute, why didn't we start with this? It would be trivial to just give them that list in the future, but more than that, why is this not somehow where their grade comes from? Why does the grade come from this mysterious alchemy of points instead of simply from: have they learned the fifty or so things that they're supposed to be able to do at the end of this course? That was it. It was that simple.
Tell us a little about the grading system and how it works.
ZURAW: It works a little bit differently in the two classes that we've implemented it in so far. In the phonology class, we've identified about fifty skills that students need to learn. To get a B in the class, they need to demonstrate mastery in fifty. To get an A, they need to do that plus advanced mastery in some number that we're always tweaking to figure out what's right. That's a little more arbitrary. And of course we have ways to figure out lower grades than that, if someone hasn't mastered all the skills. Every quiz, homework assignment, and exam is an opportunity for students to show that they've mastered various skills. For instance, on a quiz or exam, there's a little box on the side that says, "This is a question about rule ordering." So if you haven't yet successfully demonstrated your mastery of rule ordering, that's a question you would want to answer. Students have the choice on exams and quizzes of which questions they want to answer. We try to make it like a "Choose your own adventure." [...] In the phonetics class, it's a little different because this is a class where students learn to make and perceive all the sounds of the world's languages. So in a sense, there are thousands of skills—being able to produce, perceive, and transcribe "f", being able to produce, perceive, and transcribe "rr", etc. Repeat for hundreds, maybe even thousands of sounds. So it's a little different there. So instead, we grouped things into fifteen macro skills. One of them might be "transcribe sounds" and students needed to successfully demonstrate that they could do that some threshold number of times. The idea is if you've shown twenty times that you can transcribe tone correctly, then you're probably good at transcribing tone. Even if we didn't test you on every possible combination of tones, we've sampled the space well enough that you probably have more or less mastered transcribing tone. [...] In theory it could be possible that the questions you're always getting right are always about a high tone followed by a low tone, but in practice it's unlikely that you'd be able to rack up enough successes unless you were actually able to transcribe a variety of patterns.
How have students reacted to this type of assessment?
ROYER: When I've explained to students how the system works, usually, in my experience, they are interested and intrigued but a little worried, especially in the first couple of weeks. The way the system works, you're only accumulating more and more skills, so your grade is going higher and higher and never ends up dropping any lower. But what that means is—for example, in the phonology course, there's fifty skills, and in order to get a B on your skills grade you need proficiency in all fifty skills. So if we're halfway through the course and we haven't even covered material in ten of the skills, no student will be at the level of a B. And so sometimes I'd have students come to office hours and ask, "Should I drop the course? I only have a C." And I say everyone is at the same level, you're perfectly fine, this is how it works. And so once you explain to them this is how it works, it's like night and day. They flip—they end up being much less anxious about the actual exams and missing questions on particular quizzes or homeworks. They get excited about getting different skills. You might try a particular skill mutliple times and then finally get it and be like, "I finally got gestural scores, I'm so happy now!" It plays into the gamification of things as well. [...] It helps them dissociate from the kinds of anxieties that they associate with particular letter grades and whatnot, and focus more on the actual skills and building those up. "What skills do I need to work on?' instead of "What do I need to get on this assignment in order to get a particular grade?"
How do you think that this grading system would translate to teaching languages?
ZURAW: That seems like a natural fit. This is not for every course. There are a lot of kinds of courses where I can imagine that this wouldn't work—courses that are heavily based on textual analysis or that kind of thing. But I think for language courses it would be perfect! Especially in maybe the first two years of a language course, before you're getting into reading lots of literature. There are things like "Correctly construct a counterfactual conditional" or "Use the subjunctive." And more practical things like "Be able to have a conversation where you give and receive directions," that kind of thing. I think most language instructors, especially at the introductory levels, have a very specific list of skills that they expect their students to master by the end of the year.
What are some tips you would give to instructors interested in trying out a skills-based assessment approach?*
ZURAW: I think the big thing is, if you've taught the course before, look through your exam review mateirals. Think about the way you construct your exams. If it's this kind of course that's conducive to this kind of grading, then you already have in mind, "yes, I want to make sure that on the exam I have one of this type of question, and one of that type of question," because you already know what you're trying to test. So just writing that down and making a list and making that transparent to students is great. Even if you don't want to do the skills grading, I think that's a great thing to do. If there is another course that this course is a prerequisite for, then you'll want to coordinate with the people that teach that next class and think about what skills people need to have coming out of your course. [...] The thing that's tricky and that you will need to adjust a lot the first time you try it, is making sure students have enough opportunities for each skill. Some skills might turn out to be easier than you realized, so you don't need as many opportunities as you've built in, and some might turn out to be harder. You can always do things like add some extra quizzes. Extra quizzes don't become a burden to students when each quiz can only make your grade go up. Students are usually pretty excited about extra quizzes because they get more chances to demonstrate auto segmental diagrams or whatever skill.
ROYER: That's what I've found in the times that I've taught this in the summer as well. You need that flexibility. Students are asking for more assignments, let's figure out how we can give them more opportunities in order to test these things. Especially for skills that might be introduced later in the course.
I can see how what you're doing creates a growth mindset. Students aren't afraid to try new things because it's exciting for them and it's a challenge, and there are no possible negative repercussions. So they actually get to explore that inclination to challenge themselves.
ZURAW: It makes it a lot easier too. Everyone knows, "yes, I should see my failures as learning opportunities," but that's hard to do when your failures cost you five points that you are never going to get back. And so I think this has made it a little easier for students to tap into that natural level of learning—that love of learning from your mistakes.
*See Zuraw's article for a list of considerations and suggested guidelines for instructors interested in implementing skills grading in their own courses.
|
|
Zuraw, K., Aly, A.M., Lin, I., & Royer, A.J. (2019). Gotta catch ’em all: Skills grading in undergraduate linguistics. Language, 95(4), e406-e427. doi:10.1353/lan.2019.0081.
|
|
Small Changes, Big Impact: Assessment
|
|
|
Most Universities’ Academic Success materials explain that learners should spend 3 hours weekly of outside preparation for every 1 credit hour of a course; in other words, a 3-credit humanities course requires, on average, 9 hours of outside preparation and learning. This tallies such that 25% of learning takes place inside of the classroom, and 75% outside of the classroom. Conversely, faculty assess 75-100% of the work that their learners complete, in spite of the fact that only 25% of that work is completed in physical proximity and in direct relationship to the faculty member. Summative, faculty-driven assessments don’t work well.
|
|
Instead, try to apply a common Instructional Design methodology to your assessment: Use GRADUAL RELEASE. In the classroom, this method combines Lecture—Large Group—Small Group—and Self-Directed Learning. Incorporate graduate release for assessment by incorporating the following multimodal and holistic practices:
- faculty-driven (i.e., graded assignments, feedback delivered through papers),
- peer-to-peer (i.e., peer review),
- self-directed (i.e., journaling, application cards, checklists, minute paper),
- student-driven (i.e., project wrappers, in-class polling, student generated test questions);
- Synchronous: face-to-face conferencing treated as a conversation and joint effort to revise an assignment rather than a student’s passive absorption of the faculty’s corrective feedback
- Multimodal: Screencapture or Audio-Feedback (i.e., use Jing or QuickTime), Sketchnote (draw or color code as you and the student discuss their work together)
- Gamified assessment (see Kie Zuraw’s interview this month!)
Changing up your assessment practices should be gradual and steer towards putting more opportunities into learners’ hands to assess the 75% of the work they are completing outside of the classroom. This month, try distributing these Application Cards after one of your lectures; they provide an opportunity for learners to connect knowledge to another discipline, career path, or activity of daily living:
|
|
Mary Flanagan: Critical Play
|
|
|
Mary Flanagan (Distinguished Professor at Dartmouth, Founder of Tiltfactor) offered a workshop in Critical Play to EPIC Mellon Fellows of the Emerging Technologies STE this month.
|
|
|
In her article “Lost in Translation: Comparing the Impact of an Analog and Digital Version of a Public Health Game on Players’ Perceptions, Attitudes, and Cognitions,” Flanagan explains that “games can effectively facilitate a ‘systems thinking’ approach to real-life issues.”
|
|
|
|
Critical play allows for an immersive and low-barrier-to-entry activity through which learners can apply their knowledge without fear of punishment or failure (a poor grade, peer or instructor criticism), and can strengthen their position as they explore wicked problems in subjects including social justice, health care, and language barriers. Games (just like our learning environment) are structured using Rules, Objectives, Procedures, Boundaries, Resources, Players, and Conflict. In your courses, you and your learners can engage in critical play using the following process:
|
|
To get started with introducing critical play into your course:
- Review the Values at Play article (https://www.valuesatplay.org/wp-content/uploads/2007/09/Values-at-Play_Curriculum_2014.pdf).
- Build a game around a pedagogical problem (i.e., interpreting an Emily Dickinson poem, teaching the subjunctive, root cause analysis of the SoCal Water Crisis).
- Implement the game in your course (or host a game day at the quarter’s end).
|
|
Photo Gallery: Ready, Set, Teach! Supporting First Gen Students
|
|
|
EPIC partnered with UCLA First To Go for our Winter quarter Ready, Set, Teach! roundtable luncheon on teaching practices that best support first-generation college students.
|
|
|
|
Conversation was lively at the "Resources for First Gen Students" roundtable.
|
|
|
|
Jean Paik-Schoenberg, Rhona Blaker, and Lisa Felipe lead discussion at the "Supporting First Gen Transfers" roundtable.
|
|
|
David Schaberg speaks with Gaby Barrios and Armando Romero at the "Getting to Know First Gen Students" roundtable.
|
|
|
|
Letty Treviño, Lisa Felipe, Sarah Valdovinos, and Symone Morales.
|
|
|
Participants listen at the "Getting to Know First Gen Students" roundtable.
|
|
|
|
Symone Morales, Maritza Lopez, Armando Romero, Letty Treviño, Lisa Felipe, and Sarah Valdovinos.
|
|
Photos by Alejandra Campoy
Graduate Student Researcher, EPIC Program
PhD Student, Comparative Literature
|
|
|
On Teaching and Learning: Resources Round Up
There are a TON of resources, scholarship, and advice for instructors out there! Here’s just some of what we have been reading and exploring this month.
|
|
“Plenty of excellent evidence suggests that whatever knowledge students bring into a course has a major influence on what they take away from it. So a sure-fire technique to improve student learning is to begin class by revisiting, not just what they learned in the previous session, but what they already knew about the subject matter.”
By James M. Lang, “Small Changes in Teaching: The First 5 Minutes of Class”
“Give students clear guidelines for class components, so they know what learning they are accountable for, including how they will be graded and why. Explicit articulation of learning objectives and goals, transparency around performance expectations and criteria-based grading systems empower students to share the responsibility for their learning and to develop growth mind-sets.”
By Soulaymane Kachani, Catherine Ross and Amanda Irvin, “5 Principles as Pathways to Inclusive Teaching”
|
“Inclusive assessment often challenges our notion of fairness by asking us to think in terms of equity rather than equality. Shouldn’t we be measuring students in the same manner if we are to be fair? That practice makes the assumption that all students are the same and therefore can be assessed the same way, an assumption we know has never been true.”
By Donna M. Qualters, “Inclusive Assessment: Equal or Equitable?”
Additional Resources on Assessment Techniques:
|
|
|
This newsletter is edited by Alejandra Campoy.
|
|
|
|
|
|
