I’ve had many experiences building teams of educators to review instructional materials, both in my previous district of Washoe, NV and, most recently, for EdReports.org. In my experiences, I’ve learned some organizing principles that have remained true regardless of the size of the review team, the tool you use, or the number of materials you’re considering.
1. Identify your leaders first and support them throughout the process.
Your leaders are critical to your review process. It is important to select and train your review team leaders first to ensure that they all have a strong understanding of the standards, the Shifts, and how to recognize them in instructional materials. As part of the interview process for leaders and reviewers, I recommend including a performance task to see how they connect their Common Core understanding to the process of reviewing materials. Your leaders should also receive training on how to build their teams, facilitate conversation, and deal with issues that may arise – such as problems with engagement.
How many leaders you’ll need will depend on your local needs and how you want to structure your review teams. I’ve found it helpful to create different review teams, each with their own leader, for each textbook series or set of materials you plan to review. That way, no group is expected to look at every series – which can be an overwhelming task. After you’ve weeded out options which clearly aren’t aligned, you might consider creating grade-level specific teams to carefully review a smaller set of options.
2. Be honest with your reviewers about the time required.
Being a part of a review process is an incredible opportunity but also a big responsibility – one which will take a large time commitment. Your team’s leaders should work together to try to estimate the time commitment required before they begin to recruit reviewers. The more information you can provide upfront, the better your chances that your reviewers will be able to stay committed throughout the entire process. In the past, I’ve asked reviewers to commit 5-6 hours of work per week throughout the review process.
3. Introduce your review tool first, train later.
A key part of training successful reviewers is allowing them to become familiar with the tool you’re using in your review. I recommend previewing the tool with them first by introducing it at a high level and then providing more in-depth training later. Following this two-step process will allow reviewers to formulate questions and digest how the tool works without being distracted by trying to master logistical elements like scoring. This will help them gain a deeper understanding of the tool before diving into the details.
4. Help reviewers focus on the evidence.
In my experience, after you introduce a review tool, reviewers can easily become wrapped up in what the score will be and lose sight of their primary responsibility: to find evidence of the alignment criteria. It’s important to remind them that determining a score is a byproduct of the evidence they report. One helpful practice I’ve found is for someone to bring the evidence they’ve collected to the group without a score and allow the rest of the group to use the evidence (as the reviewer has written it) to determine a score. This is a good way to see if reviewers are using evidence the same way and if evidence is being written clearly and effectively.
In your training, always bring the discussion back to the language of the review rubric you’re using. Make sure to spend time practicing differentiating between different criteria. Reviewers may be unclear about what makes each criterion or metric unique and cite the same piece of evidence multiple times. Each criteria element in a rubric is there for a reason and demands different evidence; if it feels like there’s overlap then there’s likely more evidence that needs to be collected.
5. Use examples to train your reviewers.
Using examples can be the most efficient way to train reviewers. Many reviewers may be able to recite the Shifts and even identify them in instruction, but being able to recognize the Shifts in materials is a different skill. I recommend using real examples, such as pictures from teachers’ manuals, so that reviewers can see what they’re trying to identify. Exemplars aren’t enough though; I also recommend using ‘head-scratchers,’ or examples which aren’t as clear-cut, to foster discussion during training and help reviewers work through trickier questions about alignment.
Perhaps the most important thing to remember is that training doesn’t stop once reviewing begins. It is critical that your reviewers continue to norm their technique along the way. Remember, the time you spend building strong reviewers is an investment in your implementation. Reviewers will leave a review process having had a clear influence on the materials that will be chosen, but they will also have a strong influence on how those materials are used in the classroom. They can apply what they’ve learned about alignment to their own work as classroom teachers and share their new expertise with colleagues.
