I know that, at this time of year especially, thinking about your state’s assessment may evoke some intense emotions. While I don’t expect you to ever love assessment season, I’d like to try to help you feel more comfortable with summative assessments, at least in regard to their overall quality and the thoughtfulness behind them. You might be surprised by how big a role educators play in assessment development. Teachers are involved in every step of the process!

Before a state or consortium can even begin to think about the development of test questions, they have to make sure several important features are in place, including things like:

1. A clear set of academic standards that outline the expectations for students to prepare them for college and careers.
2. Thoughtful test design to ensure that the test is a fair, reliable, and valid measure of students’ skills. This step is usually referred to as “creating a blueprint.” The blueprint provides guidance on the number of items or points on the assessment, which standards are covered, what constructs that will be reported on, etc.
3. Explanatory documents detailing what test items that are aligned to their standards look like. These documents are typically referred to as “item specifications” and help get the item writers grounded in the expectations of the standards. They also provide guidance on how to effectively use item types to access the standards in deep and meaningful ways.

That list seems pretty simple but getting all that pulled together can take a year or more. Why? Because there are so many people involved in the process, including educators!

Teachers play a large role in their state and/or consortium assessments, and their value cannot be overstated. The goal of summative assessments is to provide high-level information about student performance toward mastery of the state’s academic standards, and the only way to do that is to accurately measure what students have learned. Teacher involvement helps test developers ensure the right material is tested in ways that represent what is happening in the classrooms.

Teachers also have input at a much more granular level, which is equally important. What follows is a description of the life cycle of an item and where in the cycle teachers exert influence.

Stimuli Development

1. For literacy, the passage selection process is key to the future success of the assessment. Students should be provided with texts worthy of their time and attention while still being grade-level appropriate. After vendors have chosen potential texts and have put them through a thorough review for both quality and appropriate complexity, teachers review the texts at passage review meetings. If there are objections, that text won’t find a home on an assessment.
2. For math, programs differ in approach regarding stimuli, with some programs having educators review the stimuli before an item is developed and others reviewing the stimuli at the same time they review the item. But either way, teachers weigh in on clarity, accuracy, and quality for each stimulus that may get used on an assessment.

Item Development

Item creation is typically handled by testing vendors, although some states or consortia use in-state teachers to write items. The people hired by the testing vendors are usually seasoned writers with teaching experience at the grade level. Having classroom experience when writing items allows the writer to not only better address the standard (They designed their instruction around them!) but also to know what kids at that grade level can do (They taught them!). They also know what language kids are familiar with and the appropriate vocabulary to use (They interacted with them!).

The items are sent from item writers back to the vendor, with security being a top priority. All people involved in assessment sign a non-disclosure agreement to keep the material secure. Once the vendor receives the items, the questions go through a rigorous internal review process. Content specialists, who are very often former teachers, review the materials for alignment to standards, clarity, clear correct answers, etc., to make sure the questions are worth asking and will yield valuable information about student mastery of a skill. Then the item receives an editorial review. At this point, the state or consortium client (usually made up of former teachers) reviews the items to prepare to present them to teachers. There tends to be a lot of back and forth at this stage in a collaborative effort to get the items as close to perfect as possible.

Item Review

1. Once the items have been revised to incorporate the feedback collected, they are sent to committee review. That committee is composed of teachers from the state. Imagine a room full of teachers examining every item and commenting! Having participated on both the test developer side and as a teacher reviewer, I can tell you the process is both exhilarating and exhausting. It’s no small task to read questions that will end up on an assessment that is used for important decision making. It’s helpful, though, that the reviewers are trained on what to look for in each item, making sure the question is fair and worth asking.
2. After item review, the materials are edited based on teacher feedback. Then back through editorial they go! At that point, they are considered ready for possible selection as a field-test item.
3. After items have been field tested, meaning they’ve been “tried out” with students in an authentic setting so data can be gathered without counting against student scores, educators again have a role: they review the data looking for anything that sends up a red flag about the item. For example, did more students pick a single incorrect answer over what was indicated as the correct answer? That may mean either the item was miskeyed or all along the way reviewers were thinking differently about the item. It doesn’t necessarily always indicate there is something wrong with the question, as sometimes it may be just a hard item or testing a skill students struggle with. But it does require a close analysis by educators to make sure the item is valid.

Reviewers also focus on whether the item favored or disadvantaged a particular subgroup (e.g., males vs. females, White vs. African American). If an item is shown to favor or disfavor any particular group, the item is not used. This step of the process is referred to as “data review.” If the item is found to be valid, it is then eligible to go on an operational form and becomes part of the item bank from which future test administrations are pulled.

Operational Forms

Test developers then create operational forms using the items that have survived the various rounds of educator reviews. Test developers have to match the testing blueprint created much earlier in the process. Teachers weigh in on whether the form works as a whole: does it meet the statistical parameters for difficulty, the requirements for standard coverage, and other technical aspects, but also does it hold together as a whole, thus providing useful, valid information to educators, parents, and students? One thing teachers watch out for at this stage is “clueing,” where the answer to one item give away the answer to another. They also pay attention to item order to ensure not all the difficult items clump together to make the testing experience unbearable. For ELA, teachers look to ensure that the passages represent a nice balance of genres and topics to keep the students engaged. Based on the feedback of the educators, the forms are revised and then sent through the publications process, whether that is paper-and-pencil or computer delivery.

Depending on the state or consortium, there may be even more places in the cycle  where teachers are involved. I hope that knowing that your fellow educators have a part in every step of the process will offer you peace of mind that not only are the tests carefully scrutinized before being put in front of students, but also they are scrutinized by people who know kids, know the standards, and know classroom best practice. If you are interested in becoming part of the process, reach out to your state education agency about possible opportunities for involvement.