The Professional Educator Standards Board (PESB) contracted with ETS because of its standards for quality assurance. Controls exist in the scoring process to manage the quality of scores produced. Test validity, reliability and fairness are of foremost importance. Scoring adheres to the highest industry standards for evidence-based assessments that are required for high-stakes licensure. Throughout the scoring process, steps are taken to control for any inherent biases that may impact scoring. Scorers are trained to recognize when a personal, societal or professional bias might interfere with their ability to fairly score a response. Through adherence to industry standards, candidates can be confident that portfolio scoring is conducted to achieve the highest levels of fairness and reliability.
Fair and Unbiased Scoring
There are numerous checks in place to provide fair and valid scores. Scorers are required to participate in a rigorous training program that includes demonstration of understanding of the standards, criteria, entry directions, rubrics and more. Scorers must demonstrate mastery of the scoring process through multiple practice sessions conducted by experts who are trained in qualifying scorers. Scorers must take and pass a test verifying their mastery of accurate scoring processes.
All identifying information is removed from responses so scorers are prevented from knowing candidate identity. For resubmitted entries, scorers are not aware of any previous scores. They will not know that any entry is a resubmission, nor will they have access to any prior candidate scores.
Scorer agreement is a desirable goal for all evidence-based assessments scored using rubrics. The ProTeach Portfolio is scored using a 4-point rubric. It is important that different scorers who have been trained to score candidate responses closely agree in the scores they assign to the same candidate's response. In other words, the score a candidate receives should not be dependent on which particular trained scorer happens to score the response. ETS will compute the extent of scorer agreement on the ProTeach Portfolio using a Kappa statistic, which ranges from 0 (no rater agreement) to 1 (complete rater agreement). The significance of the Kappa value is evaluated by comparing that value to the value that would be expected by a chance level of agreement. If the Kappa value exceeds what would be expected by chance, it may be concluded that the extent of rater agreement is statistically significant.
A Word of Caution
Each entry in your ProTeach Portfolio, whether initial or resubmitted, must be entirely your work. While we encourage collaborative teaching and utilizing a support provider, each entry must be distinctly your own work. Software is utilized to scan all written commentary for overlap with previous submissions or with another candidate's submissions. If such overlap is detected, scores will be withheld and a review by the ETS Office of Testing Integrity (OTI) will be initiated.
If your portfolio is the subject of such a review, the following steps will occur:
- Your scores will be placed on hold.
- On, or before, the scheduled score release date you will be informed that your portfolio is under review, and you will be provided the opportunity to submit additional materials to ETS's Office of Testing Integrity (OTI) to support your case.
- Members of ETS's Board of Review will independently review all materials and provide an independent assessment of the case to ETS's OTI.
- You will receive a letter from ETS's OTI notifying you of the Board of Review's decision after the review has been completed.
- If one or more of the members of the Board of Review conclude that your scores should be released, the hold on your scores will be removed and you will be able to view your score report online.
- If the Board of Review arrives at a unanimous consensus that there is substantial evidence to support cancellation of your scores, the scores will be voided and all materials will be forwarded to Professional Educator Standards Board (PESB) (WAC 181-87-050, Misrepresentation or Falsification in the Course of Professional Practice) for further review.
- PESB will review OTI's results to determine if there is sufficient evidence of unprofessional practice to request an investigation by Office of Professional Practice (OPP) in accordance with WAC 181-87-050.
You can learn more about the specifics of scoring by clicking on the following links: