Who Scores the Keystone Exams - Not Teachers

The testing company that Pennsylvania uses for the Keystone exams and PSSA's is Data Recognition Corporation or DRC.  They create, distribute and score the Keystone exams and the PSSA's.  DRC hires employees they call 'scorers' to score the open ended portion of the exams. Open ended questions are non-multiple choice questions that require a person to read and score.

The only prerequisite for a potential 'scorer' is that they have proof of a 4 year degree.  Some implications are that these are people who are likely otherwise unemployed and are not required to have an education background or teaching experience.  

A company called “Glassdoor” has a web site that describes interviews for jobs from all types of companies, based on what interviewees reported to them. I have included comments posted on the Glassdoor web site of individuals who were interviewed for the position of scorer for the DRC.   You can read them online HERE and I urge you to do so. 

It seems to me that the standard for obtaining scorers is extremely low.  

According to John Weiss, Director, Pennsylvania Department of Education, Bureau of Assessment and Accountability:

Ten percent of the responses are independently read by two readers for the purpose of monitoring and maintaining inter-rater reliability.  Additionally, to ensure that the 90% of the responses that are read by one reader are scored reliably, pre-scored responses are randomly included to ensure that scorers have not drifted from the rubric.    

In such a high stakes test that is used as a graduation requirement, it would seem to be essential to have two graders grading every subjective question, and if the scores deviate, to bring in a third, particularly because, as was stated before, there are legitimate concerns as to the quality and credentials of the graders.  While this would be expensive, it would seem to be the fairest way to assure accuracy in grading.

Below are comments from DRC scorers from the Glassdoor WEBSITE:

Show up, present a college diploma, take a simple test on reading/writing/math skills and have a brief interview.

They want to know if you're capable of following orders like a good factory worker. That's all. Just say that you have no opinions on education and you'll just do whatever they say.

Take it or leave it. There is no negotiation for temporary work that, frankly, an intelligent 15 year old could do, but for which standards require a college degree.

Honestly, there was nothing you couldn't answer easily off the cuff, this is not applying to Harvard.   

It's kind of like working in a factory but instead of making widgets you are grading tests. 

They had a short presentation on the job. There were two short test (very short and easy). You had to take orginal degrees or transcripts of Bachelors degree or higher. Then there was a short one on one interview. You were then offered the job or not and told what assignment you were on.

The most frustrating part of this job is the tendency for the team leader to reinterpret the scores you gave to students in a completely different way that doesn't fit with the rubric and call it a "holistic" approach to scoring the test. This seemed to occur when the score matrix needed to skew in a slightly different direction.

You will leave at the end of each day for the first 2 - 3 weeks completely drained and not able to do much of anything that evening. The quiet scoring rooms coupled with the monotony of reading similar answers to the same question for hours will leave you exhausted. It can be quite boring.

Occasional death marches. Depressingly sterile office. A lot of turnover. Mediocre technology. Many weak hires. Uneven management.

This is a repetitive job that can be less than stimulating but some of the kids responses to essay questions can make you laugh.

People are appointed to group leaders and scoring directors based on longevity rather than talent or education.

Very sedentary working environment that involves staring at a screen reading bad handwriting all day is hard on the body.

Brought in batches to be tested for middle school intelligence in a computer lab. Taken in for individual interviews just to basically confirm we had a pulse, didn't forge our diplomas, and weren't a threat to others or ourselves. Hired on the spot.

Interview Question – x+1=2   

Come in to the office, listen to a presentation, and take two written assessments. One is a pre-algebra level math test, and the other is a writing sample, my prompt was "describe your best achievement in the last two years". After turning in your assessments, you talk one on one with the HR rep for a short (10 minute) interview where she basically wants to see if you will be comfortable with the close quarters of the working environment, the repetition and monotony of grading, and if you can be consistently impartial. If you're smart you just say yes.

Honestly, there was nothing you couldn't answer easily off the cuff, this is not applying to Harvard.   

There is no negotiation.  It's a temporary FT job.  Salary is 13.00 or 14.25 if you complete 40 hours of scoring

These are the people who score the Keystone exams that determines if our children will graduate.  The results of the PSSA and Keystone exams are used to evaluate our teachers and schools as successful or failing.