The testing company that Pennsylvania uses for the Keystone exams and PSSA's is Data Recognition Corporation or DRC. They create, distribute and score the Keystone exams and the PSSA's. DRC hires employees they call 'scorers' to score the open ended portion of the exams. Open ended questions are non-multiple choice questions that require a person to read and score.
The only prerequisite for a potential 'scorer' is that they have proof of a 4 year degree. Some implications are that these are people who are likely otherwise unemployed and are not required to have an education background or teaching experience.
A company called “Glassdoor” has a web site that describes interviews for jobs from all types of companies, based on what interviewees reported to them. I have included comments posted on the Glassdoor web site of individuals who were interviewed for the position of scorer for the DRC. You can read them online HERE and I urge you to do so.
It seems to me that the standard for obtaining scorers is extremely low.
According to John Weiss, Director, Pennsylvania Department of Education, Bureau of Assessment and Accountability:
Ten percent of the responses are independently read by two readers for the purpose of monitoring and maintaining inter-rater reliability. Additionally, to ensure that the 90% of the responses that are read by one reader are scored reliably, pre-scored responses are randomly included to ensure that scorers have not drifted from the rubric.
In such a high stakes test that is used as a graduation requirement, it would seem to be essential to have two graders grading every subjective question, and if the scores deviate, to bring in a third, particularly because, as was stated before, there are legitimate concerns as to the quality and credentials of the graders. While this would be expensive, it would seem to be the fairest way to assure accuracy in grading.
Below are comments from DRC scorers from the Glassdoor WEBSITE:
Show up, present a college diploma, take a simple test on reading/writing/math skills and have a brief interview.
They want to know if you're capable of following orders like a good factory worker. That's all. Just say that you have no opinions on education and you'll just do whatever they say.
Take it or leave it. There is no negotiation for temporary work that, frankly, an intelligent 15 year old could do, but for which standards require a college degree.
Brought in batches to be tested for middle school intelligence in a computer lab. Taken in for individual interviews just to basically confirm we had a pulse, didn't forge our diplomas, and weren't a threat to others or ourselves. Hired on the spot.
Interview Question – x+1=2
Come in to the office, listen to a presentation, and take two written assessments. One is a pre-algebra level math test, and the other is a writing sample, my prompt was "describe your best achievement in the last two years". After turning in your assessments, you talk one on one with the HR rep for a short (10 minute) interview where she basically wants to see if you will be comfortable with the close quarters of the working environment, the repetition and monotony of grading, and if you can be consistently impartial. If you're smart you just say yes.