Keystone Exams - Cut Scores Exposed

Thank you to Joanne Yurchak who provided much of the information on this page.  More detailed information can be found in her EMAIL CORRESPONDENCE with the Pennsylvania Department of Education. 

_________________________________________________________________________________

The Keystone Exams (and PSSA's) are not graded in the typical way. 85/100 does not equal 85% or a B grade.  In fact a passing score, or proficient, is determined well after a student takes the exam, and after the exam is scored!   A group of individuals (who might or might not have a background in education) put the statewide student scores in order, collaborate and decide subjectively where to put the score range.  That is why they use Proficient, Advanced or Basic in scoring; these words cover the range that is not determined until after the tests are scored.  

The range can be placed so that the scores are high and many students are Proficient, or the range can be placed lower so the scores are poor and many students are Basic or Below Basic and "need" remediation.  In other words, the range can be placed wherever - it is subjective, and can be manipulated and controlled.  As a result, tests scored in this manner are not an accurate portrayal of a students academic abilities.

In Pennsylvania, students take the Keystone Exams (and the PSSA's) in May and school districts get the student scores by the second week of July.  Parents and students do not receive their scores until the first week of September.  Why the 4 month delay?

Per PA School Code Chapter 4.51b (g)

The Department will use widely-accepted psychometric procedures to establish the cut scores. Cut scores shall be presented at a public meeting of the Board for its review at least 2 weeks prior to scheduled Board action on the cut scores.

Below is the process for the spring Keystone exams:

• Keystone Exam taken in May

• Multiple choice questions are graded electronically; open-ended questions are graded by temporary employees hired by DRC.

• School Districts receive student scores by the 2nd week of July.

• A group of selected individuals will collaborate and determine where the range will fall, also known as a cut score.    

• Students who do not score Proficient or Advanced can take the test again or might be placed in a remedial class or Keystone focused study hall (and miss out on elective classes).  

This is the process that determines whether Pennsylvania students graduate!  

I'll first clarify the testing company and then will describe the process of scoring the Keystone Exams. 

The testing company that Pennsylvania uses for the Keystone Exams (and PSSA's) is Data Recognition Corporation or DRC.  They create, distribute and score the Keystone Exams (and the PSSA's).  DRC hires employees they call 'scorers' to score the open ended portion of the exams.  Open ended questions are non-multiple choice questions that require a person to read and score.

The only prerequisite for a potential 'scorer' is that they have proof of a 4 year degree.  Some implications are that these are people who are likely otherwise unemployed and are not required to have an education background or teaching experience.  

A company called “Glassdoor” has a web site that describes interviews for jobs from all types of companies, based on what interviewees reported to them. I have included two comments posted on the Glassdoor web site of individuals who were interviewed for the position of scorer for the DRC.   A total of thirteen comments are posted on the site for DRC; I copied two for scorers and have included them below. Other comments from interviews at DRC are similar.  You can read them online HERE and I urge you to do so.  It seems to me that the standard for obtaining scorers is extremely low.  

Data Recognition Corporation Interview Questions & Reviews from Glassdoor

Mar 11, 2014 Accepted Offer
Neutral Experience
Very Easy Interview Scorer Interview <http://www.glassdoor.com/Interview/Data-Recognition-Corporation-Interview-RVW3849561.htm> (Neutral Experience; Very Easy Interview)

Anonymous EmployeeAustin, TX I applied online and the process took a day - interviewed at Data Recognition Corporation in March 2013. Interview Details – Brought in batches to be tested for middle school intelligence in a computer lab. Taken in for individual interviews just to basically confirm we had a pulse, didn't forge our diplomas, and weren't a threat to others or ourselves. Hired on the spot. Interview Question – x+1=2   View Answer <http://www.glassdoor.com/Interview/x-1-2-QTN_625329.htm>
_______________________________________________________________________________________________

Jul 10, 2013
Accepted OfferPositive ExperienceVery Easy Interview Test Scorer Interview <http://www.glassdoor.com/Interview/Data-Recognition-Corporation-Interview-RVW2808752.htm> (Positive Experience; Very Easy Interview)

Anonymous EmployeeAustin, TX I applied through college or university and the process took a day - interviewed at Data Recognition Corporation in April 2013. Interview Details – Come in to the office, listen to a presentation, and take two written assessments. One is a pre-algebra level math test, and the other is a writing sample, my prompt was "describe your best achievement in the last two years". After turning in your assessments, you talk one on one with the HR rep for a short (10 minute) interview where she basically wants to see if you will be comfortable with the close quarters of the working environment, the repetition and monotony of grading, and if you can be consistently impartial. If you're smart you just say yes.

Interview Question – Honestly, there was nothing you couldn't answer easily off the cuff, this is not applying to Harvard.   View Answer <http://www.glassdoor.com/Interview/Honestly-there-was-nothing-you-couldn-t-answer-easily-off-the-cuff-this-is-not-applying-to-Harvard-QTN_487582.htm> 
_________________________________________________________________________________________

As far as pay, this is from a Glassdoor interview response:  

Negotiation Details – There is no negotiation. Its a temporary FT job. Salary is 13.00 or 14.25 if you complete 40 hours of scoring.

How many scorers are there per question to ensure consistency and a check for accurate scoring?

According to John Weiss, Director, Pennsylvania Department of Education, Bureau of Assessment and Accountability:

Ten percent of the responses are independently read by two readers for the purpose of monitoring and maintaining inter-rater reliability.  Additionally, to ensure that the 90% of the responses that are read by one reader are scored reliably, pre-scored responses are randomly included to ensure that scorers have not drifted from the rubric.    

In such a high stakes test that is used as a graduation requirement, it would seem to be essential to have two graders grading every subjective question, and if the scores deviate, to bring in a third, particularly because, as was stated before, there are legitimate concerns as to the quality and credentials of the graders.  While this would be expensive, it would seem to be the fairest way to assure accuracy in grading.

Summarizing, the testing company DRC electronically scores the multiple choice portion of the exams and hires temporary scorers for the open-ended portions.  Cut scores are used for scoring the Keystone Exams.  The subjectively determined cutoffs establish how many students pass the Keystone (scoring either Proficient or Advanced).

Here again is the process according to John Weiss, Director, Pennsylvania Department of Education, Bureau of Assessment and Accountability:

After the initial administration of an assessment, the tests are scored and given a scaled score.  The tests are then arranged in order of scores.  Pennsylvania educators then participate in a Standard Setting activity where they determine the cut off points between each performance level (Advanced, Proficient, Basic, and Below Basic).  The range of scaled scores is then determined to identify each performance level.  Once established, these ranges do not change.