Keystone Exam Information and Q&A

Questions and Answers Regarding Pennsylvania’s Keystone Exams

From the year 2017 and thereafter, testing Proficient or Advanced in three Keystone exams (Algebra I, Language Arts, and Biology) will be a high school graduation requirement for all PA students.  There is a great deal of information on the PDE’s web site concerning these exams.  The web site: http://optoutpa.blogspot.com/ also provides invaluable and detailed information on both the Keystones and PSSA’s.  The questions answered below should be helpful for understanding some aspects of the Keystones that are not normally publicized.  They specifically focus on the Algebra I Keystone, but answers are similar for the other Keystones.  Thank you to John Weiss of the PDE for providing many valuable insights and clarifications.

Question: Can a student graduate from high school if s/he is unable to pass a Keystone?  What are the effects on students and school districts from circumventions of this graduation requirement?

Answer: Yes. There are avenues that the PDE has put in place that enable students who cannot pass the Keystones to graduate; however, these circumventions take an insurmountable toll on both students and school districts.  Students can retake the Keystone any number of times to try to test Proficient.  (Note that this is quite lucrative for the test-making companies!)  Once they've taken it twice and still can’t test Proficient, students can then take what's called a "Project Based Assessment”[1]. However, the time involved in these processes and the various types of remediation that frequently occur simultaneously, preclude students from taking courses that are more essential to their chosen career goals, thus inhibiting them from reaching their individual maximum educational potentials.  (Children aren’t cookies made from the same cookie cutter. “One-size-fits-all” does NOT apply!)  The mental and physical stress inflicted on many students during this process is well-documented and would be avoidable by abolishing this requirement.  Last, but certainly not least, is the enormous fiscal impact on school districts, who are already financially strapped.  The huge expenditures required by school districts to comply with this unfunded mandate could be utilized in far more effective ways in improving our educational system.

Question: What types of questions are on the Algebra I Keystones?

Answer: There are MULTIPLE CHOICE questions (fill in the blank, and graded by machines), and SUBJECTIVE questions (AKA “constructed response”) which are graded by scorers and for which students have to show their work.

Question: How are the Algebra I questions proportioned between MULTIPLE CHOICE and SUBJECTIVE (AKA “constructed response”)?

Answer: There are 36 multiple choice (1 point each) and 6 subjective questions (4 points each) for every exam.  This adds up to a total of 60 points.  The subjective questions (24 pts.) comprise 40% of the exam score.

Question: Who grades the subjective questions?

Answer: Temporary employees hired by Data Recognition Corporation (DRC - the company contracted to handle PA’s assessments) hand-grade the six, 4-point subjective questions which account for 40% of the overall score (Algebra I).  Scorers are provided with rubrics for this task.

Question: How many scorers grade each subjective question?

Answer: 90% of the subjective questions are graded by only one scorer, 10% by two scorers.  

Think of this… 90% of the subjective questions that determine 40% of the final score on this high stakes test are graded by only one scorer!

Note: For such a high stakes test, it would seem to be essential to have two graders grading every subjective question, and if the scores deviate, to bring in a third, particularly because, as will be indicated, there are legitimate concerns as to the quality and credentials of the graders.  While this would be expensive, it would seem to be the fairest way to assure accuracy in grading.

Question: What are the qualifications of the scorers of the subjective questions?

Answer: Although the PDE stands by, and supports what they SAY are appropriate qualifications and training of scorers, there are legitimate concerns regarding the manner in which they are hired by DRC.  First of all, these scorers are temporary employees that earn between $11-14/hour.  Their only required qualification seems to be a four year degree, which could be virtually meaningless, considering what is posted on-line by interviewees commenting on the interview process.  Although the PDE insists that “numerous quality control measures are implemented to monitor and maintain accuracy and reliability” during the scoring of the subjective questions, the comments below [2]: certainly don’t allay the legitimate concerns regarding the suitability of scorers who will be determining 40% of the grade in such a high-stakes test.  Four of these comments are:

       

·      “Brought in batches to be tested for middle school intelligence in a computer lab. Taken in for individual interviews just to basically confirm we had a pulse, didn't forge our diplomas, and weren't a threat to others or ourselves. Hired on the spot. (Sample Interview Question – x+1=2)”

·      “We were asked to write and(sic) essay and complete some math problems to demonstrate proficiency. We were then interviewed in the order we finished. The interview was short and just asked basic questions about why you were qualified for the job. You were also asked how you would be able to handle a job that required you to sit and read responses for eight hours.  Interview Question – The questions were very basic. You just have to prove you have the credentials and focus to do the scoring.”

·      “Come in to the office, listen to a presentation, and take two written assessments. One is a pre-algebra level math test, and the other is a writing sample, my prompt was ‘describe your best achievement in the last two years’. After turning in your assessments, you talk one on one with the HR rep for a short (10 minute) interview where she basically wants to see if you will be comfortable with the close quarters of the working environment, the repetition and monotony of grading, and if you can be consistently impartial. If you're smart you just say yes. Interview Question – Honestly, there was nothing you couldn't answer easily off the cuff, this is not applying to Harvard.”

·      “It was a group interview. They would then take each person individually to a room and ask them what their area of expertise was, such as English, Math or Social Sciences. They would then place you in whatever room best fit your expertise in order to score the state's standardized test scores. Interview Question – Where do you see yourself in five years?”

Question: After the Keystone is scored and results have been obtained, can a parent review the test?

Answer: Yes, upon approval by the PDE.  For a processing fee of $75, the parent can view the student’s Test Booklet, but cannot see the questions to which the student’s answers refer. 

Question: Can a parent have their child’s test rescored?

Answer: Yes, they can, for a fee of $75. According to John Weiss of the PDE, “A parent would request a rescore following the same procedure as a request for the student’s response.  The rescore process costs $75.00 and includes a manual verification that the multiple choice items were captured correctly, as well as a rescoring of all of the student’s constructed responses.  Each step is verified by a second person.  The results of the rescore are documented and forwarded to the school/district.”

Question: What are the different sections and raw score point distributions for the Algebra I Keystone?

Answer: There are TWO MODULES, each of which is worth 30 points.  (Remember that the total number of points for the raw score is 60, with 36 points for multiple choice questions and 24 points for the subjective questions.)  Below is the distribution:

Module 1: Operations and Linear Equations and Inequalities: (30 pts.)

Operations with Real Numbers and Expressions (9 pts.)

Linear Equations (10 pts.)

Linear Inequalities (11 pts.)

                      Module 2: Linear Functions and Data Organizations: (30 pts.)

Functions (10 pts.)

Coordinate Geometry (10 pts.)

Data Analysis (10 pts.)

Question: What are the scale scores that show the ranges that indicate whether a student has tested Below Basic, Basic, Proficient or Advanced?  Note that Below Basic and Basic are not passing.)

Answer: The following chart shows the scale scores.  These scale score ranges have been set arbitrarily; they range from 1200-1800, just as SAT scores arbitrarily range from 200-800.  .  While the raw score to scale score correspondence (AKA “cut scores”) might change slightly from one administration to another, the scale score ranges designating the cutoffs (shown below) remain the same for every administration of a Keystone and do not change from year to year.  In other words, 1500 and above is always passing for each of the three Keystones.

 Keystone Exam Scale Score Ranges

Question: Since there are two modules, can a student score Proficient on one module and Basic on another and still test Proficient overall?

Answer: Yes, if the raw scores of the two modules added together produce a total raw score that corresponds to 1500 or above on the scale score.

Question: If a student tests Proficient on one module and Basic on the second module, does s/he have to redo both modules in a retest?

Answer: No, the student only has to retake the module that s/he did not pass.  However, Mr. Weiss of the PDE stated: “It may be in the student’s interest to retake both modules since a higher score on the passed module with no improvement on the other module could result in a total test score (highest total test scale score to date) that is Proficient or above.”

Question: How are the raw scores (60 pts maximum) converted to the scale scores ranging from 1200 – 1800?  Is the raw score to scale score correspondence (AKA “cut scores”) the same for every test administration?

Answer: According to Mr. Weiss, the raw score to scale score correspondence is not the same for every test administration.  He explained: “Raw scores or percent correct scores cannot be used to compare students’ achievement across administrations because they depend on the difficulty of the tests.  Raw scores are converted to scale scores through the process of equating and scaling.  Equating is a statistical process used to establish comparability between scores from different test administrations.  Equated test scores can be used interchangeably even though the test forms consist of different items.”

Question: Shouldn’t we be concerned that the raw score to scale score correspondence (AKA “cut scores”) could be manipulated and subjectively placed in order to control the percent of students who pass?

Answer: According to Mr. Weiss, this does not happen.  He said: “Keystone technical reports, which include raw to scale conversion tables, are posted online on PDE’s website. Within that document, you can see that the equating and scaling procedures are reviewed and approved by Pennsylvania’s technical advisory committee (TAC), strictly followed, and replicated prior to reporting.”  Also, according to PA School Code Chapter 4.51b (g), the PDE “will use widely-accepted psychometric procedures to establish the cut scores. Cut scores shall be presented at a public meeting of the Board for its review at least 2 weeks prior to scheduled Board action on the cut scores.”

Question: What company handles the testing procedures for Pennsylvania?  What are the costs?

Answer: The testing company that Pennsylvania contracted with from 2009 – 2015 to create, distribute and score the Keystone Exams and the PSSA's, is Data Recognition Corporation or DRC.  As was discussed previously, DRC hires the temporary scorers to score the subjective portions of the exams.  The 2009-2015 contract with DRC involved $200+ million.  It was rumored that PA was going to change to a company called Measured Progress after the DRC contract ran out, but they have extended DRC’s contract through 2016 instead.

Question: How much time do students have for the Algebra I Keystone?

Answer: Estimated time for multiple choice questions is 1.5 minutes/question; estimated time for subjective questions is 10 minutes/question.  The estimated total test time is therefore approximately two hours.  However, students can take as much time as is necessary to complete the exam; it is essentially “untimed.”

Question: When are the May Keystone results available to school districts and to students?

Answer: School districts receive scores by the second week in July, but many districts do not make these results available to students until the first week in September.  This delay can be problematic since scheduling often has to be readjusted for students who have not demonstrated proficiency in a given Keystone, either because they might have to retake the course, or because they will have to adjust their schedules for remediation. 

Question: What information is provided in the report that is given to the student to show his/her results on the Keystone exam?  Is it helpful in pointing out his/her specific strengths and weaknesses in the subject material?

Answer: A sample report is given below.  It is very sketchy and provides negligible feedback for enabling students and parents to understand the student’s strength and weaknesses in various areas.  It gives only the scale score for the two modules; there is no breakdown for the six sub-sections (3/module).  This unfortunately is an indication that the educational aspects of the test results are secondary to the primary purpose of the test, which seems to be its use in evaluating teachers and schools.  Mr. Weiss has stated that: “Raw score by Assessment Anchor is provided for each student receiving a performance level. This detailed information is provided in the District Student Data Files (DSDF).   Along with the DSDF, a file layout is posted that has the total points possible by Assessment Anchor. Similar to the Performance Summary by Assessment Anchor, this breakdown is based on the current test administration.”  If this is the case, it is hard to understand why the student’s report doesn’t include these data.

Prepared by Joanne Yurchak (jyurchak@comcast.net) with the help of Dawn Sweeney (http://optoutpa.blogspot.com/), and with some information provided by John Weiss of the Pennsylvania Department of Education.

---

[1] http://static.pdesas.org/Content/Documents/Keystone%20Project%20FAQ.pdf

---

[2] http://www.glassdoor.com/Interview/Data-Recognition-Corporation-Interview-Questions-E233679.htm