Cut Score Email Correspondence

E-MAIL CORRESPONDENCE BETWEEN JOANNE YURCHAK AND JOHN WEISS RE: VARIOUS QUESTIONS RELATING TO THE IMPLEMENTATION AND GRADING OF KEYSTONE EXAMS (PARTICULARLY ALGEBRA I)

Note that Mr. John Weiss is with the PA Dept. of Education.  His titles are: “Director of Bureau of Assessment and Accountability” and “Acting Director of the Bureau of Teaching and Learning.”  I, Joanne Yurchak (JYY), initially contacted him after having read on the Internet about what I considered to be a disturbing manner in which the scorers for the subjective questions of the Keystones were hired by DRC (Data Recognition Corporation).  The E-Mail “conversations” expanded to multiple questions regarding the grading and implementation of the Keystones, particularly focusing on the Algebra I test.  Although I still have serious issues with the grading and implementation of the Keystones, I greatly appreciated Mr. Weiss’ prompt, respectful and thorough responses to my multiple questions in our series of E-Mail correspondences.

 

May 17, 2014 E-Mail from JYY to John Weiss

From: Joanne [mailto:jyurchak@comcast.net]
Sent: Saturday, May 17, 2014 9:24 PM
To: Weiss, John (PDE)
Cc: Senator Andrew Dinniman; Senator Mike Folmer; Representative Dan Truitt
Subject: Manner in which scorers are hired for Keystone and PSSA exams

Note: The question I would respectfully request an answer from you is in red font below.  Please scroll down.

 Hello Mr. Weiss,

I've been researching the manner of grading of the open-ended questions on the Keystone exams.  As I'm sure you know, a significant portion of the Keystone exams are open-ended (non-multiple choice) questions.  I have been told that 25-40% of the questions on the Algebra I Keystone are open-ended.  Data Recognition Corportation (DRC) recruits scorers for these open-ended questions on both the Keystone and PSSA exams.  The following information describing how scorers are selected is given on p. 84 of the PDE Technical Report (www.portal.state.pa.us/.../2011_keystone_exams_technical_report_pdf).   The report is written for the 2011 year but I assume that current procedures are similar if not identical.

READER RECRUITMENT AND QUALIFICATIONS

DRC retains a number of readers from year to year. This pool of experienced readers was used to staff the approximate 273 readers who were needed for the 2011 scoring season. To complete the reader staffing for this project, DRC placed advertisements in local papers and also utilized a variety of websites. Open houses were held and applications for reader positions were screened by the DRC recruiting staff. Candidates were personally interviewed and a mandatory, on-demand writing sample, plus a mathematics sample for those applying to score mathematics, was collected, along with references and proof of a four-year college degree. In this screening process, preference was given to candidates with previous experience scoring large-scale assessments and with degrees emphasizing expertise in the subjects being scored. Since readers had to have a strong content-specific background, the reader pool consisted of educators and other professionals who were valued for their experience, but who were also required to set aside their own biases about student performance and accept the scoring standards.

 A company called “Glassdoor” has a web site that describes interviews for jobs from all types of companies, based on what interviewees reported to them. I have included four comments posted on the Glassdoor web site of individuals who apparently interviewed for the position of scorer for the DRC.   A total of thirteen comments are posted on the site for DRC; I just copied the ones for scorers and have included them below.  Other comments from interviews at DRC are similar.  You can read them on line and I urge you to do so.  http://www.glassdoor.com/Interview/Data-Recognition-Corporation-Interview-Questions-E233679.htm   (Information regarding Glassdoor is at the end of this E-Mail.) 

I would greatly appreciate your opinion with justification, if possible, of the accuracy of the comments.  It certainly seems to me that the standard for obtaining scorers is extremely low.

Data Recognition Corporation Interview Questions & Reviews from Glassdoor

Mar 11, 2014

Accepted Offer

Neutral Experience

Very Easy Interview

Scorer Interview (Neutral Experience; Very Easy Interview)

Anonymous Employee

Austin, TX

I applied online and the process took a day - interviewed at Data Recognition Corporation in March 2013.

Interview Details – Brought in batches to be tested for middle school intelligence in a computer lab. Taken in for individual interviews just to basically confirm we had a pulse, didn't forge our diplomas, and weren't a threat to others or ourselves. Hired on the spot.

Interview Question – x+1=2   View Answer

Mar 11, 2014

Accepted Offer

Positive Experience

Easy Interview

Test Scorer Interview (Positive Experience; Easy Interview)

Test Scorer

Pittsburgh, PA

I applied in-person and the process took a day - interviewed at Data Recognition Corporation in February 2014.

Interview Details – We were asked to write and essay and complete some math problems to demonstrate proficiency. We were then interviewed in the order we finished. The interview was short and just asked basic questions about why you were qualified for the job. You were also asked how you would be able to handle a job that required you to sit and read responses for eight hours.

Interview Question – The questions were very basic. You just have to prove you have the credentials and focus to do the scoring.   Answer Question

Negotiation Details – There is no negotiation. Its a temporary FT job. Salary is 13.00 or 14.25 if you complete 40 hours of scoring.

Feb 19, 2014

Accepted Offer

Positive Experience

Easy Interview

Test Scorer Interview (Positive Experience; Easy Interview)

Anonymous Employee

Plymouth, MN

I applied through a staffing agency - interviewed at Data Recognition Corporation in February 2014.

Interview Details – It was a group interview. They would then take each person individually to a room and ask them what their area of expertise was, such as English, Math or Social Sciences. They would then place you in whatever room best fit your expertise in order to score the state's standardized test scores.

Interview Question – Where do you see yourself in five years?   Answer Question

Negotiation Details – No negotiation

Jul 10, 2013

Accepted Offer

Positive Experience

Very Easy Interview

Test Scorer Interview (Positive Experience; Very Easy Interview)

Anonymous Employee

Austin, TX

I applied through college or university and the process took a day - interviewed at Data Recognition Corporation in April 2013.

Interview Details – Come in to the office, listen to a presentation, and take two written assessments. One is a pre-algebra level math test, and the other is a writing sample, my prompt was "describe your best achievement in the last two years". After turning in your assessments, you talk one on one with the HR rep for a short (10 minute) interview where she basically wants to see if you will be comfortable with the close quarters of the working environment, the repetition and monotony of grading, and if you can be consistently impartial. If you're smart you just say yes.

Interview Question – Honestly, there was nothing you couldn't answer easily off the cuff, this is not applying to Harvard.   View Answer

Negotiation Details – Hourly wages are non-negotiable.

Information on Glassdoor.com from their web site:

Website www.glassdoor.com

Headquarters Sausalito, CA

Size 150 to 499 Employees

Founded 2007

Type Company - Private

Industry Information Technology

Competitors Indeed, Simply Hired, LinkedIn

Glassdoor is a social jobs and career community that is changing the way people find jobs and companies recruit top talent. Founded in 2007, Glassdoor offers members access to the latest job listings, the ability to see Inside Connections™ via their Facebook network, and get access to proprietary user-generated content including company-specific salary reports, ratings and reviews, CEO approval ratings, interview questions and reviews, office photos, and more. Plus, employers can get involved in the conversation through Glassdoor’s suite of social recruiting solutions to reach target job candidates when they’re making career decisions. Glassdoor is backed by Benchmark Capital, Sutter Hill Ventures, Battery Ventures and DAG Ventures.

Mission: To help people everywhere find jobs and companies they love.

 

May 19, 2014 E-Mail from John Weiss to JYY

From: "Weiss, John" <jweiss@pa.gov>
To: "Yurchak, Joanne" <jyurchak@comcast.net>
Cc: "Dinniman, Senator Andrew" <andy@pasenate.com>, "Folmer, Senator Mike" <mfolmer@pasen.gov>, "Truitt, Representative" <dtruitt@pahousegop.com>, "Aaron Shenck" <ashenck@pa.gov>
Sent: Monday, May 19, 2014 1:17:35 PM
Subject: RE: Manner in which scorers are hired for Keystone and PSSA exams

You have asked for my opinion of the comments from individuals who interviewed to be scorers for DRC.  Based upon the comments of the interviewees, you have drawn the conclusion that the standards for obtaining scorers is extremely low.  My interpretation of the comments is that the interview process was relatively simple, however, that is not a reflection on the standard of the individuals hired to score the assessments.  Remember that to even qualify for an interview, potential scorers must have a minimum of a four-year degree.  During the interview process, candidates must provide an on-demand writing or math sample to demonstrate fluency in the skills that they may be assessing, and they are evaluated during one-on-one interviews.

Those who are hired are fully trained and qualified before they can serve as scorers.  Scorers are thoroughly trained using DRC’s training and qualifying processes, which are ISO certified. Training and qualifying are conducted on every item that is scored.  After going through training, potential scorers must pass a qualifying test before they are allowed to score.  Scorers who do not qualify are removed from the project and do not score any actual student responses. 

During scoring, numerous quality control measures are implemented to monitor and maintain accuracy and reliability. If a scorer demonstrates inaccurate scoring, he or she is re-trained or removed from the project, and the scores that he or she gave are removed so that the responses may be accurately re-scored.

I appreciate your concern to the quality of the scoring of PA’s assessments, and I can assure you that I have been to scoring sites where I’ve observed scoring, and I have the utmost confidence that the scorers who are hired by DRC are adequately trained and monitored to ensure a true assessment of a student’s open ended response.

John Weiss | Director, Department of Education | Bureau of Assessment and Accountability

Acting Director | Bureau of Teaching and Learning

333 Market Street | Harrisburg, PA  17126

Phone:  717.214.4394  |  Fax:  717.705.8021

www.education.state.pa.us      

 

Combination of May 20, 2014 E-Mail from JYY to John Weiss and

response from John Weiss to JYY on June 17, 2014 written in blue font below questions.

Note that the delay in the response from Mr. Weiss was due to an honest mistake in E-Mail communications on the part of the PDE.

From: "Weiss, John" <jweiss@pa.gov>
To: "Yurchak, Joanne" <jyurchak@comcast.net>
Cc: "Dinniman, Senator Andrew" <andy@pasenate.com>, "Folmer, Senator Mike" <mfolmer@pasen.gov>, "Truitt, Representative" <dtruitt@pahousegop.com>, "Aaron Shenck" <ashenck@pa.gov>, "Dumaresq, Carolyn" <cdumaresq@pa.gov>
Sent: Tuesday, June 17, 2014 9:29:54 AM
Subject: RE: Questions on Keystone Exams re: their grading and administration

Good morning, Ms. Yurchak,

My answers follow your questions below.  I’m not sure that any response will allay your concerns, but I can assure you that the Department of Education takes very seriously the validity and reliability of the Keystone Exams results.  Again, I appreciate your patience in awaiting this response.    

 John Weiss

From: "Yurchak, Joanne" <jyurchak@comcast.net>
To: "Weiss, John" <jweiss@pa.gov>
Cc: "Dinniman, Senator Andrew" <andy@pasenate.com>, "Folmer, Senator Mike" <mfolmer@pasen.gov>, "Truitt, Representative" <dtruitt@pahousegop.com>, "Aaron Shenck" <ashenck@pa.gov>
Sent: Tuesday, May 20, 2014 12:33:52 AM
Subject: Re: Manner in which scorers are hired for Keystone and PSSA exams

Hello Mr. Weiss:

Thank you very much for responding so promptly to my question re: the manner in which scorers are hired for the Keystone and PSSA exams.  You seem to be the right person to ask, so I have some additional questions re: the Keystones:

1.   Are parents of students who do not score proficient in a Keystone allowed to review their child's test with the corresponding rubric? 
Parents are not allowed to review their child’s test.  A certain number of questions on a Keystone Exam are used on the next administration of the exam to allow for equating, thus allowing parents/students to see those items with their corresponding answers would result in a violation of test security. 

 2.   How many scorers are there/question?  Is there only one, or is the score an average of more than one grader?
Ten percent of the responses are independently read by two readers for the purpose of monitoring and maintaining inter-rater reliability.  Additionally, to ensure that the 90% of the responses that are read by one reader are scored reliably, pre-scored responses are randomly included to ensure that scorers have not drifted from the rubric.    

3.   Are the norms determined prior to giving the test or afterwards?
I’m not sure I understand what you mean by “norms.” 

 4.   Are Keystones given throughout PA identical for a given time period?  I ask this question, because I know that different school districts are giving the Algebra I Keystone AT LEAST two weeks from one another.  If the questions are identical, there is no question that the test's integrity would be compromised.  Kids remember questions, particularly ones that they've struggled with, and particularly when they essentially don't have a time limit.  Although they are not supposed to discuss the questions with others, in this day and age of the Internet and social media, I assure you...it WILL happen.  I just hope that you have prepared different exam questions for tests that are given at varying times. 

The Keystone Exams that are administered in one of the administration windows contain the same questions, and you are correct that the concern about students sharing information exists.  The Department of Education has implemented a very strict test security policy/process, but in the event that the Department recognizes a significant violation, we have produced a “breach” form to be administered.   

Thank you again for your prompt response to my initial question.  I have to say that I think total fairness and equality of grading of open-ended questions is difficult to attain, although it is essential in such a high-stakes test such as the Keystones.  It is disturbing that such a high percentage of the grade is based on subjective interpretation.  Although your point that the scorers are trained after being hired is well-taken, I am still skeptical as to the competence of individuals who are hired for such a low salary with essentially a four year degree as the major qualification, particularly when they report things like "Taken in for individual interviews just to basically confirm we had a pulse, didn't forge our diplomas, and weren't a threat to others or ourselves. Hired on the spot, " and were given an interview question of X + 1 = 2.  You admit that the interview process is simple; I just hope that the training isn't at that same level.

 Joanne Yurchak 

 

Combination of July 27, 2014 E-Mail from JYY to John Weiss and

response from John Weiss to JYY on July 28, 2014.

From: "Weiss, John" <jweiss@pa.gov>
To: "Yurchak, Joanne's widener" <yurchak@science.widener.edu>
Cc: "Dinniman, Senator Andrew" <andy@pasenate.com>, "Folmer, Senator Mike" <mfolmer@pasen.gov>, "Truitt, Representative" <dtruitt@pahousegop.com>, "Aaron Shenck" <ashenck@pa.gov>, "Dumaresq, Carolyn" <cdumaresq@pa.gov>
Sent: Monday, July 28, 2014 9:26:40 AM
Subject: RE: Additional Questions and Comments on Keystone Exams re: their grading and administration

Good morning Ms. Yurchak,

I in no way feel that you are badgering me.  In fact, I wish more people would ask the questions you’re asking.  If one chooses to disagree with large scale assessment but has no information on which to base that disagreement, I find that an argument that’s hard to defend.  However, if one understands the purpose, development, and scoring of the assessment and bases his/her decision upon that, that is at least an informed and intelligent decision.  I also understand that some differences will always occur simply because of philosophical differences.  Finally, before I answer your questions, please let me extend the offer to call me at any time.  The number shown in my signature block is my direct line, and I’d welcome the opportunity to discuss your questions and concerns.

I hope that my answers, provided in brown, provide you more information.  Again, if you’d ever like to discuss these questions/answers, I’d be happy to talk to you on the phone. 

John Weiss | Director

Department of Education | Bureau of Assessment and Accountability

Acting Director | Bureau of Teaching and Learning

333 Market Street | Harrisburg, PA  17126

Phone:  717.214.4394  |  Fax:  717.705.8021

www.education.state.pa.us      

Note that Mr. Weiss’ responses to JYY’s questions are

written in this E-Mail in BROWN font.

From: Joanne [mailto:jyurchak@comcast.net] On Behalf Of yurchak
Sent: Sunday, July 27, 2014 11:34 PM
To: Weiss, John (PDE)
Cc: Dinniman, Senator Andrew; Folmer, Senator Mike; Truitt, Representative; Shenck, Aaron; Dumaresq, Carolyn
Subject: Additional Questions and Comments on Keystone Exams re: their grading and administration

Hello again Mr. Weiss:

About a month ago, you patiently and thoroughly responded to several questions I posed to you regarding the grading and administration of the Keystones and I thank you for that.  At the end of this E-Mail, I have included the thread of E-mails from me and your responses in chronological order (reading from bottom to top) for purposes of completeness.

As I explained to you previously, two of my grandsons took the Algebra I Keystone in early May of 2014 and I am very concerned as to some of the procedures involved with administering and grading these high-stakes Keystones, not only for their sake but for the sake of ALL of PA's students and their parents.  Please know that in contacting you again, my intent is not to badger you, but to obtain accurate answers from you to ensure that whatever I say to others re: this testing is completely correct. 

In an attempt to make what I am requesting as understandable as possible, I am including my (JYY) initial questions to you below (which are in red font) and your answers (which are in blue font).  The additional comments and questions that I am posing in this current E-Mail are in green font.

---

1.   Initial Question by JYY: Are parents of students who do not score proficient in a Keystone allowed to review their child's test with the corresponding rubric? 

Answer by Weiss: Parents are not allowed to review their child’s test.  A certain number of questions on a Keystone Exam are used on the next administration of the exam to allow for equating, thus allowing parents/students to see those items with their corresponding answers would result in a violation of test security.

QUESTION: The Chapter 4 regulations state that parents who are considering opting out from a Keystone must first review the test beforehand to determine whether they have legitimate objections.  If parents can review the test beforehand, why can't parents review the graded test in which their child did not score proficient, both to:
     (1) determine in what specific areas their child needs help, and  A Student’s Individual Student Report gives information about the Reporting Categories where students scored well or not as well.  Rather than have students concentrate on a particular question that may have been incorrect, pointing out a specific Reporting Category that includes more than one question can demonstrate an area where a student may need to spend more time.
     (2) to determine whether there are inaccuracies in the grading?  If parents believe that an open ended (constructed response) item was not scored correctly, the parent may request through the school that the item(s) be rescored by the testing company. 

Please note that the intent to review the test before the administration and after the scoring are for different purposes.  The only reason a parent/guardian may review a test prior to the administration is to determine if the content of the test in any way conflicts with the parents’/guardians’ religious belief; it is not to ascertain answers to the questions.  If a parent/guardian and student were to review the test after it has been scored, and the parent/guardian and student work through a question that the student had answered incorrectly, the student will probably be able to answer that particular question correctly if it appears on the test the next time he/she takes it; but has that child necessarily learned more about the concept upon which the question is based, or merely the correct answer to a specific question?  Additionally, to prevent students from simply knowing the answers to questions rather than understanding the concepts or having a broad range of knowledge around a Reporting Category, the items that had been reviewed would have to be replaced.  If parents/guardians have the opportunity to review the entire test, all the items would have to be replaced, and I believe I pointed out in previous emails both the cost involved and the inability to equate.

QUESTION: Are the test questions that parents review beforehand (to determine whether or not to opt their child out) identical to those that the student would be given, or are they only a prototype?  They review an actual test.  Again, the purpose is to determine if any content in the test conflicts with their religious beliefs.

QUESTION: Shouldn't a primary educational purpose of a test be to help the student (and parent or teacher) understand exactly where the student's reasoning failed, and how to improve his understanding of individual concepts?  Note: I realize that the tests are divided into sections and that the student will probably be told in which topics s/he is deficient, but that is not the same as being able to review specific incorrect answers to help him understand his/her errors and why his reasoning and/or explanations are faulty.  In my response above I talked about the ability to answer individual questions compared to understanding full concepts around Reporting Categories.  If a student scored particularly weak in a specific Reporting Category, the school should address this area in the student’s instruction.

QUESTION:   If parents/students cannot review the tests of students who did not score proficient, can a TEACHER, or someone else in authority in the child's school do so?  It is essential that someone connected to the child be allowed to check the accuracy of the grading of the subjective questions that comprise 25-40% of the Algebra I Keystone, particularly since, as is pointed out below, the quality and credentials of the graders could be questionable, and particularly since, as you stated, 90% of the subjective questions are graded by only one grader.  If a parent/guardian, teacher, or school believes that a student’s score is incorrect, a request to have the test rescored can be made.

 2.   Initial question by JYY: How many scorers are there/question?  Is there only one, or is the score an average of more than one grader?
Answer by Weiss: Ten percent of the responses are independently read by two readers for the purpose of monitoring and maintaining inter-rater reliability.  Additionally, to ensure that the 90% of the responses that are read by one reader are scored reliably, pre-scored responses are randomly included to ensure that scorers have not drifted from the rubric.

COMMENT: In such a high stakes test that is used as a graduation requirement, it would seem to be essential to have two graders grading every subjective question, and if the scores deviate, to bring in a third, particularly because, as was stated before, there are legitimate concerns as to the quality and credentials of the graders.  While this would be expensive, it would seem to be the fairest way to assure accuracy in grading.  (Of course another way, and probably the most sensible alternative, would be to eliminate the subjective questions entirely!)  This is an area where I believe you and I will choose to disagree.  The Department does not believe the open ended items are subjective.  Each item has a specific rubric to determine how the item is scored.  Each scorer is trained to that rubric, and the checks and balances in place ensure that scorers are not deviating from that rubric.

3.   Initial question by JYY: Are the norms determined prior to giving the test or afterwards?
Answer by Weiss: I’m not sure I understand what you mean by “norms.” 

COMMENT: My fault...I used the wrong terminology.  I should have used the word "cut-offs" instead of "norms."  The question should have been:
QUESTION: Are the cut-offs determined prior to giving the test or afterwards?  After the initial administration of an assessment, the tests are scored and given a scaled score.  The tests are then arranged in order of scores.  Pennsylvania educators then participate in a Standard Setting activity where they determine the cut off points between each performance level (Advanced, Proficient, Basic, and Below Basic).  The range of scaled scores is then determined to identify each performance level.  Once established, these ranges do not change.

4.   Initial question by JYY: Are Keystones given throughout PA identical for a given time period?  I ask this question, because I know that different school districts are giving the Algebra I Keystone AT LEAST two weeks from one another.  If the questions are identical, there is no question that the test's integrity would be compromised.  Kids remember questions, particularly ones that they've struggled with, and particularly when they essentially don't have a time limit.  Although they are not supposed to discuss the questions with others, in this day and age of the Internet and social media, I assure you...it WILL happen.  I just hope that you have prepared different exam questions for tests that are given at varying times. 

Answer by Weiss: The Keystone Exams that are administered in one of the administration windows contain the same questions, and you are correct that the concern about students sharing information exists.  The Department of Education has implemented a very strict test security policy/process, but in the event that the Department recognizes a significant violation, we have produced a “breach” form to be administered.

COMMENT: There is no question that allowing these identical exams to be administered on different dates throughout the state has a high probability of resulting in a serious breach of security.  Realistically speaking, it would be difficult, if not impossible, to determine whether such security violations occurred.

QUESTION: Do you intend to correct this potential security breach by mandating that the Keystones be given on the same day throughout the Commonwealth?  This is an issue that comes under discussion every year, and while we may not limit the testing window to one day, I believe we may be moving toward shortening the window.

QUESTION: How will you handle such similar breaches of security for test makeups which would have to be done on a different day?  We have developed a breach test.

5. The U.S. Dept. of Ed. “permitted” PA to use the Keystones as our assessment tool instead of the national assessments that are being used in most states that have signed onto Common Core. This leaves no doubt whatsoever that the Feds are in charge! PRESUMABLY, the use of the Algebra I, Literature and Biology Keystones as a graduation requirement was the decision of the PDE, even though the PDE SHOULD have anticipated the chaos that would ensue both educationally and fiscally as a result of this unfunded mandate that THEY supposedly directed.

QUESTION: When the Feds finally OK’d PA’s use of the Keystones as our assessment tool about a year ago,  were there any non-publicized conditions attached? Specifically, did the Feds stipulate that we could only use the Keystones as our assessment tool if the three specified Keystones were made to be a graduation requirement?  The determination to make the Keystones a graduation requirement was a state requirement, and it was made before the Department ever approached USDoE.

QUESTION: Can you state unequivocally that there was no stipulation of this sort attached to the Keystone graduation requirement?  A number of reasons went into the state’s decision to make the Keystones a graduation requirement.  First, it is important that students graduating from School A have the same knowledge as students graduating from School B.  Every school in the state should be preparing students for success, but if schools aren’t held to the same standard, this will not occur; therefore, every school needs to teach the same high standards to prepare students to be successful in our global economy.  The state, the public, the legislature all determine if schools are accomplishing this by student results on the state assessment.  Sadly, if there are no consequences for students, some don’t always perform their best, and the schools’ accomplishment may not be portrayed accurately.  Additionally, schools are now obligated to ensure that students grasp the standards to meet graduation requirements and that additional help be offered to those who may need it.  Schools must address the educational needs of every student and cannot sweep under the rug those who may not be advancing as rapidly as others.  The graduation requirement is solely Pennsylvania’s.

While I realize that there has been no public mention whatsoever of such a stipulation, it is the only explanation that makes any sense to me.  In spite of the many poor decisions that the PDE has made in the past, I can’t imagine why they would impose such a senseless, irresponsible unfunded mandate on all of our school districts unless they were “encouraged”  to do so in some way by the Feds. The PDE knows that a large percentage of students is not likely to score proficient in these tests.  They HAD to have realized that the remediation required will have a HUGE fiscal impact on all PA taxpayers, and result in innumerable harmful consequences, particularly on the poorer districts.  The graduation requirement defies common sense!

QUESTION: There is currently a bill in the PA State Legislature that would allow the local school districts to determine how they should use the results from the Keystones -- i.e., whether or not to use them as a graduation requirement.  If this bill becomes law, is there any possibility whatsoever that the Feds would remove their approval of the Keystones as our assessment tool and require PA's students to take the national assessments?  Though we have a waiver from the Adequate Yearly Progress portion of the No Child Left Behind Act (NCLB), we are still subject to the provision that we must test every student in Mathematics and Reading/English Language Arts in grades 3 – 8 and at least once in high school.  We must test every student in Science at least once in elementary, middle, and high school.  Nowhere in NCLB does it state that the high school assessment must be a graduation requirement, nor is it stated anywhere that states must participate in any of the consortia assessments.  I cannot speak to what may happen when the Elementary and Secondary Education Act is reauthorized.

6. QUESTION: When will grades be provided to students who took, for example, the Algebra I Keystones at the beginning of May?  Districts received their District Student Data Files July 10.  Individual Student Reports will arrive in districts September 5.

I realize that the tests could take more than a few weeks to grade, particularly since 25-40% of the questions are subjective, but students who do not score proficient should be told this ASAP since their schedules for the school year 2014-2015 might have to be readjusted. For this reason, students should be provided with their grades several weeks prior to the beginning of the school year.

 I await your responses to these questions.  I realize that I am only one person and that you have many responsibilities, but I assure you that many others have the same concerns.  Perhaps you might consider publishing these as FAQ's on your web site.  It is important that educators, taxpayers, students, parents and grandparents receive straightforward answers to the multiple, legitimate questions that arise from the implementation of these high stakes exams.    Thank you very much.

Joanne Yurchak

1397 Springton Lane

West Chester, PA

 

August 4, 2014 E-Mail from JYY to John Weiss

From: Joanne [mailto:jyurchak@comcast.net] On Behalf Of yurchak
Sent: Monday, August 04, 2014 1:12 AM
To: Weiss, John (PDE)
Cc: Yurchak, Joanne's widener; Dinniman, Senator Andrew; Folmer, Senator Mike; Truitt, Representative; Shenck, Aaron; Dumaresq, Carolyn
Subject: Re: Additional Questions and Comments on Keystone Exams re: their grading and administration

Hello Mr. Weiss:

First of all, thank you so very much for your prompt and thorough response to my questions noted below.  While I still have serious issues with the grading and administration of the Keystones (specifically the Algebra I), I cannot help but be impressed with the amount of time you spent in your almost immediate response to my E-Mail.  Although I have multiple complaints with the manner in which the PDE operates, your (and others in the PDE with whom I've dealt) extreme diligence in promptly and thoroughly responding to my concerns is NOT one of them. 

I have just ONE MORE QUESTION that is unrelated to what you've answered before:  "It is my understanding that PA was once a participating member of PARCC as well as an advisory member in the Smarter Balanced Assessment Consortium.  Does written documentation exist that indicates that PA has removed itself from the PARCC Consortium and all national tests?  If so, can you provide it for me or tell me how to access it on line?"  Since I know that many others are concerned about whether PA is OFFICIALLY removed from these national tests and would like an answer to this question as well, I would greatly appreciate getting a response whenever it's convenient for you.

I will say that I still have many issues with some of your explanations and am including just a couple of comments here regarding them.  Please note, however, that I absolutely am not requesting further explanations from you on the comments or questions below as I've certainly taken enough of your time.  Besides, we're running out of colors for the fonts!  :):):):) 

• For example, you say that "If parents believe that an open ended (constructed response) item was not scored correctly, the parent may request through the school that the item(s) be rescored by the testing company." I question how a parent can believe that a specific "open ended" question wasn't scored correctly if s/he can't look at the test. It IS reassuring, however, to know that "if a parent/guardian, teacher, or school believes that a student’s score is incorrect, a request to have the test rescored can be made."  I assume that this option has been noted to school districts.

• Your explanations below would indicate that many, if not all, of the test questions on the Keystones are the same from one year to another.  I realize that this is one way to keep the costs down, but have to say that with all of the people handling and taking the tests over weeks and evidently years, this doesn't seem to be an effective way of maintaining the integrity of these high-stakes exams, no matter how many safeguards and regulations you require.

• I realize that I sound like a broken record, but I still am very leery of using scorers (who are paid $11-14/hour) to grade subjective (AKA open-ended) questions that comprise 25-40% (in the case of Algebra I) of these high stakes tests and who seem to be hired off the street with a college degree as their primary qualification. 

Note: The following discussion in small font is not directly related to the Keystone issue.

Finally, I TOTALLY agree with you that "schools must address the educational needs of every student."  Although you might mean well, don't you see that what the PDE is doing by implementing the Common Core "one-size-fits-all" paradigm and mandating the three-Keystone graduation requirement, is diametrically opposed to "addressing the educational needs of EVERY student?"  The PDE needs to change course and rethink/restructure our entire educational system.  In order to help EACH student reach his maximum potential and prepare for his/her ultimate college/workforce goals, it is essential that you make individual differences and needs your top priority.  In the remainder of this E-Mail, I'd like to present you with some "food for thought" for more individualized alternatives to the "one-size-fits-all" educational model (Common Core) that is currently being implemented in Pennsylvania.

Decades ago, schools awarded different types of diplomas in accordance with an individual student's goals.  (Some examples of diplomas that have been given in the past were termed: Academic, Commercial Arts, Business, General Curriculum, Industrial Arts, Trade Preparation, Vocational Arts, etc.)  Courses in each category were tailored to effectively prepare students for the specific path that THEY chose to take after high school graduation, whether it was college, an apprenticeship, trade school, direct entry into the workforce, or whatever.  This worked out quite well.  After all, it was that generation that put a man on the moon!  Today, there is only ONE diploma type awarded in most high schools.  

When a student receives a COLLEGE degree, that degree indicates that the student has taken, and performed satisfactorily in, courses that are designed for that specific degree.  Why can't a similar situation be utilized in high schools?  For example, why should a student who intends to go directly into the workforce be forced to demonstrate proficiency in the Common Core-aligned Algebra I in order to graduate, and possibly subjected to hours of remediation for material that he will probably never use, when his time could be far better spent in a business math course that focuses on material that is applicable to his chosen vocation?  (A business math course could certainly include some of the basic parts of algebra such as simple equation solving and simple graphing that might be useful for everyone, but could omit the more "rigorous" material that would be more appropriate for STEM students.)  Wouldn't it make more sense for HIGH SCHOOLS TO PROVIDE CHOICES OF DIFFERENT TYPES OF DIPLOMAS, WITH DIFFERENT STANDARDS FOR EACH, AND ENABLE STUDENTS TO DECIDE FOR THEMSELVES WHICH WOULD WORK BEST FOR THEM IN ACCORDANCE WITH THEIR ABILITIES, APTITUDES, AND ULTIMATE LIFETIME GOALS?

President Obama, in presenting his view of educational reform in a 2/22/2010 White House statement said: “educating every American student to graduate prepared for college and success in a new work force is a national imperative.” The mindset that either directly or indirectly suggests that everyone should have a college education is an elitist viewpoint that forces many into a mold that is not appropriate for them. Too many high-ranking politicians in addition to proponents of the CCSS (Common Core State Standards) parrot this mantra.  A college degree does not guarantee success, happiness, or the ability to find an appropriate job as innumerable college graduates who have acquired tremendous debt along the way are finding out.  Many young people could be happier in a suitable trade and far more successful in securing employment if they listened more to their hearts than to the various “powers-that-be” that unfortunately and incorrectly suggest that attaining a college degree should be the ultimate goal for almost everyone.

Knowledgeable individuals opine that the goal of the CCSS seems to focus on getting students ready for a two year college. Does this make sense?  Is this adequate for students who intend to go into STEM careers or have their sights on a four year college?  On the other side of the coin, what about the students who don’t need a two year college degree for their chosen field and might be going to a trade school, to apprenticeship training, or even directly into the job market?  Why should their choices be marginalized by an elitist federal or state government that is trying to impose a “one-size-fits-all” set of standards on them in what can only be described as an “I know best,” Nanny State mentality?  The PDE is doing exactly that when you require all students to take, for example, the Common Core-aligned Algebra I, and also to display proficiency on that assessment in order to graduate from high school.  Much of the more “rigorous” material in Algebra I will never be used or needed by a significant percentage of students.   It's time for our educational system to scrap the Common Core "one-size-fits-all" model and focus on individual differences to enable all students to maximize their chances of becoming productive members of society.

I realize that I have deviated from our original dialogue regarding the Keystones, but your comment "schools must address the educational needs of every student" precipitated this additional discussion.  The testing mandates are only one aspect of the Common Core implementation that many of us who oppose Common Core find so objectionable.  I realize that legislators and those who hear our complaints are justifiably saying: "Well, if not Common Core, then what?"  The solution is extremely complex and I'm sure that others will present many excellent ideas in the future.  For now, I'd like to present just a few thoughts and suggestions of my own:

• Evaluate each individual's abilities, aptitudes and interests, beginning in middle school and continuing throughout high school, recognizing that these can change over time and maintaining flexibility for the student to change course midstream.  It is crucial that a student not be locked into choices made in his early years as is done in some European countries. 
• Provide several options for different types of high school diplomas for students and, with guidance from capable professionals, enable students and their parents to choose which type best meets the student's individual needs, capabilities and aspirations. IT IS CRUCIAL, HOWEVER, THAT THE ULTIMATE CHOICE BE MADE BY THE STUDENT.
• Design standards and associated assessments that are tailored for the specific diploma the student has chosen.
• Allow local school districts to determine whether the student has met the requirements for his/her chosen diploma type from his/her course grades and assessment scores.

Individual differences and preferences must be recognized and valued and this must begin in our local schools. Replacing the “one-size-fits-all” model of Common Core with a system that focuses on the individual needs of students will go a long way towards enabling every student to succeed in his/her own way and become a productive member of society.

I hope you will at least consider the suggestions presented in this E-mail. Thank you for “listening,” and thank you for your patience and prompt responses to my many questions regarding the administration and grading of the Keystone exams.

Very truly yours,

Joanne Yurchak

1397 Springton Lane

West Chester, PA 19380

P.S.: Just an FYI… I speak first of all from the perspective of a grandmother of four, two of whom were subjected to the Common Core-aligned Algebra I last year at Paxon Middle School in the Marple-Newtown School District.  In observing a class one day, I found their faddish Common Core-aligned program called Power Teaching to have a slightly cultish atmosphere.  The teachers even wore aprons that had "Power Teaching" label on them!  Is this really necessary?  I am attaching a description of the manner in which they were taught, which I considered to be educationally counterproductive, and quite frankly very disturbing.  I have to add that the teacher herself seemed quite competent; it was the methodology she was undoubtedly forced to use that was objectionable.

I also speak from the perspective of one who taught for decades as a chemistry instructor at the university level, having retired several years ago.  During my college teaching years I encountered far too many students who would have been much better off choosing a different path after high school, but who, for whatever reason, chose the college route. Of course the colleges were more than delighted to admit them and “take” (I could use a different word but the word “libel” comes to mind) their money.

An experience I had with an electrical engineering student who was in my freshman chemistry lab molded some of the opinions that I’ve expressed in this E-mail. The young man was doing quite well academically and was a joy to have in class. One day in mid-semester, he came to me and told me he was dropping out of college. Quite distressed, I asked him why and told him I thought he was certainly capable and a very good student. He said: “Mrs. Yurchak, I decided that electrical engineering isn’t for me; I’d rather be an electrician!” This was a young man who knew his own mind and followed his heart. What could I do but wish him well and say “Go for it!”

 

August 4, 2014 E-Mail from John Weiss to JYY

From: "Weiss, John" <jweiss@pa.gov>
To: "Yurchak, Joanne's widener" <yurchak@science.widener.edu>
Cc: "Dinniman, Senator Andrew" <andy@pasenate.com>, "Folmer, Senator Mike" <mfolmer@pasen.gov>, "Truitt, Representative" <dtruitt@pahousegop.com>, "Aaron Shenck" <ashenck@pa.gov>, "Dumaresq, Carolyn" <cdumaresq@pa.gov>
Sent: Monday, August 4, 2014 10:17:38 AM
Subject: RE: Additional Questions and Comments on Keystone Exams re: their grading and administration

Good morning Ms. Yurchak,

 Your acknowledgement of prompt and thorough responses is appreciated.  Thank you. 

 PA is still a participating member of PARCC and an advisory member of Smarter Balanced, but we are not a governing state in either consortium.  Essentially that means that we get to listen to what the consortia have to say, but we don’t get to offer any input.  We also are not authorized to administer either test developed by either consortium.  However, PA would not administer either test because it’s written into Pennsylvania Title 22. Education Part I. State Board of Education Chapter 4. Academic Standards and Assessment. 

 §4.51  State Assessment system (h)   (2)  Subject to paragraph (3), the Board will not, and the Department may not, be a governing state in any consortium for the development of a National assessment for the purpose of utilization as part of the State assessment system.

I’m not sure if this is what you mean by documentation in writing, but obviously we would be in violation of Chapter 4 if we ever administered either the PARCC or Smarter Balanced assessment.

My use of the sentence, “Schools must address the educational needs of every student” was made in reference to a practice that occurred too often in schools many years ago where those who didn’t progress as quickly as others were often overlooked.  I recognize that your concern is that the educational needs of students may be different from student to another, thus your concern about the emphasis on preparing students to be college and career ready with your observation that “college ready” may be more heavily emphasized.  I’m also aware that many others share your concern and if enough voices are heard, policies may be subject to change.

Thank you again for your thoughtful, intelligent questions concerning testing.  I feel quite confident that the state assessments administered in Pennsylvania are valid and reliable measurements of what students should know and be able to do as prescribed by the standards.  Again, I offer my phone number if you’d ever like to discuss. 

 John Weiss | Director

Department of Education | Bureau of Assessment and Accountability

Acting Director | Bureau of Teaching and Learning

333 Market Street | Harrisburg, PA  17126

Phone:  717.214.4394  |  Fax:  717.705.8021

www.education.state.pa.us