Test your Knowledge of School Grades in New York City

Version 2 (12/21/07)

Written by Professor Celia Oyler

(send comments to oyler@tc.edu)

In November, 2007, The New York Department of Education issued a letter grade of A through F to each school in the city. Each grade is based on a very complex set of formulas. Test your knowledge about these new school grades.

1. The school grade is based on three "elements": school environment, school performance and school progress. At the elementary and middle school level, what percentage of the final grade is derived from achievement test scores?
   1. 10%
   2. 25%
   3. 55%
   4. 85%
   
   
2. The New York State achievement tests used to calculate progress are designed by psychometricians and are normed in advance on a large group of students to ensure that the items at each grade level are appropriate for that grade level.

TRUE FALSE

3. From a psychometric point-of-view, New York State achievement test scores offer a reasonably adequate tool to measure progress of learners from year-to-year.

TRUE FALSE

4. Under No Child Left Behind, schools are expected to show that children in grades 3 through 8 have-on average-made one year of progress as measured by achievement tests.

TRUE FALSE

5. In theDOE's formula, the year of progress is calculated using statistical methods that take measurement error into account.
   
   TRUE FALSE

6. The 55% of each school grade (in elementary and middle schools) that the DOE calls "progress" (and is based on the averages of 2 achievement tests scores at each grade level) takes into account the unreliability of the average gains in achievement within each school.

TRUE FALSE

7. To get the highest score of a "4" (1 is lowest) on last year's English Language Arts test (ELA), in 5th grade, a child can only get one question wrong on the multiple choice section.

TRUE FALSE

8. The scoring of the writing sample of the achievement tests uses a rubric and is conducted by:
   1. Department of Education personnel to ensure that all results are reasonably fair
   2. Teachers across the city who sometimes know the schools they are grading for
   3. Personnel from the New York State Department of Education who are trained to not take into account such factors as the children's handwriting
   4. Temporary workers hired by each school
   
   

9. Each school in New York City is subjected to a Quality Review where a trained observer rates the school on many dimensions of curriculum, instruction, and assessment of learning.

TRUE FALSE

10. The results of these Quality Reviews are then factored into the final grades each school receives.

TRUE FALSE

11. A school can receive a "proficient" on its Quality Review and still receive a school grade of "F".

TRUE FALSE

12. Circle all that are correct: The school grades are based on how well each school:
    1. Teaches children to solve problems
    2. Uses culturally relevant pedagogy
    3. Integrates the arts
    4. Provides time for children to exercise
    5. Prepares children to make healthy food choices
    6. Helps teachers work cross-racially and cross-culturally
    7. None of the above
       
       
13. The scores that New York City students achieve on the New York State tests show basically the same trends as those that a sample of New York City students achieved on the national achievement test (called the National Assessment of Educational Progress and administered since 1969 to samples of students across the country).

TRUE FALSE

14. There is a strong correlation between the list of schools that New York State has rated as failing and the ones that received a grade of "F" by the New York City Department of Education.

TRUE FALSE

15. Of the 346 schools in New York City that the State of New York has flagged as having the most difficulty (SINI: Schools in Need of Improvement; SURR: Schools Under Reregistration Review), how many received a grade of A?
    1. 10
    2. 25
    3. 40
    4. 50
       
       
16. The ARIS computer system specifically designed by IBM for the DOE and intended to track student progress on annual and periodic assessments cost approximately
    1. Eighty million dollars
    2. Eight million dollars
    3. Eight hundred thousand dollars
    4. Eighty thousand dollars
       
       
17. The DOE assigns each school its final grade based on the actual score in relation to all the peer schools so in theory every school could achieve an A, if all students showed a year of progress.

TRUE FALSE

18. A school can receive an "F" even if 98% of its students are rated on grade level in math and 86% are on grade level in language arts, as measured by the New York State tests.

TRUE FALSE

19. After all the large number of calculations are completed-including being compared to the schools in the "peer group"--each school receives a final score. These scores are then converted into a final grade.

TRUE FALSE

20. The final score of one school may be only one hundredth point (0.01) away from another school, but one school can get a higher letter grade than the other.

TRUE FALSE

21. Short Answer (extra credit): Since these school grades are: so expensive to produce; not based on many important aspects of what many educators and parents consider central aspects of schooling; do not take into account multiple measures of student progress and school quality; do not take into account standard statistical measures of error; and are based predominantly (in elementary and middle schools) on state tests not designed to be used to make year-to-year comparisons of student growth, why are these school grades being used by the Bloomberg/Klein administration?

 

Answer Key

1. d. School Environment counts for 15% of score, and is calculated from attendance and the results of parent, student, and teacher surveys. Student Performance counts for 30% of score and is measured by elementary and middle school students' scores that year on the New York State tests in English Language Arts and Mathematics. Student Progress counts for 55% of score and is measured by comparing each student's score on the state math and state English Language tests from one year to the next. So test scores equal 85% of the total grade.

2. False. The New York State tests are not norm referenced tests.

3. False. The New York State achievement tests are designed to measure proficiency in reference to a learning content standard, and are unreliable to measure progress, particularly for students who are the furthest away from the expected proficiency level. Since the state tests are tied to grade level content standards, a student at the low or high end of achievement may make excellent gains in learning that are not detectable on the state tests.

4. True. This is why elementary school principals are often worried if their 3rd grade test scores are really high; this means that when those children are in 4th grade, they have to go even higher to show progress.

5. False

6. False. Each school's average gain in achievement is an estimate, and there is a confidence interval (i.e. margin of error) around that estimate. The progress score ignores the fact that the confidence intervals for different schools are likely to overlap considerably. Thus, what looks like a real difference in achievement averages between schools may be no real difference in actual achievement at all. Confidence intervals are an essential element to this type of analysis.

7. True. Last year's ELA score of 4 required 30 or 31 correct answers on the multiple choice section, out of 31 items. (NYSED website)

8. b

9. True

10. False

11. True

12. g

13. False. See http://nycpublicschoolparents.blogspot.com/2007/11/how-school-grading-system-is-fiasco.html

14. False

15. d

16. a

17. False. The grades are calculated "on a curve" with only a certain percentage being allowed in each category. See http://www.eduwonkette.com/2007/11/is-this-wake-up-call-for-people-who.html

18. True. See http://www.nydailynews.com/news/2007/11/07/2007-11-07_how_a_good_school_fails.html

19. True

20. True

21. Possible correct answers to this question remain a bit of a mystery. Hypotheses abound, but what Joel Klein says is: "Everyone knows what A and F mean. Summing up all relevant measures with a single, simple grade draws sharp attention to the great work at many schools and the stagnation that might otherwise escape notice elsewhere. Grades make people face facts." (New York Times, letter, 11/16/07)

Acknowledgements:

The author of this test thanks the numerous colleagues who offered expert advice on the wording of items for maximum accuracy. Professor Aaron Pallas provided invaluable assistance related to measurement error; many other colleagues wish to remain anonymous. A

Published Wednesday, Dec. 12, 2007