Making the Grade: Do Current Restaurant Grading Systems Fulfill Stakeholder Expectations?

Jessica Egan

Research Scientist/Epidemiologist

New York State Department of Health

International Food Protection Training Institute (IFPTI)

2017 Fellow in Applied Science, Law, and Policy: Fellowship in Food Protection

Author Note

Jessica Egan, Research Scientist/Epidemiologist, New York State
Department of Health.

This research was conducted as part of the International Food Protection Training Institute’s Fellowship in Food Protection, Cohort VI.

Correspondence concerning this article should be addressed to Jessica Egan, New York State Department of Health, Bureau of Community Environmental Health and Food Protection, Empire State Plaza, Corning Tower, Room 1395,

Albany, New York 12237; Email: jessica.egan@health.ny.gov

Abstract

A restaurant grading system is a program in which food service establishments are assigned a score reflective of the sanitary conditions of the facility. These scores take a variety of formats ranging from letters to numbers to colors. Most restaurant grading systems are based on similar grading system elements. A 2012 National Association of County & City Health Officials (NACCHO) survey of local food regulatory programs found that 38% of respondents reported that their jurisdiction had a restaurant grading system in place. (NACCHO, 2016)  

These systems are not fully supported as they are frequently criticized as being only reflective of a ‘snapshot’ in time. (Baer, 2015)  From December 2016, to January 2017, this study gathered 2,370 responses from restaurant grading system stakeholders (food safety regulators, food safety academics, members of the food industry, and consumers) regarding the elements they believe restaurant grading systems should be based on; what other elements, if any, they believe should be factored into a restaurant’s overall grade; and, their perceptions as to the meaning behind certain grades.

Survey results show that 70% of respondents believe that a restaurant’s grade should be based on the results of more than one inspection; 70% of respondents believe that non-critical violations are very or somewhat representative of the level of food safety at a restaurant, while at the same time, 82% of respondents believe that restaurant grades should be based on the results of inspections done by inspectors who have attended a risk-based inspection training program; and across all survey groups, one quarter of respondents were unsure what color grades represented in terms of the level of food safety at a restaurant. These results indicate that a gap exists between stakeholder expectations regarding restaurant grading system elements and the basis of most existing restaurant grading systems. When considering a restaurant grading system, jurisdictions should take into account stakeholder expectations and perceptions in order to implement the ideal grading system.

Background

          As consumers continue to demand access to information about the sanitary conditions at their local restaurants, more jurisdictions across the country have adopted restaurant grading systems. The intent of these grading systems is to interpret the results of a facility’s most recent inspection in a quick and easily understandable manner so that the public can use this information when making decisions about dining out. Existing restaurant grading systems use a color, number, letter or combination thereof to represent the restaurant’s assigned grade. The challenge for food safety regulators is to develop a grading system that accurately reflects the overall level of food safety at a facility. The challenge for the restaurant industry is to adhere to food safety regulations to provide safe food for their customers, and to maintain a positive public image as reflected in the grade they are assigned.

While consumers and some food safety programs are in favor of restaurant grading systems, many regulatory and industry stakeholders have long been opposed to the systems because they feel that grading systems are inconsistent, economically detrimental, and only representative of the conditions at the facility during the brief time of last inspection (National Restaurant Association, 2012). Though some states and municipalities, such as Boston, Massachusetts, have recently adopted restaurant grading systems, others, such as the state of Colorado, have restricted them (Food Safety News, 2016).  

Problem Statement

          Stakeholder perceptions and expectations regarding restaurant grading system elements are largely unknown.

Research Questions

          The objective of this project was to answer the following questions:

1.     What are the common elements of existing restaurant grading systems?

2.     To what extent do stakeholders think that these elements reflect the level of food safety at restaurants?

3.      Are there any other elements that stakeholders believe should be included in a grading system?

4.     How do stakeholders perceive the meaning of different types of restaurant grades?

Methodology

          This research project surveyed food safety regulators, academics in food safety, members of the restaurant industry, and consumers to determine if the basis of restaurant grading systems met the expectations of the different stakeholder groups. The groups included in the survey were those believed to have an interest or concern in restaurant grading systems, thereby making them stakeholders in the issue.

This research project began with an extensive literature review to identify existing restaurant grading systems and the elements on which the grades are based. One limitation of this study was the lack of a dataset identifying existing restaurant grading systems. For the purpose of this research project, restaurant grading systems were identified using a state by state internet search, as well as various news articles referencing grading systems. As a result, all restaurant grading systems may not have been identified.

A survey was developed using Survey Monkey, and was sent to individual stakeholders. The survey primarily consisted of four sections asking respondents to rank the following:

·       How representative different grading system elements are of the level of food safety at a restaurant

·       How many inspections should be factored into a restaurant’s grade to best represent the level of food safety at the facility

·       The level of agreement related to other statements about restaurant grading systems

·       The respondent’s understanding of the different restaurant grades most often used today.

After ranking each of the items in the four sections, an open-ended comments box was provided for additional information respondents may wish to provide. By asking stakeholders their opinion on how reflective restaurant grading system elements are of the level of food safety at a restaurant, this project attempted to identify the most valuable components on which a restaurant grading system should be based.

An email introducing this research project was sent to members of Association of Food and Drug Officials (AFDO), the Conference for Food Protection (CFP), and the Center for Science in the Public Interest (CSPI). Included in the email was the link to the survey. Recipients were given two-weeks to complete the survey. The survey asked respondents to self-identify as a member of one of the following groups: Regulatory – local government, Regulatory – state government, Regulatory – federal government, Academia, Industry, Consumer, or Other.

          Prior to being sent to stakeholder groups, the survey was pilot tested by sanitarians and research scientists at the New York State Department of Health to identify and address any problems such as formatting issues and spelling errors. SAS statistical software version 9.4 was used to analyze survey results and identify similarities and differences among the four groups of respondents.  

Results

          Of the 109 existing restaurant grading systems identified by this research, the majority are based on a numeric score assigned at the most recent inspection. These scores are generally calculated by assigning each restaurant a score of 100 before the inspection begins, and deducting points for each violation identified during inspection.

There were 2,370 responses to the survey. The majority (70.93%) of respondents self-identified as consumers, followed by local regulators (9.37%), academics (6.84%), state regulators (6.29%), industry members (5.15%), and federal regulators (1.43%) (see Table 1).

Table 1

Survey Respondent Affiliation

While the results of each survey question were thought-provoking, three findings were selected for further discussion.

          The majority of respondents (70%) believe that restaurant grades should be based on the results of more than one inspection (see Figure 1).

Figure 1. Restaurant Grade Based on More than One Inspection.

Overall, 70% of respondents believe that non-critical violations are very or somewhat representative of the level of food safety at a restaurant. While at the same time, 82% of respondents believe that restaurant grades should be based on the results of inspections done by inspectors who have attended a risk-based inspection training (see Figures 2 and 3). For the purposes of this research project, risk-based inspection training was defined for respondents as a training that teaches inspectors to focus on the conditions most likely to cause foodborne illness.

Figure 2. Non-Critical Violations – Representative of Food Safety.

Figure 3. Restaurant Grade Based on Risk Based Inspection.

          Across all survey groups, one quarter of respondents were unsure what color grades represented in terms of the level of food safety at a restaurant (see Table 2).

Conclusions

          The results of this survey show that the majority of stakeholders believe that restaurant grades should be based on the outcome of more than one inspection. This concept is in direct contrast to the way that most restaurant grading systems currently operate, as most restaurant grading systems currently in place assign a grade based solely on the results of the most recent single inspection.

          The majority of stakeholders believe that non-critical violations are very or somewhat representative of the level of food safety at a restaurant. These violations focus primarily on the general sanitation or maintenance of a restaurant, and have not been shown to directly cause foodborne illness. At the same time, most stakeholders believe that restaurant grades should be based on inspections done by inspectors who have attended a risk-based inspection training program. Risk-based inspection training teaches inspectors to focus on conditions that have been shown to cause foodborne illness. The premise for this question was that an inspector who attended risk-based inspection training would apply risk-based inspection principles while conducting restaurant inspections. That is to say, inspectors trained in conducting a risk-based inspection would focus on the violations most likely to cause foodborne illness, rather than the general sanitation violations that make up non-critical violations.

          On average, one quarter of all stakeholder groups surveyed were unsure of the meaning of color restaurant grades; in particular, the colors red and green. While some restaurant grading systems use color as all or part of the representation of their grades, most do not use color alone. The results of this survey indicate that color alone may be confusing to some stakeholders.         

Recommendations

1.     When designing a grading system, jurisdictions should consider incorporating more than one inspection score into a restaurant’s overall grade. While there are existing restaurant grading systems, such as that in Lincoln-Lancaster County, Nebraska, that utilize a statistical model to calculate restaurant grades from the results of multiple inspections, the majority of existing systems base a restaurant’s grade on results of only one inspection.

2.     Risk-based inspection training should be increased. This survey identifies a contradiction in that stakeholders see non-critical violations as indicative of the level of food safety at a restaurant, but at the same time believe that the inspections used to calculate a restaurant’s grade should be done by inspectors trained to conduct risk-based inspections. This contradiction highlights the need for more education on the relationship between critical and non-critical violations and food safety. While the average consumer stakeholder may not fully understand how violation categories differ in terms of the implications for foodborne illness, the results of this survey show that there are some misconceptions at the food safety regulatory level as well. Additional risk-based inspection training opportunities could help strengthen the understanding of the relationship between violations and foodborne illness.

3.     Grading systems should not use color alone. Informing consumers is the primary goal of restaurant grading systems, therefore information about restaurant conditions must be communicated in an easy to understand manner. Given that one quarter of stakeholders, particularly consumers, are unsure what different color grades represent, the results of this survey illustrate that color alone should not be used to represent a restaurant’s grade.

4.     Further research should be undertaken. It is the hope that these results will further the development of an ideal restaurant grading system to both satisfy the desire of consumers to better understand, and to accurately reflect the sanitary conditions at their local restaurants. The ideal grading system will likely incorporate the results of more than one inspection into a restaurant’s grade. While this survey asked stakeholders to specify how many inspections should be factored into a restaurant’s grade and over what period of time these inspections should take place, formatting issues with the survey responses prevented in-depth analysis of this data. However, preliminary data analysis showed that stakeholders believed that a minimum of two inspections should be used to determine a restaurant’s grade. The amount of time over which these inspections should take place will likely depend on each jurisdiction’s time frame for frequency of inspection. Future studies can delve deeper into these questions.

Acknowledgements

I would like to thank Darby Greco, Marta Gomez, and Brian Miner at the New York State Department of Health for the time they spent helping me brainstorm ideas and proofreading my work. I am very grateful to Joe Corby and the AFDO staff, Dr. David McSwane of the Conference for Food Protection, and David Plunkett at Center for Science in the Public Interest for sharing my survey with their members. Many thanks to the International Food Protection Training Institute for the opportunity to be a part of this Fellowship, specifically Jerry Wojtala, and to all the IFPTI staff for taking such good care of us. I would like to say thank you to Charlene Bruce, Joe Corby, Dan Sowards (deceased March 30, 2017), Cameron Smoak, and Steve Steinhoff for sharing their experiences in the field of food safety and for making us laugh! I also want to thank Paul Dezendorf for all the time he spent reviewing my research project and providing feedback. A big thank you to my co-Fellows: Autumn, Joann, Kyle, JoAnna, Matthew, Sherri, Skya, Renita, and Odeisa—I am so glad we were all thrown into this together! Most importantly, I am very thankful for my husband Scott, our son Grant, daughter Elizabeth, and my parents for their love and support during this Fellowship.

References

Baer, S. K. (2015, June 25). What that restaurant letter grade isn't telling you about health and cleanliness. San Gabriel Valley Tribune. Retrieved from http://www.sgvtribune.com/lifestyle/20150625/what-that-restaurant-letter-grade-isnt-telling-you-about-health-and-cleanliness

 Governor lets bill banning restaurant letter grades become law. Food Safety News. (2016, June 10). Retrieved from http://www.foodsafetynews.com/2016/06/127448/#.WBo79i0rJpg

National Association of County & City Health Officials (NACCHO). (2016). Summary of retail food inspection scoring, grading, and placarding systems. Washington, DC: National Association of County and City Health Officials.

NYC operators give letter grades an 'F'. National Restaurant Association: News & Research. (2012, March 13). Retrieved from http://www.restaurant.org/News-Research/News/NYC-operators-give-letter-grades-an-F