This is a paper I wrote back in 2009 for a course I was taking at the time (only slightly modified from the original version).  My story of failing the clinical exam, summarized at the beginning of the paper, will be a future podcast topic.  I will go into the story in much more detail at that time.  Maybe I’ll cry.  No promises : ).

 

The Efficacy and Ethics of One-Shot Clinical Exams for Dental Hygiene Licensure

Abstract

I reviewed available literature on the efficacy, ethics, and alternatives to the traditional use of high stakes, one-shot clinical exams with live patients that are utilized by dental boards in the licensure process for dentists and dental hygienists.  The general consensus seems to be that these exams are not valid or reliable for making important licensure decisions.  They also pose serious ethical problems for all involved.  Some of these problems include over treating the patient or significantly delaying needed treatment for the sake of saving the patient for the board exam.  Three prominent alternatives are discussed in the literature—portfolios, residencies, and Objective Standardized Clinical Evaluations (OSCE).  All three alternatives avoid the ethical problems posed by the one-shot clinical exams, and at least two (portfolios and residencies) offer the benefit of providing multiple assessments over a long period of time.

My Personal Experience

After many years of hard work and sacrifice, I finally achieved the major goal that I set for myself back in 2001—I graduated from dental hygiene school, a very demanding program, in June of 2007.  By graduation, I was looking forward to starting my new career as a dental hygienist, but unfortunately, an unexpected and surprising obstacle stood in my way—I received the results of my North East Regional Board (NERB) clinical exam, and to my disbelief, I failed the exam.  I could not receive my dental hygiene license to practice as planned, so I had to wait until August to retake the NERB clinical exam, which I passed with a perfect score this time.  Six weeks later, after many months of worry and stress, I received my long awaited dental hygiene license in mid-September.

The NERB clinical exam is only one of many requirements for dental hygiene licensure mandated by the Massachusetts Board of Registration in Dentistry.  Other requirements included passing the NERB written exam, passing the National Board Dental Hygiene Examination (NBDHE), graduating from a dental hygiene program accredited by the Commission on Dental Accreditation (CODA) of the American Dental Association, and obtaining letters from three licensed dentists to confirm my good moral character.

The NERB clinical exam involved finding a patient with moderate periodontal disease (gum disease beyond gingivitis) who required non-surgical periodontal therapy (a.k.a. “deep cleaning”).  During the exam, my patient only received partial treatment of selected teeth, as required by the exam, which were then assessed by multiple examiners.  The challenge of finding a qualified patient willing to sit for the exam was very difficult.  Throughout dental hygiene school, especially the last year, my classmates and I were on the constant look out for “board” patients.  All patients entering our school clinic were potential candidates.  Unfortunately, some students, including myself, had to settle for less than ideal patients for the exam.

Failing the NERB clinical exam was quite devastating for me.  After four years of being a full-time student with no income, my savings had become depleted and for the first time in my life, I incurred what I considered significant debt.  I faced a summer of having to find another “board” patient, of waiting to retake a clinical exam that I could conceivably fail again, and of being in a financial position where I eventually cashed in an IRA fund to pay my bills.  The summer of 2007 was very difficult for me financially, emotionally, and spiritually.

The result of the NERB clinical exam was difficult for me to accept for one major reason.  I strongly believed, and the dental hygiene faculty strongly concurred, that the exam was not a valid assessment of my clinical skills.  I excelled in all other assessments.  For example, my overall GPA in the classroom and clinic was 3.98, my score on both the National Board Dental Hygiene Examination (NBDHE) and the NERB written exam was 97%.  My score on the NBDHE put me in the top 1% nationwide.  The NBDHE is an eight hour exam that involved over 500 questions with 200 of those based on detailed case studies.  And I also finished at the top of my class.  Suffice it to say, I was an excellent student.  In my opinion, the NERB clinical exam is unreliable, invalid, and unfair.  I later discovered, after doing a little research at the time, that I wasn’t the only one who held this opinion.

Efficacy of One-Shot Clinical Exams

My review of the literature showed a general support for the view that the reliability and validity of high stakes, one-time clinical exams using live patients is very low, and that they should be replaced with other types of assessments.  This view applies to both dental and dental hygiene clinical exams.

With a national debate in progress about the validity and reliability of dental licensure examinations, dental hygiene is not immune from this controversy.  The issues are clearly the same for dental hygiene, yet very little has been published regarding dental hygiene clinical licensure examinations.  Both dental and dental hygiene educators have observed many of their best and most clinically competent students fail to pass these clinical licensure examinations.  The toll that failure extracts from the students, their families, educational institutions, and the public at large is only justifiable if there is sufficient data to demonstrate the validity of these high-stakes examinations (Gadbury-Amyot, Bray, Branson, Holt, Keselyak, Mitchell & Williams, 2005, p. 363)

One problem with the clinical exam is that it fails to measure the comprehensive nature of dentistry today.

In particular, the one-shot evaluation format is limited to testing one-shot tasks.  Increasingly, dentistry is about managing patients and their oral health over extended periods of time.  No matter how well designed, a test of one-shot performance will not be able to measure the large realm of dental practice behaviors that occur in context and over time (Chambers, Dugoni & Paisley, 2004b, p. 245).

A second problem with the clinical exam is that it might be passing graduates who should not be licensed.

…the current one-shot initial dental licensure system misclassifies at least 20 percent of candidates who must retake the tests, plus an unknown number of candidates who pass the tests by luck and should not have been granted a license. (Chambers et al., 2004b, p. 245)

A third problem with the clinical exam involves the variables that are beyond the student’s control (e.g. patient not cooperating).  These variables overwhelm the ability of the one-time nature of this exam to be valid and reliable.

A single clinical procedure examination on a live patient does not reflect competence in performing that procedure.  In fact, regional boards report that most applicants pass within twelve months of failing a first examination without remedial coursework. This means that something other than the candidate’s ability was the cause of the original failure (Formicola, Shub, & Murphy, 2002, p. 607).

Those closest to the students, the dental and dental hygiene faculty, support significant change in the licensure process.  A 2003 survey concludes:

Administrative leaders of the country’s dental schools think that third-party evaluation of graduates is appropriate, but they do not have confidence in current clinical tests for licensure. Most deans or their designated respondents to a survey on issues in clinical testing for licensure think that the tests lack validity for decision purposes. They strongly think that change in the process for licensure is needed and that the change in testing should be at the national level in a way that eliminates concerns for the use of human subjects (Ranney, Haden, Weaver & Valachovic, 2003, p. 1157-1158)

Most studies consistently show that there is a weak or no correlation between the clinical exam assessment and other proven forms of student assessments.

The lack of concordance between previously validated measures of dental hygiene student competency or predictors of success (Overall GPA, NBDHE, and Portfolios) and a one-shot clinical licensure examination (CRDTS) [This is the NERB equivalent for the central region of the country] raises serious concern about the validity of our current dental hygiene licensing procedure which uses CRDTS to make decisions about granting licenses to practice.  State licensing boards and educational institutions should be encouraged to work cooperatively to develop defensible and valid methods for assessing competency of students (Gadbury-Amyot et al., 2005, p. 369)

Ethics of One-Shot Clinical Exams

The lack of validity and reliability of high stakes, one-shot clinical exams that use live patients is not the only problem with these exams.  These clinical exams are also fraught with many ethical problems.

… survey data indicated that more than half of responding licensed dentists “knew with certainty” of ethical problems with clinical licensure examinations.  These included failure to provide follow-up care, unnecessary radiographs, treatment that was not in the patient’s best interest, and creation of a lesion [carious] for purposes of the examination (Ranney, 2006, p. 151-152)

Under the tremendous pressure of passing the clinical exam, many of my dental hygiene classmates, including myself, violated the dental hygiene code of ethics on some level.  The more common ethical violations were the delaying of needed treatment (sometimes up to two years) to save the patient for the clinical exam, and the other was asking the patient to withhold medical information to make the process easier for the student.  These ethical violations were not unique to my school.

Patients presenting at the dental school clinics with pathologies that might be appropriate are selected and shepherded for use in the NERB examination.  Necessary treatment is withheld until the examination without concern for the patient’s needs.  Treatment and, often, direct payment of expenses and financial stipends are used as incentives to help guarantee that patients will appear at the examination site. (Lasky & Shub, 2003, p. 299)

One of the ethical problems involves the overtreatment of patients.  In the case of dental hygiene students, the patient may experience excessive and overly aggressive instrumentation.

In the case of dental students, dental lesions that have not cavitated and are located only in the enamel layer of the tooth are allowed to be restored in the majority of states.  “The scientific literature does not support the decision to restore a tooth if the radiolucency is in enamel or up to the DEJ” [this is the junction between the enamel layer and inner dentin layer] (Anusavice & Benn, 2001, p. 370).  In many cases, maybe most, the lesion will not progress past the enamel or may be reversed with very conservative treatment (e.g. prescription fluoride rinses).

Today, a new controversy exists on how incipient carious lesions, the disease of choice to demonstrate clinical abilities on clinical examinations, should be treated. New research casts doubt on the irreversible nature of incipient lesions. The method of treating the incipient carious lesion is changing to a much more conservative chemotherapeutic method from a more radical surgical approach.  Treating such lesions radically on licensing exams actually ignores recent research findings rather than tests for good clinical skills. In fact, there is a testing-practice mismatch that undermines the credibility of the profession (Formicola et al., 2002, p. 606)

Alternatives to the One-Shot Clinical Exam

Even though the consensus seems to be that we should do away with the high stakes, one-shot clinical exams that involve live patients, there remains general agreement that we should still keep external (outside school) assessments of clinical skills.  These external assessments must be completed in other ways.  Some proposed alternatives include the use of portfolios, residencies, and Objective Standardized Clinical Evaluations (OBSE).  Portfolios offer multiple assessments from many different sources over a long period of time.

In dentistry, competency can be demonstrated only with live patients and with the results of managing the care of representative families of patients over an extended period.  It requires an authentic portfolio.  One-shots and simulations measure at the beginner level, but they do not measure competency (Chambers, 2004a, p. 176)

We utilized portfolios in dental hygiene school, which was a relatively new concept for the program.  I had mixed feelings about their value at the time, but I feel they have great potential.  The next alternative is residencies.  This has recently become an option for dental students in the state of New York.

New York State has taken a bold, progressive step in becoming the first state to allow completion of a residency approved by the Commission on Dental Accreditation (CODA) to replace the traditional clinical examination as a requirement for licensure. With the passage of this new law, the dental profession in New York truly enters the twenty-first century, shaking the very foundations of the traditional dental community by adopting more uniform licensure requirements for all of its surgical professions: medicine, podiatry, and dentistry (Lasky et al., 2003, p. 295)

In New York State the residency requirement is one year in length.  If the residency option were to be opened for the dental hygiene graduate, I think a six month residency would be more appropriate.  The dental hygiene residency option would offer many benefits.  The residency would provide the graduate a better transition from dental hygiene school to independent practice. And the graduate would gain valuable experience in a supervised environment while still earning at least a stipend.  The third alternative involves the utilization of Objective Standardized Clinical Evaluations (OSCE).  A Wikipedia article summarizes what these are very well.

An OSCE usually comprises a circuit of short (usual is 5–10 minutes although some use up to 15 minute) stations, in which each candidate is examined on a one-to-one basis with one or two impartial examiner(s) and either real or simulated patients (actors).  Each station has a different examiner, as opposed to the traditional method of clinical examinations where a candidate would be assigned to an examiner for the entire examination.  Candidates rotate through the stations, completing all the stations on their circuit.  In this way, all candidates take the same stations.  It is considered to be an improvement over traditional examination methods because the stations can be standardised enabling fairer peer comparison and complex procedures can be assessed without endangering patients health (“Objective”, 2009).

The OCSE alternative is currently being utilized in Canada.

In Canada, examiners and educators worked together to establish a licensure examination that recognized that clinical performance is best demonstrated by multiple observations over time and therefore assessed by the faculties of its dental schools. That assessment is supplemented by an independent verification by the National Dental Examining Board of Canada through the written examination and OSCE not involving live human subjects.  Study of results for more than 2000 graduates of 9 schools from 1995 through 2000 confirmed the significantly positive relationship (P < .001) between both the OSCE and the written examination with performance in the final year of dental school. (p.150)

Some disagree with the efficacy of the OSCE alternative, and I find myself in this camp of thought.

The Objective Standardized Clinical Evaluation is not properly a level of testing.  It is a format for presenting items, which may be authentic, but usually are simulations.  A better alternative is portfolios—multiple replicates of multiple sources of evidence regarding the range of skills, understanding and values required to start independent practice (Chambers, 2004a, p. 180)

I think it’s vital that an external assessment be made of the clinical performance on many live patients over a long period time.  This can’t be accomplished by OSCEs or one-shot clinical exams.

Conclusion

The bottom line seems to be that high stakes, one-shot clinical exams that involve live patients should be replaced with better alternatives.  These exams are not valid or reliable, and they also pose significant ethical problems.  Possible alternatives include portfolios, residencies, and Objective Standardized Clinical Evaluations (OSCE).  For dental hygiene licensure, I would favor mandating the following requirements—a passing grade on the National Board Dental Hygiene Examination (NBDHE), graduation from an accredited dental hygiene program, passing an external assessment of the student’s portfolio, and passing an ethical evaluation.  The ethical evaluation should be a 360-degree evaluation with input from the student’s classmates, faculty, and randomly chosen patients.

References

Anusavice, K. & Benn, D (2001).  Is It Time to Change State and Regional Dental Licensure Board Exams In Response to Evidence from Caries Research?  Critical Reviews in Oral Biology & Medicine, 12(5), 368-372.  Retrieved from http://cro.sagepub.com/cgi/reprint/12/5/368

Chambers, D. (2004a).  Portfolios for Determining Initial Licensure Competency.  Journal of the American Dental Association, 135(2), 173-184.  Retrieved from http://www.jada-plus.com/cgi/reprint/135/2/173

Chambers, D., Dugoni, A., & Paisley, I. (2004b).  The Case Against One-Shot Testing for Initial Dental Licensure.  California Dental Association Journal, 32(3), 243-252.  Retrieved from http://mercyimaging.com/page/Library/cda_member/pubs/journal/jour0304/chambers.pdf

Formicola, A. Shub, J. & Murphy, F. (2002).  Banning Live Patients as Test Subjects on Licensing Examinations.  Journal of Dental Education, 66(5), 605-609.  Retrieved from http://www.jdentaled.org/cgi/reprint/66/5/605

Gadbury-Amyot, C., Bray, K., Branson, B., Holt, L., Keselyak, N., Mitchell, T. & Williams, K. (2005).  Predictive Validity of Dental Hygiene Competency Assessment Measures on One-Shot Clinical Licensure Examinations.  Journal of Dental Education, 69(3), 363-370.  Retrieved from http://www.jdentaled.org/cgi/reprint/69/3/363

Lasky, R. & Shub, J. (2003).  Dental Licensure Reaches a Crossroads: The Rationale and Method for Reform.  Journal of Dental Education, 67(3), 295-300.  Retrieved from http://www.jdentaled.org/cgi/reprint/67/3/295

Objective structured clinical examination. (2009, November 2). In Wikipedia, The Free Encyclopedia. Retrieved 01:25, November 3, 2009, from http://en.wikipedia.org/w/index.php?title=Objective_structured_clinical_examination&oldid=323379009

Ranney, R., Haden, N., Weaver, R., & Valachovic, R. (2003).  A Survey of Deans and ADEA Activities on Dental Licensure Issues.  Journal of Dental Education, 67(10), 1149-1160.  Retrieved from http://www.jdentaled.org/cgi/reprint/67/10/1149

Ranney, R., Gunsolley, J., Miller, L. & Wood, M. (2004).  The Relationship Between Performance in a Dental School and Performance on a Clinical Examination for Licensure: A Nine-Year Study.  Journal of the American Dental Association, 135(8), 1146-1153.  Retrieved from http://www.adajournal.com/cgi/reprint/135/8/1146

Ranney, R. (2006).  What the Available Evidence on Clinical Licensure Exams Shows.  Journal of Evidence Based Dental Practice, 6(1), 148-154.  Retrieved from http://www.sciencedirect.com/science/article/B75B8-4JW7C3W-1C/2/6e95d1b3b5951b3c521513a8cc933aca