ChatGPT Uncovers the Shortcomings of Our Licensing Exams

This op-ed was originally published in Deseret News on April 11, 2023.

Recently, ChatGPT performed at or near the threshold of 60% accuracy on the United States Medical Licensing Exam without specialized input. This artificial intelligence program has also passed the bar exam and earned passing grades on law school essays. A list of other challenging exams passed by AI can be found here.

The performance of ChatGPT on examinations often used to gatekeep entry to professions reveals a few things. First, these exams don’t do a great job of accounting for the hands-on training or experience often required to become a licensed professional (see ChatGPT passing the medical licensing exam without clinician input). Second, these exam results display that prospective professionals can recall the same fact-based information, but often at a much slower pace than AI. 

If ChatGPT has the potential to pass exams used in occupational licensing without any specialized input, are these tests really an effective way to guarantee consumer safety or ensure a qualified workforce?

Ensuring Competency 

Undoubtedly, making sure professionals are competent and well-trained enough to safely perform their professional duties is imperative. Nobody wants a clueless surgeon or an untrained pilot. However, current licensing exams for many professions may not be the indicator they are touted to be. Instead, these exams may actually present unneeded barriers to entry for professionals and can disproportionately exclude minority groups from licensure. 

An example of this is demonstrated through social worker examination requirements. Many states, including Utah, require or have required various types of social workers to pass a licensing examination administered by the Association of Social Work Boards in order to become fully licensed professionals.

These exams are intended to gauge a social worker’s ability to practice ethically and safely. They are currently mandatory for gaining licensure in nearly every U.S. state at various levels of licensure. However, the necessity and value of these exams have been challenged. 

Recent pass-rate data from the social work association has revealed significant disparities in passing rates between white test-takers and those of other racial and ethnic groups. This data suggests that the exams administered by the association may be biased and, as a result, harm hopeful Black, Latino/Hispanic and Indigenous social workers. Instead of serving their intended purpose, these exams appear to inadvertently promote economic protectionism and insulate those already licensed from the potential competition of newly licensed professionals.

It is worth noting that HB250, passed during the 2023 Utah legislative session, removed some of these examination requirements for licensure as a certified social worker or social service worker.

Clearly, these examination requirements need to be rethought in their use or ability to demonstrate competency. Many examination requirements could likely be removed without causing any danger, and for those that remain in place, a greater emphasis should be placed on uniquely human components, not just memorization.

The Silver Lining

There is a silver lining to AI demonstrating competencies on such exams: it may be able to complete busywork, freeing humans to do uniquely human things. However, making use of AI’s capabilities will require more regulatory flexibility.

As explained by the State Policy Network, a regulatory sandbox is a “legal classification that creates a space where participating businesses won’t be subject to onerous regulations — usually for a limited amount of time. The point is to allow these businesses to play in the sandbox without regulations to see if innovative ideas and products can get traction and enter the market.”

These sandboxes can allow for innovative AI tools to become available to the public quickly and safely. Such a framework authorizes the state and its various industries to work quickly to maximize AI’s potential in ways that can better communities and effectively serve Utahns. 

Utah’s lawmakers will have a plethora of regulatory decisions to make surrounding these tools. With these decisions, lawmakers must allow the benefits of the tools to be reaped by their citizens to their fullest extent. It’s also crucial that legislators learn from these tools about the regulatory structures that successfully incorporate them into the state’s economy (e.g., regulatory sandboxes) and what structures AI reveals a need for reform (e.g., licensing exams).