OpenAI, the company behind the large language model that powers its chatbot ChatGPT, made the claim in March last year, and the announcement sent shock waves around the web and the legal profession.
But it turns out the artificial intelligence model only scored in the top 10% when compared with repeat test takers. When Martínez contrasted the model's performance more generally, the LLM scored in the 69th percentile of all test takers and in the 48th percentile of those taking the test for the first time.
To investigate the results further, Martínez made GPT-4 repeat the test again according to the parameters set by the authors of the original study. The UBE typically consists of three components: the multiple-choice Multistate Bar Examination ; the Multistate Performance Test that makes examinees perform various lawyering tasks; and the written Multistate Essay Examination .
"Although the leap from GPT-3.5 was undoubtedly impressive and very much worthy of attention, the fact that GPT-4 particularly struggled on essay writing compared to practicing lawyers indicates that large language models, at least on their own, struggle on tasks that more closely resemble what a lawyer does on a daily basis," Martínez said.