25. April 2025
California Law Board Admits Ai Helped Develop Bar Exam Questions
The State Bar of California has acknowledged that some multiple-choice questions on its February 2025 bar exam were developed with the assistance of artificial intelligence (AI). This revelation has sparked controversy and raised concerns about the use of AI in legal education and assessment. The February 2025 bar exam was widely criticized for its technical issues, which prevented many test-takers from completing their exams.
A total of 23 scored multiple-choice questions were developed using AI by ACS Ventures, a company hired by the state bar to create and assess the exam. These questions were created with the help of AI algorithms designed to mimic the style and structure of real law school exams. The questions were then reviewed and approved by experts in the field before being included on the actual exam.
This raises questions about the role of human judgment in the development and assessment of legal knowledge. Mary Basick, assistant dean of academic skills at the University of California, Irvine School of Law, expressed her shock and concern about the use of AI to develop bar exam questions. “The debacle that was the February 2025 bar exam is worse than we imagined,” she said.
Basick’s comments highlight the potential risks of relying on AI to create and assess complex legal concepts. Leah Wilson, executive director of the state bar, downplayed concerns about the use of AI in exam development, stating that the organization has “confidence in the validity” of the questions. However, Katie Moran, an associate professor at the University of San Francisco School of Law who specializes in bar exam preparation, was more critical.
Moran argued that the state bar may be trying to downplay the significance of using AI in exam development by implying that non-lawyers can create and assess complex legal concepts without proper oversight. Andrew Perlman, dean of Suffolk University Law School and an advisory council member of the American Bar Association taskforce on the law and artificial intelligence, shared Moran’s concerns.
“I’m not surprised to hear about AI being used to develop bar exam questions,” Perlman said. “But what I am worried about is making sure that everything that comes from an AI tool is vetted carefully by experts in the subject matter.” Perlman emphasized the importance of human judgment and oversight in ensuring the accuracy and validity of legal knowledge.
The use of AI in law school education has been growing rapidly, with many institutions incorporating AI-powered tools into their curricula. However, the question remains whether these tools can truly replace human expertise and judgment. Perlman noted that while AI has the potential to enhance learning and assessment, it also raises concerns about the role of human expertise and judgment in developing and assessing legal knowledge.
The state bar’s decision to adjust test scores for those who took the February 2025 bar exam has sparked debate about the role of AI in legal education and assessment. While some argue that this move is necessary to address technical issues, others question whether it is a sufficient response to the controversy surrounding AI use.
Ultimately, the future of legal education and assessment will depend on our ability to harness the power of AI while preserving human judgment and expertise. By prioritizing transparency and accountability, we can ensure that lawyers are equipped with the knowledge and skills necessary to succeed in their careers.
The controversy surrounding the use of AI in bar exam development is a wake-up call for the education and assessment community. It highlights the need for greater attention to detail and a commitment to quality control in the development and assessment of legal knowledge. By prioritizing human judgment and oversight, we can ensure that these tools are used effectively and accurately.
The use of AI in law school education and assessment represents both opportunities and challenges for the profession. While it has the potential to enhance learning and assessment, it also raises concerns about the role of human expertise and judgment in developing and assessing legal knowledge. By prioritizing transparency and accountability, we can harness the power of AI while preserving the integrity of our education and assessment systems.
As the field of law continues to evolve, it is essential that we stay vigilant about the use of AI in education and assessment. By doing so, we can ensure that lawyers are equipped with the knowledge and skills necessary to succeed in their careers.