Over the past year or so, ChatGPT’s Large Language Model (LLM) has demonstrated an uncanny ability to best people at something that is a cornerstone of our young professional lives.
it was pass Passed three notoriously difficult exams for medical school, the law school bar exam, and an MBA from the University of Pennsylvania’s Wharton School of Business.
Also: What is ChatGPT and why is it important? Here’s what you need to know
The scores posted by the LL.M. were modest passing grades. But its next incarnation — GPT-4 — is supposedly a joke good The student, ahead of his parent, cruised with 90th percentile score in the bar exam and got close to-perfect Marks in GRE Oral Test.
So, it must come as a huge source of both satisfaction and relief to us folks that there’s at least one thing that LLMs like ChatGPT aren’t good at — or actually terrible at: accounting.
Also: How to Use ChatGPT to Write Excel Formulas
Many users of ChatGPT have publicly commented on how the simplest math functions have foxed it. However, there is a large and strictly executed Study ChatGPT’s accounting capabilities include what Brigham Young University (BYU) accounting professor David Wood adopted several months ago.
Test conditions
Wood decided to harness the power of the global accounting fraternity with a pitch on social media that asked ChatGPT to help pace a variety of global accounting exams.
There was a flood of recipients: 327 co-authors from 186 educational institutions located in 14 countries participated in the study. They collectively submitted 25,181 classroom accounting exam questions — as well as more than 2,000 questions from his own department at BYU — to ChatGPT.
A comprehensive accounting exam usually has questions covering all major topics. such as financial accounting, auditing, managerial accounting, tax, and others, and had different types (multiple choice, short answer, true/false) and difficulty levels.
Also: How to use ChatGPT to create charts and tables
The results were unequivocal: ChatGPT produced a 47.4% result which, in and of itself, wasn’t bad. The students, however, scored an overall average of 76.7% and easily bested the machine.
According to the survey, the LLM did fine in things like auditing. But according to Wood’s paper, it had trouble getting its artificial neurons around problems that dealt with tax, financial and managerial valuation problems — and these were categories that involved a lot of math.
AI’s math doesn’t add up
Many people cannot quite come to terms with AI’s fearsome reputation as a potential killer of humanity, sometimes with AI’s inability to do simple math.
Also: ChatGPT seems to be confused about when the wisdom ends
Yet the truth is that ChatGPT is essentially a glorified one prophetic Text program — given a large amount of data and then trained to identify correct and incorrect answers.
Its ability to spit out conversational answers to questions to be unusually human is because it is built to understand the underlying patterns of language and connections between words, but not numbers. (This is why it is called the ‘language’ model.)
This depends on the output of AI LLM possibility, and is not correct. The output, by design, is architected to present an answer that has the highest statistical probability for the question asked.
Also: How does ChatGPT actually work?
And the numbers, sadly, don’t work like that.
Answers involving math or many forms of accounting must be specific and not approximate. They depend on an exact output, like what a calculator gives you, and not on relationships between words.
Paulo Shakarian, an associate professor of engineering at Arizona State University who runs a lab exploring the challenges facing AI, Study which measures ChatGPT’s performance on mathematical word problems.
Solving these word problems involves multiple steps, which require translating the words into mathematical equations. But this kind of multi-step process also requires logical reasoning, which algorithms are not designed to do.
Also: Can generative AI solve computer science’s biggest unsolved problems?
“Our preliminary tests on ChatGPT, which were conducted in early January, indicate that performance for state-of-the-art algorithms for math word problem solvers is significantly below 60% accuracy,” added Shakarian.
bright spot
So, where is LLM like ChatGPT Excel?
Another professor at the University of Pennsylvania’s Wharton School of Business, Christian Terwish, had a very different view. experience Including a case study of business school recruits.
“On some problems, the math was horrible,” Tarvish said said.
Also: Can AI code? Only in baby steps
However, when given a case involving troubleshooting a bottleneck process at a hypothetical iron ore plant in Latin America, ChatGPT excelled.
“Wow! The answer is not only correct, but also brilliantly explained,” Terwish wrote in a paper about his experiment. “I see no reason to deduct points from this answer: A+!”
The overall grade for the entire MBA exam was a B or close to a B-, Terwish said, primarily because of the bot’s strength in operations management and process analysis, which many finance and management staff are paid heavily to do. .
Another area of high AI expertise: tearing down tedious tasks, such as processing invoices, tabulating and categorizing expenses, working with data entry, and similar fields.
Also: Extending ChatGPT: Can AI Chatbot Plugins Really Change the Game?
But above all, ChatGPT gave Wood, the BYU professor, an unparalleled ability to introspect on what staff were teaching students — and how they were doing it.
“When this technology first came out, everyone was concerned that students could now use it to cheat,” he said.
“But the opportunity to cheat is always there. So for us, we’re trying to focus now on what we can do with this technology that we couldn’t do before to improve the teaching process for faculty and the learning process for students. The experiment was eye-opening.”
Meanwhile, letting an AI LLM do your taxes for you is probably not a good idea.