A research paper has examined the performance of ChatGPT and GPT-4’s ability to respond to mock examination questions from the Chartered Financial Analyst Institute (CFA Institute).
The research paper issued on 12 Oct by Queen’s University Canada, Virginia Tech and JP Morgan, found that ChatGPT would not be able to pass the CFA’s exam, but GPT-4 may have a “decent chance” of passing with proper prompting. The paper gives important insight into the degree of automation workers will face in an age of AI.
Go deeper with GlobalData
The study also found that GPT-4 outperformed ChatGPT in almost every topic of both levels I and II examinations.
The CFA Institute’s exam comprises three levels, including straightforward multiple choice questions and vignette-support multiple choice questions on a variety of subjects related to ethics, professional standards, and financial analysis.
Notably, the researchers found that both ChatGPT and GPT-4 answered the questions more accurately when prompted with “few-shot prompting”, which requires giving example questions.
Few-shot prompting is the technique of providing a few examples, known as “shots,” to teach an LLM to generate desired outputs either as text, code, or images.
How well do you really know your competitors?
Access the most comprehensive Company Profiles on the market, powered by GlobalData. Save hours of research. Gain competitive edge.
Thank you!
Your download email will arrive shortly
Not ready to buy yet? Download a free sample
We are confident about the unique quality of our Company Profiles. However, we want you to make the most beneficial decision for your business, so we offer a free sample that you can download by submitting the below form
By GlobalDataAccording to research company GlobalData’s Executive Briefing on AI issued in October, the financial services sector has a high exposure to AI disruption ranking third after tech, media and telecoms and healthcare.
Verdict spoke with Ethan Callanan, one of the researchers behind the study on how the paper came about.
Callanan’s supervisor, Professor Xiaodan Zhu of Queen’s University Canada, had collaborated with JP Morgan’s AI Research through the JP Morgan Faculty Research Award.
In pursuit of his Master’s degree in AI, Callanan had been tasked with finding financial tasks or problems that had not already been tested with LLMs.
“One of the first things [Prof. Zhu and our JP Morgan collaborators] suggested I look for was how LLMs perform on the CFA exam since it is a reputable exam for humans to demonstrate their financial abilities,” Callanan explained, “we were surprised to find there was no formal evaluation for the CFA yet, especially considering other exams like the LSAT had all been done relatively early on.”
Asides from a scattering of anecdotal blog posts and informal articles, the researchers found no evidence that ChatGPT or GPT-4 had been used on the full CFA exam.
“So, we proposed the idea and quickly decided it was well worth doing, devised a research plan and got to work,” stated Callanan.
Whilst it may seem futuristic to put ChatGPT in the role of a financial analyst, financial services company Moody’s recently announced it would be incorporating LLMs into its financial analysis this October. Furthermore, around 28% of businesses answered that they would feel extremely confident incorporating ChatGPT into their daily workflows and systems.
So how worried should financial analysts be?
For Dr. Fabian Stephany of the Oxford Internet Institute mass workforce automation seems an unlikely prospect at this time. Whilst this paper suggests that LLMs can pass the CFA, which Stephany reminded Verdict was a “prominent foundation” for many financial careers, LLM technology currently lacks many human qualities needed to be a successful analyst.
“Passing the CFA is just a necessary but far from sufficient condition of being a successful financial analyst,” stated Stephany, “It takes far more, in terms of human interaction, intuition, experience, to master finance.”
Stephany also stated that LLMs have access, to a point, to internet data and information which is why they are able to answer CFA questions. A good benchmark for future similar research, states Stephany, may be comparing ChatGPT’s answers to those of an unskilled human with internet access.
Whilst current LLMs may not be able to completely automate analysis, their ability to understand large quantities of data almost instantly may make them a useful tool for analysts in the future.
Our signals coverage is powered by GlobalData’s Thematic Engine, which tags millions of data items across six alternative datasets — patents, jobs, deals, company filings, social media mentions and news — to themes, sectors and companies. These signals enhance our predictive capabilities, helping us to identify the most disruptive threats across each of the sectors we cover and the companies best placed to succeed.
