OpenAI Study Predicts 80% Of Jobs Will Be Impacted By AI.

Oil On The Hype Fire Or Sound Analysis?

We all have heard an uncountable amount of predictions about how AI will terk err jerbs!

However, here we have a proper study on the topic from OpenAI and the University of Pennsylvania. They investigate how Generative Pre-trained Transformers (GPTs) could automate tasks across different occupations [1].

Although I’m going to discuss how the study comes with a set of “imperfections”, the findings still make me really excited. The findings suggest that machine learning is going to deliver some serious productivity gains.

People in the data science world fought tooth and nail for years to squeeze some value out of incomplete data sets from scattered sources while hand-holding people on their way toward a data-driven organization. At the same time, the media was flooded with predictions of omniscient AI right around the corner.

Let’s dive in and take an exciting glimpse into the future of labor markets!

What They Did

The study looks at all US occupations. It breaks them down into tasks and assesses the possible level of for each task. They use that to estimate how much automation is possible for a given occupation.

The researchers used the O*NET database, which is an occupation database specifically for the U.S. market. It lists 1,016 occupations along with its standardized descriptions of tasks.

The researchers annotated each task once manually and once using GPT-4. Thereby, each task was labeled as either somewhat (<50%) or significantly (>50%) automatable through LLMs. In their judgment, they considered both the direct “exposure” of a task to GPT as well as to a secondary GPT-powered system, e.g. LLMs integrated with image generation systems.

To reiterate, a higher “exposure” means that an occupation is more likely to get automated.

Lastly, they enriched the occupation data with wages and demographic information. This was used to determine whether e. g. high or low-paying jobs are at higher risk to be automated.

So far so good. This all sounds pretty decent. Sure, there is a lot of qualitative judgment going into their data acquisition process. However, we gotta cut them some slag. These kinds of studies always struggle to get any hard data and so far they did a good job.

However, there are a few obvious things to criticize. But before we get to that let’s look at their results.

Key Findings

The study finds that 80% of the US workforce, across all industries, could have at least some tasks affected. Even more significantly, 19% of occupations are expected to have at least half of their tasks significantly automated!

Furthermore, they find that higher levels of automation exposure are associated with:

• Programming and writing skills

• Higher wages (contrary to previous research!)

• Higher levels of education (Bachelor’s and up)

Lower levels of exposure are associated with:

• Science and critical thinking skills

• Manual work and tasks that might potentially be done using physical robots

This is somewhat unsurprising. We of course know that LLMs will likely not increase productivity in the plumbing business. However, their findings underline again how different this wave is. In the past, simple and repetitive tasks fell prey to automation.

This time it’s the suits!

If we took this study at face value, many of us could start thinking about life as full-time pensioners.

But not so fast! This, like almost the other studies on the topic, has some flaws.

Necessary Criticism

First, let’s address the elephant in the room!

OpenAI co-authored the study. They have a vested interest in the hype around AI, both for commercial and regulatory reasons. Even if the external researchers performed their work with the utmost thoroughness and integrity, which I would be willing to assume they did, the involvement of OpenAI could have introduced an unconscious bias.

But there’s more!

The occupation database contains over 1000 occupations broken down into tasks. Neither GPT-4 nor the human labelers can possibly have a complete understanding of all the tasks across all occupations. Hence, their judgment about how much a certain task can be automated has to be rather hand-wavy in many cases.

Flaws in the data also arise from the GPT-based labeling itself.

The internet is flooded with countless sensationalist articles about how AI will replace jobs. It is hard to gauge whether this actually causes GPT models to be more optimistic when it comes to their own impact on society. However, it is possible and should not be neglected.

The authors do also not really distinguish between labor-augmenting and labor-displacing effects and it is hard to know what “affected by” or “exposed to LLMs” actually means. Will people be replaced or will they just be able to do more?

Last but not least, lists of tasks most likely do not capture all requirements in a given occupation. For instance "making someone feel cared for" can be an essential part of a job but might be neglected in such a list.

Take-Away And Implications

GPT models have the world in a frenzy - rightfully so.

Nobody knows whether 19% of knowledge work gets heavily automated or if it is only 10%.

As the dust settles, we will begin to see how the ecosystem develops and how productivity in different industries can be increased. Time will tell whether foundational LLMs, specialized smaller models, or vertical tools built on top of APIs will be having the biggest impact.

In any case, these technologies have the potential to create unimaginable value for the world. At the same time, change rarely happens without pain. I strongly believe in human ingenuity and our ability to adapt to change. All in all, the study - flaws aside - represents an honest attempt at gauging the future.

Efforts like this and their scrutiny are our best shot at navigating the future. Well, or we all get chased out of the city by pitchforks.

Jokes aside!

What an exciting time for science and humanity!

As always, I really enjoyed making this for you and I sincerely hope you found value in it!

If you are not subscribed to the newsletter yet, click here to sign up! I send out a thoughtful 5-minute email every week to keep you in the loop about machine learning research and the data economy.

Thank you for reading and I see you next week ⭕!

References:

[1] https://arxiv.org/abs/2303.10130