# Artificially Intelligent

Any mimicry distinguishable from the original is insufficiently advanced.

| 3574 words

# Key Takeaways

Arguments for earning to give (EtG) as an impactful career have a simplified conception of the market for talent. Assuming a functioning market, I argue there are many forces pushing the price of talent upwards. I also argue that the market for talent is dysfunctional and buying talent has to navigate supply not meeting demand. Finally, I provide reasons why buying talent is undesirable, even if it’s possible.

Overall, these arguments lend nuance to how money can be used altruistically. In particular, they suggest that those considering EtG as their primary career path might want to consider direct work instead.

# Introduction

When I was first introduced to effective altruism, I thought that money was the main lever used to move the world. I thought that money was relatively fungible with nearly all resources. In particular, money allowed you to change the distribution of problems on which talented people were working.

I accepted that earning to give (EtG) dominated direct work because you can always funge money with someone else doing direct work. Since you’ll earn the “market rate” for your talent, the only cost is living expenses. Living expenses are low, so EtG is at least about as good as direct work. EtG has higher flexibility it is likely better.

This argument is flawed. One reason is the “market rate” people get paid is less than the value they could contribute by doing direct work.1 However, the main reason is that I no longer think money can be easily and efficiently converted into talent, which I will argue.

This post will mostly be written from an impatient longtermist perspective. This means I am mostly thinking about talent as the ability to help prevent global catastrophic risks, e.g. unaligned artificial general intelligence or global pandemics. I expect many of these arguments to fail when applied to non-longtermist cause areas.

The argument:

Some people are more talented than other people: more productive, more intelligent, more personable, etc. Talented people are better able to achieve goals, like creating software, managing people, or conducting research. Organizations want to hire talented people. This gives us a supply of and demand for talented people.

The economy uses pricing to match supply with demand. Money should be able to efficiently buy talent just like it can buy other goods. Thus, being willing to pay more money for talented people should increase the amount you can attract.

This argument is compelling. However, it makes multiple false assumptions about the market for talent.

# Challenges

## Money can’t always buy quality

In the market for iPhones, you can buy more iPhones for more money. At reasonable levels of iPhones, this relationship will be linear. If you buy a significant fraction of the current supply of iPhones, you increase demand and pay more per iPhone. If you want to buy more iPhones than exist on Earth, then you have to pay for new iPhones factories, which will make the marginal iPhone cost more.

Suppose you wanted an iPhone 15. How much would this cost you? There isn’t any supply. To obtain one, you would have to make it. Apple is spending billions on R&D. How much money would you need to speed up the process? At least 1000x the price of an iPhone 12.

It’s tempting to think of a unit of talent like an iPhone - you can buy some talent for some price, so you can buy double the talent for double the price. This is technically true if you hire two people. However, due to communication and managerial overhead, two people will do less than twice the work of one person or cost more than twice as much. One person with twice the talent is more valuable than two people, just like an iPhone 15 is more valuable than two iPhone 12s.

As another analogy, consider microSD cards. I can buy a 512 GB microSD for $65. A 1 TB microSD card costs$330, about 5x as much for a 2x storage increase. How much would I have to pay for a 2 TB microSD? It is not yet available generally, but it might be available in an R&D department. Said department would probably not part with it for a million dollars.

Applied to talent, imagine Alice can think twice as fast as Bob. How much more is Alice worth? Naively, twice as much as Bob. However, suppose that Alice and Bob work at competing trading firms. Both firms have capital, so they can maximally exploit any market opportunities their traders spot. Thus, it’s winner takes all; whichever firm spots the market inefficiency first gets all of the profit. Alice thinks twice as fast as Bob, so she always spots the best market inefficiencies first. If some market inefficiencies are 10x more profitable than others, Alice is worth 10x more than Bob.

In general, additional copies of a good can usually be purchased for the same marginal price. However, higher quality products are a different market where one might have to pay arbitrarly more for marginally higher quality.

In the limit, there are things that people with nearly unlimited resources cannot purchase.

As the American Revolution was heating up, a wave of smallpox was raging on the other side of the Atlantic. An English dairy farmer named Benjamin Jesty was concerned for his wife and children…When smallpox began to pop up in Dorset, Jesty decided to take drastic action. He took his family to a nearby farm with a cowpox-infected cow, scratched their arms, and wiped pus from the infected cow on the scratches…Throughout the rest of their lives, through multiple waves of smallpox, they were immune…

The same wave of smallpox which ran across England in 1774 also made its way across Europe. In May, it reached Louis XV, King of France. Despite the wealth of a major government and the talents of Europe’s most respected doctors, Louis XV died of smallpox on May 10, 1774.

This observation about quality only establishes that the price of talent might be non-linear. Why might we expect it to be sharply non-linear in practice?

## Talent is rare

Price is controlled by supply and demand. What’s the supply? Pretty low. If you’re reading this post, you’ve probably spent your entire life interacting with people who have already been selected for intelligence, motivation, conscientiousness, etc.

As an example, students at top US universities frequently have ACT scores of 35+.2 In 2020, about twenty thousand students out of 1.7 million scored 35+, which is about 1%.3 These students have already been selected from those who take the ACT, which have already been selected from those who are going to college.

But percentile rankings don’t matter. Reality doesn’t grade on a curve. How hard is it to find talented individuals? The 95th percentile isn’t that good. The 99th percentile probably isn’t good enough either. Translating percentiles into talent is difficult. My experience has been interacting with a sample, then realizing that the sample is more skewed than I initially thought.

Additionally, EA organizations desire particular talents. For example, many organizations don’t have much training capacity. Such organizations are interested in hiring people that can “hit the ground running.” Ben Todd:

[Open Philanthropy is] not particularly constrained by finding people who have a strong resume who seemed quite aligned with their mission, but they are still constrained by someone who can just kind of like hit the ground running as a researcher. And some evidence for this is that they trialed 12 people, I think for like three to six months. But of those, they only hired five. And they’re like a multi-billion dollar foundation. So they clearly have the funds to hire more people if they found people above the bar.

The supply of talented individuals is low. What about demand?

## Talent is very desirable

Hedge funds turn talent into profit. In particular, the compensation an individual receives is directly proportional to their talent. Thus, hedge funds create a demand for talent at the appropriate level.

Assuming the market functions properly, the amount one talented individual costs is the amount they would have been paid elsewhere. Since talent is also useful for making money, this amount can be large. To be specific, I have friends that might become quantitative traders/analysts. Their expected salaries are at least $1 million per year over their careers, plausibly 2-10 times larger. I expect many top longtermist researchers to be able to obtain similar salaries if they switched to finance. At$1 million per year, $10 billion buys 10,000 researcher-years, equivalent to 250 researcher-careers.4 That’s not a lot. And also probably an overestimate. Some people are altruistic and might be willing to do direct work for less than their expected salary. Might we be able to use money to buy these people? Not quite. Assuming market efficiency, the savings of supplementing an altruistic individual is balanced by lost counterfactual donations. Suppose that Alice currently makes$200,000 a year and is willing to do direct work for $100,000 a year. Hiring Alice seems like a good deal - we get$200,000 of talent for 100,000. However, if Alice would do direct work for half the salary, she should also be willing to donate half her salary. Our savings are balanced by lost donations. This argument is false because many altruistic individuals are willing to take a larger pay-cut than they’re willing to donate; people value the type of work they do. Additionally, people rarely get paid an equal to the value of their work, so doing direct work is more efficient than EtG. In practice, using money to hire talented altruistic individuals is highly impactful. Assuming a market for talent, we have high demand. Is there even a market? ## Talent isn’t a perfect market Many of my friends work as software engineers at tech companies and haven’t looked for jobs outside of big tech. These individuals might be able to double their salaries at hedge funds or startups. What’s going on here? (Thanks to Zvi for inspiring this analysis. In what follows, the producers are the talented individuals and the consumers are the companies that want to hire them. I am unaware of the strength of most of these effects.) Products/producers are not homogenized; information about them is costly for consumers. Even relatively homogenized products, like generic software engineers, differ vastly in quality between individuals. There is no cheap and reliable way to differentiate between levels of talent without costly trial periods.5 Hiring employees with specific sub-skills is difficult. Reputation and experience matter. Hiring requires long-term predictions. There is often a substantial lag between hiring someone and having them start, and when an employee isn’t a good fit this can take time to determine. At the same time, the nature of GiveWell’s work is constantly changing. Thus, doing good evaluations of employees is both difficult and very important; the cost of an overly optimistic hire or evaluation can be significant. Consumers are not homogenized; information is costly for producers. Personal fit is essential for productivity, happiness, career advancement, etc. However, it is difficult to determine fit; what was initially exciting might become boring after a few months. You might attempt to endure only to burn out. People don’t know their own abilities. They don’t know their market rate and don’t shop around to find out. Stigma around salaries means people don’t know they’re being underpaid. Producers have imperfect information. People do not know if they will want to switch jobs. Jobs at large companies allow for transfering to different departments, should the desire arise. Large companies offer stability. People might not know some jobs exist. Consumers have imperfect information. Even with internship programs, employers cannot entirely measure a person. Hires from top universities possess more background knowledge. People might quit, which is costly for employers. Each person also alters workplace culture, which cannot be measured but must be maintained. Fixed costs exist for producers. Searching for jobs is an expensive and time-consuming process. The skills one needs to interview well require cultivation (read: leetcode). If you’re employed, spending a day interviewing might cost hundreds in potential earnings. Applying for jobs can be demoralizing and damage advancement in your current job. Fixed costs exist for consumers. Hiring employees is expensive and time-consuming, often requiring dozens of full-time employees. Technical interviews are conducted by technically skilled people whose time is expensive.6 Economies of scale exist; high variance high value actions must be taken many times. Asking people to apply for jobs rarely works. Large companies attract talent with reputation In order to reach the point where employees are producing work that we can rely on and integrate into our output, senior staff need to invest significant time in training, evaluation and management. Producer preferences differ. People have different job fits. Someone could be a great trader if only they were passionate about arbitrage. Doing enjoyable work might be a requirement. People also have preferences about workplace culture, job respectability, job stability, coworker demeanor, hour flexibility, office location, etc. Producers might care about individual consumers. People often have dreams of working in specific industries. If these dreams are strong enough, they might be unpersuadable. People have ethical qualms about working in other industries, e.g. finance. Producers are not strategic. People who ask for raises make more than people who don’t, and yet there are people who don’t ask for raises. College graduates frequently take their first offer they get because they’re not applying to enough companies. People make mistakes in negotiation that cost them hundreds of thousands of dollars of potential earnings, yet they spend very little time practicing. People are unmotivated to develop the skills they lack or discover their fair market value. Consumers are not strategic. Biases plague job interviews, and yet employers often don’t take basic measures like blinding. Technical interviews are conducted by volunteers instead of trained individuals. Coding quizzes are developed on an ad hoc basis instead of treated as serious endeavors. Basic statistics correlating interview performance to job performance are not taken. Employers give large weight to unreliable subjective impressions of “fit” and “character”. Consumers are constrained. Employees at large companies systematically make less than the value they generate. Small companies pay with equity instead of capital which carries more risk. Hires must be legibly defensible. The market for talent is flawed; supply doesn’t always meet demand. Can people be bought? ## People don’t value money Suppose you wanted Terence Tao to work at your hedge fund. How much would you have to pay him? I think he wouldn’t switch jobs for any reasonable price. I sometimes ask my friends how much money someone would have to pay them to live as a hermit in the woods. Some of my friends said they would not do this for any amount of money. Their current lives were good, they reasoned; what would they even do with more money? Most people suffer extremely sharp diminishing returns to large sums of money, problematizing the market for talent. As people have more money, their desires shift: work-life balance, passion, location, etc. If someone is passionate about their work, no amount of money may be sufficient. There also might be systematic tendencies for talented individuals to value money less. Paul Graham: “If I had to put the recipe for genius into one sentence, that might be it: to have a disinterested obsession with something that matters.” If Graham is right, talented people have talent precisely because they don’t care for money. Paul Erdős wouldn’t have left mathematics for any worldly thing. Many talented individuals don’t value money, but some might. Would we want to buy talent with money? ## Alignment is important An analogy: You’re at war with Examplestan. Examplestan doesn’t maintain a large standing army;the bulk of their forces are mercenaries. You are richer than Examplestan. You pay Examplestan’s mercenaries to join your side, easily winning the war.7 A hypothetical: You discover the head of an AI Safety research organization is willing to switch to any job, including harmful ones, if offered enough money. Would that make you uncomfortable? It certainly would for me. I understand AI Safety better than most, but I cannot independently verify many research directions. Part of my confidence that the work is useful stems from trust. Suppose someone is researching the evolutionary history of the immune system to help prevent global pandemics. I do not know if this is useful. I would be more comfortable funding this person if they were researching this topic for global pandemic prevention; it makes them likely to shift focus if their research is misguided. If they were researching for enjoyment, I would be wary of funding them. Paying people to do things is a value alignment problem. I want you to do what I want, so I pay you money. But how do I know that you’ll do what I want? If you don’t share my values, I have to use oversight. In disciplines like software engineering supervisors can assess performance metrics. In disciplines like research such mechanisms are hard to implement.8 If core EA organizations were older, they could make use of talented, unaligned individuals. However, the EA movement in 2020 does not have sufficient infrastructure. My guess is that employees receive only vague directions, coupled with instructions to “do what I mean” or “do what you think is best.” Many EA organizations are functional only because the employees are value aligned, so such organizations do not fall prey to Goodhart’s law.9 Given the current lack of infrastructure to make use of unaligned individuals, the benefits of hiring such people is low. This analysis recommends building infrastructure that allows for money to be used more effectively, e.g. increasing the management and training capacity of existing organizations. # Conclusion Basic economics suggests that talent, like any other good, should be available for a price. However, this argument has multiple complications: • Highly talented individuals are a different class of good than less talented individuals and can cost arbitrarily more. • People talented in ways useful for direct work are rare. • Such people are in high demand, making them expensive. • The market for talent is inefficient. • Talented people might not value money. • Value misalignment makes it difficult to use such people given current EA management capacity. Some expensive things are worth buying. The arguments above are only strong enough to complicate the intuition that one can buy talent for money. There are also other subtleties that I have not adequately explained.10 Overall, I see many people new to effective altruism considering EtG as their primary career. Arguments for the effectiveness of EtG, especially in the longtermist space, rely on simplifications of the market for talent. I hope to add nuance to this market and encourage young EAs to consider direct work as a career. 1. An example of this is quantitative traders, who are at the limit of getting efficiently compensated for the talent and only get 10-20% of their profits. 2. https://blog.prepscholar.com/whats-a-good-sat-act-score-for-the-ivy-league-whats-the-lowest-score-i-can-get-in-with 3. https://blog.prepscholar.com/how-many-people-get-a-34-35-36-on-the-act-score-breakdown 4. Assuming 40 years per career. 5. I don’t have data on this, but it would not be surprising to me if most interns at tech companies were net-negative in terms of revenue, i.e. they cost more in money/management time than the value they produce. Of course, they’re still likely worth it to the company because determining the talent of a potential employee is worth much more than the20,000-$50,000 they pay the intern. 6. As anecdata, as part of an interview process, my friend spent an hour talking to someone who made upwards of$5 million a year. Naively, that conversation cost the company more than \$2500. In practice, that person worked more than 40 hours a week and didn’t have to spend that much energy conversing, lowering the cost.

7. I have no knowledge about mercenaries . This is also a bad example because Examplestan would not want to hire mercenaries that would switch sides in the middle of a battle. For reasons of reputation, the mercenaries would then not want to switch. However, they might switch if paid enough money that reputation no longer mattered.

8. Of course, such mechanisms probably cause less effective behavior due to Goodhart’s Law

9. I have very little experience working for EA organizations. This perspective was informed by conversing with some people who do have such experience and listening to this episode of the 80,000 Hours podcast.

10. One example is the vetting problem: how do you robustly find talented and altruistic individuals?