News Report Technology
October 18, 2023

GPT-5 Training Will Cost $2.5 Billion and Start Next Year

GPT-5 Training Will Cost $2.5 Billion and Start Next Year

Twitter user Martin Shkreli from NY posted today that GPT-5 will require an estimated $2.0-$2.5 billion for training. This training would involve 500,000 H100s Tensor Core GPU over 90 days or an alternative configuration. The training is set to commence next year.

OpenAI is actively working on enhancing GPT-4 with various capabilities, such as embodiment, agency, Socratic reasoning, knowledge graphs, world models, multimodality, planning, semantic interpretability, hive minds, control and boundedness, as well as smaller high-value tasks.

The scale of H100/A100 production raises questions. Will there be enough of these GPUs available for such a substantial undertaking? Approximately 1 million H100s are expected to be produced by the end of the year, and an estimated 5 million may be shipped the following year.

Regarding the cost, there’s a valid point about the GPUs. Including the cost of these GPUs in the training expenses might be misleading since they don’t become obsolete after the training process. Those GPUs alone could amount to $20 billion.

It’s worth noting that Chip Manufacturing firm Sustainable Metal Cloud’s (SMC) maximum production capacity for H100s is currently 15,000 units per month, but they have ramped up production to approximately 50,000 units per month.

In terms of electricity costs, they represent a relatively small fraction of the overall compute cost. To put it in perspective, 6,000,000 kWh would amount to roughly $1 million.

Related: AI Industry’s Potential to Rival National Electricity Consumption

Acquiring 500,000 H100s by next year appears to be a challenging task, even with the support of Microsoft. Additionally, questions arise about the cost of inference if the training process is indeed as compute-intensive as suggested.

In the context of Nvidia’s market performance in 2023, it’s noteworthy that their success has reportedly tripled, surpassing $1 trillion. This growth can be attributed largely to the increased adoption of Nvidia chips in AI applications. However, it’s important to consider that U.S. export constraints have limited the sale of high-end AI chips in the Chinese market, which could impact manufacturing and training costs.

Nvidia is generating nearly a thousandfold profit percentage for each H100 GPU accelerator it sells, according to Barron’s senior writer Tae Kim. This means that Nvidia’s street-price of around $25,000 to $30,000 for each HPC accelerator covers the estimated $3,320 cost per chip and peripheral components. The cost analysis is unclear, but it is believed to be a matter of pure manufacturing cost. Nvidia’s R&D costs also need to be considered, as the development of chips like the H100 requires thousands of hours from specialized workers. However, Nvidia’s AI-accelerating products are already sold through until 2024, with the AI accelerator market expected to be worth around $150 billion by 2027.

The company is benefiting from its infrastructure and product stack, but budgets and opportunity cost may limit investments in other areas or limit risks in research and development venues.

Related: The AI Revolution is Just Starting: Will GPT-5 Achieve Artificial General Intelligence?

Disclaimer

In line with the Trust Project guidelines, please note that the information provided on this page is not intended to be and should not be interpreted as legal, tax, investment, financial, or any other form of advice. It is important to only invest what you can afford to lose and to seek independent financial advice if you have any doubts. For further information, we suggest referring to the terms and conditions as well as the help and support pages provided by the issuer or advertiser. MetaversePost is committed to accurate, unbiased reporting, but market conditions are subject to change without notice.

About The Author

Damir is the team leader, product manager, and editor at Metaverse Post, covering topics such as AI/ML, AGI, LLMs, Metaverse, and Web3-related fields. His articles attract a massive audience of over a million users every month. He appears to be an expert with 10 years of experience in SEO and digital marketing. Damir has been mentioned in Mashable, Wired, Cointelegraph, The New Yorker, Inside.com, Entrepreneur, BeInCrypto, and other publications. He travels between the UAE, Turkey, Russia, and the CIS as a digital nomad. Damir earned a bachelor's degree in physics, which he believes has given him the critical thinking skills needed to be successful in the ever-changing landscape of the internet. 

More articles
Damir Yalalov
Damir Yalalov

Damir is the team leader, product manager, and editor at Metaverse Post, covering topics such as AI/ML, AGI, LLMs, Metaverse, and Web3-related fields. His articles attract a massive audience of over a million users every month. He appears to be an expert with 10 years of experience in SEO and digital marketing. Damir has been mentioned in Mashable, Wired, Cointelegraph, The New Yorker, Inside.com, Entrepreneur, BeInCrypto, and other publications. He travels between the UAE, Turkey, Russia, and the CIS as a digital nomad. Damir earned a bachelor's degree in physics, which he believes has given him the critical thinking skills needed to be successful in the ever-changing landscape of the internet. 

Hot Stories
Join Our Newsletter.
Latest News

The DOGE Frenzy: Analysing Dogecoin’s (DOGE) Recent Surge in Value

The cryptocurrency industry is rapidly expanding, and meme coins are preparing for a significant upswing. Dogecoin (DOGE), ...

Know More

The Evolution of AI-Generated Content in the Metaverse

The emergence of generative AI content is one of the most fascinating developments inside the virtual environment ...

Know More
Join Our Innovative Tech Community
Read More
Read more
SatoshiVM Plans Runes Bridge Testnet Launch Next Week, Alpha Mainnet to Follow
Markets News Report Technology
SatoshiVM Plans Runes Bridge Testnet Launch Next Week, Alpha Mainnet to Follow
April 22, 2024
Modular Blockchain Sophon Announces Node Sale And Allocates 20% SOPH Token Supply To Node Operators
Business News Report Technology
Modular Blockchain Sophon Announces Node Sale And Allocates 20% SOPH Token Supply To Node Operators
April 19, 2024
What’s Next for Aleph Zero? Antoni Zolciak Shares Mainnet Updates, Plans, and Key Partnerships at TOKEN2049
Interview Software Technology
What’s Next for Aleph Zero? Antoni Zolciak Shares Mainnet Updates, Plans, and Key Partnerships at TOKEN2049
April 19, 2024
MyShell Launches AI Consumer Layer 2 Network With AltLayer Powered By EigenDA And Optimism
Business Technology
MyShell Launches AI Consumer Layer 2 Network With AltLayer Powered By EigenDA And Optimism
April 19, 2024