News Report
September 30, 2022

Text-to-3D: Google has developed a neural network that generates 3D models from text descriptions

Google created a neural network capable of creating 3D models from text descriptions. The best part is that the most difficult aspect didn’t even need to be taught. Imagen was used as the foundation for Text-to-3D.

Text-to-3D: Google has developed a neural network that generates 3D models from text descriptions

What should you know about DreamFusion?

Diffusion models trained on billions of image-text pairs have led to recent advances in text-to-image synthesis. Adapting this approach to 3D synthesis will necessitate large-scale datasets of labeled 3D assets as well as efficient denoising 3D data architectures, neither of which are currently available. In this paper, we overcome these restrictions by performing text-to-3D synthesis with a pretrained 2D text-to-image diffusion model. We present a loss based on probability density distillation that allows a 2D diffusion model to be used as a prior for optimizing a parametric picture generator. Using this loss, we use gradient descent to optimize a randomly initialized 3D model (a Neural Radiance Field or NeRF) so that its 2D renderings from random angles have a minimal loss.

The generated 3D model of the specified text can be viewed from any angle, illuminated with variable lighting, and composited into any 3D environment. Its method requires no 3D training data and no changes to the image diffusion model, illustrating the efficacy of using pretrained image diffusion models as prior.

DreamFusion makes relightable 3D models with high-fidelity appearance, depth, and normals based on a caption. Objects are represented as a Neural Radiance Field, with a pretrained text-to-image diffusion prior like Imagen being used.

Examples of Generated 3D From Text

Prompt: photo of a squirrel wearing a medieval suit of armor playing the saxophone
Prompt: photo of a squirrel wearing an elegant ballgown sitting at a pottery wheel shaping a clay bowl
Prompt: highly detailed metal sculpture of a squirrel wearing a purple hoodie riding a motorcycle
Prompt: intricate wooden carving of a squirrel wearing a medieval suit of armor wielding a katana

Putting objects together to make a scene

How does it work?

DreamFusion optimizes a 3D scene based on a caption using the Imagen text-to-image generative model. It suggests Score Distillation Sampling (SDS), which involves optimizing a loss function to produce samples from a diffusion model. As long as we can map back to images differently, SDS enables us to optimize samples in any parameter space, such as a 3D space. To define this differentiable mapping, it employs a 3D scene parameterization that is akin to Neural Radiance Fields or NeRFs. SDS alone creates a passable scene appearance, but DreamFusion enhances geometry with extra regularizers and optimization techniques. The trained NeRFs that are produced are coherent, have excellent normals, surface geometry, and depth, and can be relit using a Lambertian shading model.

Read related articles:

Disclaimer

In line with the Trust Project guidelines, please note that the information provided on this page is not intended to be and should not be interpreted as legal, tax, investment, financial, or any other form of advice. It is important to only invest what you can afford to lose and to seek independent financial advice if you have any doubts. For further information, we suggest referring to the terms and conditions as well as the help and support pages provided by the issuer or advertiser. MetaversePost is committed to accurate, unbiased reporting, but market conditions are subject to change without notice.

About The Author

Damir is the team leader, product manager, and editor at Metaverse Post, covering topics such as AI/ML, AGI, LLMs, Metaverse, and Web3-related fields. His articles attract a massive audience of over a million users every month. He appears to be an expert with 10 years of experience in SEO and digital marketing. Damir has been mentioned in Mashable, Wired, Cointelegraph, The New Yorker, Inside.com, Entrepreneur, BeInCrypto, and other publications. He travels between the UAE, Turkey, Russia, and the CIS as a digital nomad. Damir earned a bachelor's degree in physics, which he believes has given him the critical thinking skills needed to be successful in the ever-changing landscape of the internet.聽

More articles
Damir Yalalov
Damir Yalalov

Damir is the team leader, product manager, and editor at Metaverse Post, covering topics such as AI/ML, AGI, LLMs, Metaverse, and Web3-related fields. His articles attract a massive audience of over a million users every month. He appears to be an expert with 10 years of experience in SEO and digital marketing. Damir has been mentioned in Mashable, Wired, Cointelegraph, The New Yorker, Inside.com, Entrepreneur, BeInCrypto, and other publications. He travels between the UAE, Turkey, Russia, and the CIS as a digital nomad. Damir earned a bachelor's degree in physics, which he believes has given him the critical thinking skills needed to be successful in the ever-changing landscape of the internet.聽

Hot Stories
Join Our Newsletter.
Latest News

Web3 Fundraising Report Q3 2023: A Overview of Evolving Trends and Strategies

In this report, we explore all the trends in the AI, crypto and web3 sectors and identify ...

Know More

Virtual Reality Superior Than Video Conferencing for Remote Work Collaboration: PWC Report

In today’s modern era of remote work, the traditional landscape of digital meetings and workshops is undergoing ...

Know More
Join Our Innovative Tech Community
Read More
Read more
Chainlink (LINK) Announces Stacking Mechanism v0.2 Upgrade to Improve Network Functionality
Markets News Report
Chainlink (LINK) Announces Stacking Mechanism v0.2 Upgrade to Improve Network Functionality
November 30, 2023
Seraph Studio Launches ARPG Web3 Game ‘SERAPH: In The Darkness’ on Arbitrum One
Business News Report
Seraph Studio Launches ARPG Web3 Game ‘SERAPH: In The Darkness’ on Arbitrum One
November 30, 2023
NASA and IBM Partner to Develop Geospatial AI for Weather Prediction
News Report Technology
NASA and IBM Partner to Develop Geospatial AI for Weather Prediction
November 30, 2023
AntPool to Refund Record $3 Million Bitcoin Transaction Fee After User Error
News Report
AntPool to Refund Record $3 Million Bitcoin Transaction Fee After User Error
November 30, 2023
What You
Need to Know

Subscribe To Our Newsletter.
Daily search marketing tidbits for savvy pros.