President Joe Biden is issuing an executive order pardoning all Americans who have been federally convicted of "simple possession of marijuana".
The move will only affect a relatively small number of marijuana convictions since most occur at the state level.
Mr Biden's announcement stops short of a full decriminalisation of cannabis, which he promised as a presidential candidate.
Recent polls show that a majority of Americans believe weed should be legal.
"Sending people to prison for possessing marijuana has upended too many lives and incarcerated people for conduct that many states no longer prohibit," Mr Biden said in a statement on Thursday.
"Criminal records for marijuana possession have also imposed needless barriers to employment, housing and educational opportunities," he continued, adding that racial minorities were statistically far more likely to be jailed for cannabis.
As a White House candidate, Mr Biden was criticised for writing a 1994 crime bill that stiffened penalties for drug crimes and led to more incarceration of racial minorities.
In a briefing on Thursday, a White House official told reporters there are "no individuals currently in federal prison solely for simple possession of marijuana".
The Democratic president said he would call upon all state governors to issue their own marijuana pardons.
He is also directing the Department of Justice and the Department of Health to review how cannabis is classified under federal law.
"We classify marijuana at the same level as heroin - and more serious than fentanyl," said Mr Biden. "It makes no sense."
Recreational marijuana is already legal in 19 states and Washington DC. Medical use is legal in 37 states and three US territories.
However, the drug remains illegal at the federal level, even in states where it can be legally bought and used by millions of adults.
The federal ban also prevents the legal industry from working with major US banks, which are federally protected. The prohibition bars marijuana shops, known as dispensaries, from common financial services and forces many to operate as cash businesses.
The US jails a higher percentage of its population than any other nation, but relatively few people are in prison for simple possession of marijuana.
"More than 99% of federal drug offenders are sentenced for trafficking," according to the US Bureau of Justice Statistics.
US officials estimate there are "6,500 people with prior federal convictions for simple possession of marijuana" that Mr Biden will pardon.
Cannabis company shares jumped on the the stock market by around 20% with news of Mr Biden's pardons.
