• Jacob Rees Mogg - the minister in charge of efficiency - told the BBC that they were trying to get back to 2016 levels of staffing before Brexit and the pandemic.
• Major job reductions through natural wastage - waiting for people to resign or retire rather than enforcing redundancies - would take many years, Mr Penman warned.
Boris Johnson has demanded his cabinet comes up with a plan which could cut up to 91,000 civil service jobs to free up cash to tackle the cost of living.
It is understood the PM wants to see civil service staffing levels drop to 2016 levels.
Government sources suggested the plan is about cutting people not needed and it could save £3.6bn a year.
A civil service union warned an "ill thought-out" plan could affect services.
Jacob Rees Mogg - the minister in charge of efficiency - told the BBC that they were trying to get back to 2016 levels of staffing before Brexit and the pandemic.
Mr Johnson told the Daily Mail: "We have got to cut the cost of government to reduce the cost of living."
At an away day in Stoke-on-Trent on Thursday, the prime minister gave ministers a month to produce plans to drastically shrink the civil service, as the government comes under increasing pressure over the rising cost of living.
There were 384,000 civil servants employed in 2016 - the lowest number since World War Two - but as the UK prepared to leave the EU numbers steadily rose until they reached 475,000 at the end of last year.
While large-scale job cuts are likely to take time, a government source said "instant savings" could be achieved by imposing a recruitment freeze in Whitehall and abolishing any vacancies which had not been signed off by a minister.
Examples include the passport office and 70 staff said to be still working on climate change conference COP26.
A source said the cuts were "not ideological" but about "good housekeeping".
The prime minister suggested to the Daily Mail that the savings could be used for tax cuts, saying: "Every pound the government pre-empts from the taxpayer is money they can spend on their own priorities, on their own lives."
But civil service union the FDA said the expansion of Whitehall since 2016 had been necessary to tackle "two unprecedented events", Brexit and the Covid-19 pandemic.
FDA general secretary Dave Penman said the government could decide to cut the civil service back to 2016 levels, but he said it would also have to choose "what the reduced civil service will no longer have the capacity to do" which may affect services such as passports, border control or health.
"Without an accompanying strategy, these cuts appear more like a continuation of the government's civil service culture wars - or even worse, ill-thought out, rushed job slashes that won't lead to a more cost-effective government," he said.
Major job reductions through natural wastage - waiting for people to resign or retire rather than enforcing redundancies - would take many years, Mr Penman warned.
Labour accused the government of "pointless rhetoric and lack of action" instead of implementing an emergency budget to offer more support to people struggling with bills.
A government spokesperson said that the prime minister and ministers were "clear that the civil service does an outstanding job delivering for the public and driving progress on the government's priorities".
But they added when people are facing rising costs, they "rightly expect their government to lead by example and run as efficiently as possible".