Culture Secretary Lucy Frazer has blamed heavy Tory losses in the local elections on the pandemic and the Ukraine war.
She also noted the Conservative Party had been in power for “a long time”.
The Cabinet minister conceded that voters were “frustrated and angry”, but claimed Rishi Sunak was starting to regain the public’s trust.
The Prime Minister has been blamed by some Tories after the party shed 960 councillors in Thursday’s poll.
“We’ve just had a pandemic which has disrupted many people’s lives and has had consequences for the economy, as has the war in Ukraine, which is going on.”
Pressed on whether she understood the scale of her party’s losses, she said: “Totally, it’s really important that we listen to people. I know people are frustrated and angry.”
She said she recognised the results were “not good”, adding: “I totally understand that we need to do better and I think we are going to deliver.”
The party was rebuilding trust as Mr Sunak was “starting to deliver in a quiet way” his five priorities including curbing inflation, Ms Frazer argued.
“I totally recognise we’ve had a really difficult few years.
Pressed on whether the Tories needed to offer more than Mr Sunak’s promises from January, she told the BBC’s Laura Kuenssberg On Sunday programme: “We absolutely need to reflect. I think Rishi’s only been the Prime Minister for six months.”
Kuenssberg said: “It’s not five minutes, it’s six months.”
“It’s not five minutes but these are huge challenges,” Ms Frazer replied.
Some Tories blamed the electoral hammering on the lack of tax cuts.
Ms Frazer insisted the Government was doing those things, saying: “The biggest tax cut we can make is a cut in inflation”.
Tory MP Anthony Browne denied that Mr Sunak was “under threat in any way” after the dire results.
“I think a lot of it was a protest vote,” the South Cambridgeshire MP told the BBC’s Westminster Hour, noting the pandemic and “all the shenanigans of last year, which, for people, this was their first chance to actually vote against that”.
He also claimed that “Labour has not got the momentum it needs to win a general election”.
Meanwhile, Labour was celebrating “pretty encouraging” local election results, which saw Sir Keir Starmer’s party gain 635 seats and take control of another 22 local authorities.
Shadow health secretary Wes Streeting told Ridge: “I think for the next general election Labour feels confident but not complacent.”