Defeats to Sunderland and Everton saw Spurs drop to the bottom of the Premier League, a sorry start for a team tipped to challenge for a Champions League place after spending more than £40million in the summer.
Boss Martin Jol is dealing with an injury crisis in defence and his expensive strikeforce is taking time to gel, but Gardner is refusing to contemplate defeat to Derby at White Hart Lane this weekend.
"We just need a result to go our way," the centre-back said. "It may be early but Saturday is a must-win game for us to lift our morale. The players are disappointed, the fans are disappointed."
Jol grabbed the microphone at the end of last season and addressed the fans after finishing fifth, but the supporters jeered off the team against Everton after their first home match of the season on Tuesday night.
"I can understand the fans' frustrations," said Gardner. "Expectations are high and they are looking for us to do big things.
"We finished fifth last season and we have improved the team since then so we were expecting to make a strong start but we haven't.
"I just hope the fans can be patient because we have the quality to turn this around. I train with the players every day and know how good they are and the quality of players we have bought in the summer."
Bookmakers cut the odds on Jol getting sacked following the 3-1 defeat to Everton, a clash that exposed a lack of depth in Tottenham's defence.
Ledley King spent the night as a pundit as he recovers from his knee operation, Michael Dawson had an ankle sprain and Gareth Bale has been struggling with knee and thigh problems.
It got worse for Jol when £8million signing Younes Kaboul limped off with a hamstring problem in the first half against Everton, with Paul Stalteri, Ricardo Rocha and Gardner struggling in a makeshift defence.
It has increased the spotlight on West Brom's Curtis Davies, who wants a move to the Premier League and is understood to prefer Spurs as a destination.
"Derby will be a difficult game, we have to come up with a decent team again," said Jol.
Gardner took the blame for Joleon Lescott's opener during the Everton defeat, although he then scored at the other end.
Rather than being a springboard for victory, Leon Osman and Alan Stubbs earned the points for David Moyes' men.
Gardner is confident his team-mates can overcome their poor start - which they eventually did last season - and live up to their billing of top-four contenders.
"The level of expectations hasn't got to the players," added the 26-year-old. "Sunderland was not a good game to watch, it was tight and they scored with the last kick of the game which was unfortunate for us.
"I don't think it's because of players shirking away from the expectation levels, we are all aware what's on our shoulders and we just have to go out there and do it.
"If we can beat Derby it will make everyone happy again and give us something to build on."
Dawson has not been totally ruled out of the Derby game but King is out until the end of next month. Aaron Lennon has a knee injury and is not expected to return to training until the end of the month.
Meanwhile, Spurs have submitted new plans for training facilities at Enfield. An appeal for the first application is expected to be heard in November.