Juande Ramos was linked with the White Hart Lane post over the weekend after he was spotted in the same hotel as a Spurs official, although the Premier League club have denied holding any negotiations to find a successor to Jol.
The Sevilla coach also brushed off the speculation, but Spurs' current boss is upset by the recent criticism levelled at him.
Tottenham responded to losing their opening two games of the season by thrashing Derby County 4-0 on Saturday to seemingly ease the burden on Jol's shoulders.
But the Dutchman is aware of the rumours surrounding his position and concedes he 'can't take any criticism'.
"I thought there were whispers last year and there are whispers at any club all the time," said Jol in The Sun.
"I don't know if I'm being undermined, I don't think so, but you never know. I would not have made a big bet on my situation.
"The only thing for me is that it was a bit early, you say early doors in England.
"As a foreigner you have to prove yourself because the top five clubs have foreign managers, so that is not easy to take on the chin, you know, so I have to do better.
"People talked us up before the start of the season and that's why it took only four days to bring us down again.
"But I'm probably the same as six or seven other people out of 10. I can't f****** take any criticism.
"The board never told me personally that we have to break into the top four."