But Williams, who won 100 Welsh caps and played in three World Cups, agrees with Warren Gatland and insists Wales can reach the knockout stages.
"At this moment you would definitely say it's the toughest group. They are all tough but this one stands out," said Williams, who retired from rugby after facing the Barbarians in the summer.
"Australia are not in the best place right now and have had a huge amount of injuries. But they've still managed to pick up wins and they will only get better.
"After dropping to ninth in the rankings, England were the one team we didn't want to get but all the groups are tough and we have nothing to fear.
"Without a shadow of doubt we can get out of the pool.
"Of all the teams in the third tier no one would have wanted to draw Wales, they would have wanted to avoid us.
"This team is only going to get better. We've seen what they've done in the past, at the last World Cup and Grand Slam, and by 2015 they will be at their peak.
"It's important we keep some of the old heads like Adam Jones, Gethin Jenkins and Mike Phillips going because all the previous winners have had an experienced, hard edge.
"But by 2015 all the young players will all be between 26 and 28 and at their prime."
Wales will face England three times in the build-up to the World Cup, starting in next year's RBS Six Nations.
The cross-boarder clash with the auld enemy is already the most anticipated fixture of the season but Williams, who faced England 14 times in his illustrious career, reckons it will reach a whole new magnitude.
"If it isn't big enough already, from now on every clash with England will be huge.
"It's already the one fixture that everyone in the country looks forward to. It's always a huge game in the Six Nations, whether at Twickenham or the Millennium Stadium.
"As a player it will be so exciting. I can't even begin to imagine how much hype will be around the game.
"There's a lot of water to go under the bridge but every game between the two countries will be like a warm up to the World Cup.
"Form counts for nothing in those derbies - especially in a World Cup game.
"It's all about who handles the pressure. You know you have been building towards it for four years and have just one shot."