Wales led 17-9 at half-time but the Boks hit back with two tries in quick succession after the break to prevent Wales from recording what would have been just their second-ever victory over South Africa.
And Gatland made it clear at the post-match press conference that the entire Welsh squad were hugely disappointed not to have made history at the Millennium Stadium.
"The players and management are gutted. It was a chance for us and, unfortunately, we came up just short," said Gatland.
"It was a great game of rugby if you were a neutral or weren't part of the red team.
"I'm proud of the performance and proud of the effort but we were a few points off.
"It was a step up from last week. I thought we were very positive and played some great rugby but we were a little bit off the win.
"There were a huge amount of positives but, at the end of the day, it's about winning at this level and we lost. We can't hide behind that fact.
"We scored three tries to two but all of us want to win and we didn't do that."
Wales raced into an early lead after debutant George North scored with just his second touch in international rugby and James Hook rounded off a truly sensational score that has to be one of the best the Millennium Stadium has even seen.
But the Boks worked their way back into the game through the boot of Morne Steyn before tries from Willem Alberts and Victor Matfield inside the space of four minutes put the visitors into a commanding position.
Wales responded brilliantly with another North try immediately after Matfield's close-range effort but Steyn extended the Bok lead to four points with a straight-forward penalty that proved to be the final score of the game on 65 minutes.
Gatland admitted that the period in which the Boks crossed the Welsh line twice was a critical one but he also pointed to the big-game experience that allows the Boks to win games they may look like losing.
"I felt that, if we could get to that 65-minute mark maintaining that lead, the adrenalin kicks in and it's a lot easier to hold out until the 80 minutes. That area just after half time is an area we need to improve on," added Gatland.
"We felt that, physically, they looked tired in the first half but then they played very simply, used their forward runners effectively, kept hold of the ball and put us under pressure.
"That's why we want to play the best sides in the world - so we know how to handle that pressure. They don't panic. Even when they're out of the game, they'll work their way back into it. They know how to win at this level."