Swansea had put together an excellent first half which they led 10-5. It could have been an even more impressive lead had full-back Nicky Thomas nailed two long range penalty attempts.
But it took only six minutes for the All Whites to take the lead with a try for busy scrum-half Justin James after some good interplay between him and Aled Jenkins and Richard Williams down the right. Thomas converted to make it 7-0.
Llandovery replied on 19 minutes with a try for captain and lock Phil Day, who crashed over from a line-out drive. Outside-half Ian Brooks failed with the conversion and a further two penalty attempts.
Swansea extended their lead with a Thomas penalty to go into a deserved interval lead although Llandovery had looked dangerous on the break, testing the Whites to the full.
The Drovers hit back straight after the break with a penalty for wing Lee Evans but that was immediately cancelled out by Nicky Thomas.
But there were more problems for Swansea as Llandovery counter attacked and left the home defence in tatters, wing Evans running in under the posts and converting his own score.
That saw the Drovers go into the lead for the first time at 15-13 and they did not look back from there.
Evans kicked a second penalty before a converted try for replacement hooker Luke Lewis extended the visitors' lead to 12 points.
Full credit to Swansea. Their heads did not go down and Harvey's try four minutes from time ensured they earned a bonus point.