Full-back Alan Awcock ran in six minutes later before centre Tom Riley notched their third just after the hour.
But Aberavon, who have faded after the topping the table in the first half of the season, did not cave in and came back with two tries of their own.
Centre Ashley Beck scored with 15 minutes remaining before winger Richard Carter went over with seven minutes left.
But Newport refused to crumble and pushed for their fourth try, to earn them a crucial bonus point, which duly came in stoppage time thanks to replacement Craig Hill, who was part of Wales's Sevens World Cup winning team.
Llandovery were made to sweat right until the end of their slender 16-15 win against Cross Keys, who came inches away from denying them a crucial victory.
The Drovers got off to the perfect start, with winger Ifan Evans crossing for a try in the first minute, which was converted by outside-half Tomas Marks.
Marks went on to kick three first half penalties, although Cross Keys winger Nathan Trowbridge went over straight from the restart after his first kick.
Trailing 16-5 at half-time, the Keys went out after the break with it all to play for and James Leadbetter slotted over a penalty six minutes into the second period.
Llandovery flanker Lloyd Phillips was sinned binned on 53 minutes, and with the man advantage allowed centre Gareth Maule the space to pull a try back for Cross Keys.
Leadbetter converted but missed a golden opportunity with two minutes remaining, hitting a penalty against the post.
Jimmy Norris grabbed a hat-trick as Bedwas completed a stunning second-half fight back against Pontypridd to win a thriller 41-27.
he Newport Gwent Dragons academy star lit up Bridge Field as Bedwas roared back from 22-5 down at half time with six tries.
Lewis Williams, prop Pat Palmer and skipper Nathan Strong barged over just before half-time for Pontypridd before Stuart Thomas, Matthew Pizey, John Welsh and Stuart Phillips hit back.
Glamorgan Wanderers escaped with narrow victory after Ebbw Vale's Andrew MacLaughlan missed a late conversion.
Second-row Nick Eaves looked to have grabbed a draw for the Steelmen when he touched down late on, shoved over by the rest of the Ebbw Vale pack.
But full-back MacLaughlan, who had taken over the kicking duties from injured fly-half Aaron Bramwell, scuffed a straight-forward conversion to let Wanderers off the hook for a 13th win of the season, the final score being 31-29.
And Pontypool hauled themselves off the foot of the Welsh Premiership table for the first time this season with a hard-fought 26-20 victory over fellow strugglers Bridgend.
Kiwi No8 Grant Webb barged over twice in the second half, including the winner five minutes from time, to complete the win at Pontypool Park.
That proved enough to edge ahead of Bridgend in the table according to the number of wins, after the visitors picked up a losing bonus point.
On Friday night, Swansea gained revenge on local rivals Neath when they beat them 36-8 at St Helens.
The Neath All-Blacks had put the Whites out at SWALEC Cup five days earlier, winning 32-16 semi-final at the city's other ground, the Liberty Stadium.
But Swansea turned the tables on the reigning Premiership champions, realistically ending any faint hopes they have of a title challenge, as they ran in four tries to earn a bonus point from their near neighbours.
Winger Rhys Jones scored two first-half tries for the Whites, who lead 18-8 at half-time despite opposite wing David Evans crossing for Neath.
Damian Karauna and Chris Shelmerdine ran in within the first ten minutes of the second half to put the result beyond doubt.
Karauna kicked a drop goal to seal a 36-8 win, Swansea's first victory over Neath since January 2004.