And although things got better after the break, in truth it was a deserved win for Jake White's side.
Tevita Kuridrani crossed for the only try of the game, the giant centre bursting through a poor tackle from Christian Wade to open the scoring after a couple of minutes.
The conversion went awry but after Jesse Mogg and Stuart Hogg had exchanged penalties, the Brumbies led 8-3 at the break. It could, and perhaps should, have been more. The Lions really struggled to assert any authority, especially up front, the result being that their outside backs were barely given the chance to fire.
After a similarly poor start to the second half, Warren Gatland's introduction of a series of replacements turned the tide for the Lions. By that stage, two further Mogg penalties had put the Brumbies 11 points clear, leaving the Lions chasing the game.
A second penalty from Hogg reduced the deficit and with a more solid set piece established, two kicks from replacement Owen Farrell meant the Lions were behind by two as the final whistle approached.
That was as close as they got though as the Brumbies held on for a famous win.
Williams went close to scoring in the opening minutes but the Lions produced arguably their worst performance of the tour in the opening half. It wasn't long before Kuridrani opened the scoring, the centre taking of advantage of a defensive mix up between Hogg and Wade to barge over for the first points of the day.
The Brumbies were playing some solid rugby but the Lions weren't helping themselves, Rory Best particularly wayward at the line-out. Hogg then struck a post with a simple penalty effort to sum up the disappointing start.
Mogg's penalty then pushed the Brumbies into an 8-0 lead before the Lions finally got going through the boot of Hogg.
The likes of Billy Twelvetrees, Brad Barritt and Williams had seen very little ball in the first half and that continued after the break.
And despite having four debutants in their side, the Brumbies were enjoying themselves, Mogg adding gloss to the score line from the tee.
With just under an hour gone and the result slipping away, Gatland made wholesale changes to his pack. Welshmen Richard Hibbard and Dan Lydiate as well as Geoff Parling and Owen Farrell were introduced with Hibbard and Parling in particular helping to steady the wobbly line-out.
Lydiate joined Toby Faletau and Justin Tipuric in an all Welsh back row and the replacements helped to give the Lions the grunt they had been missing up until that point.
Two Farrell penalties then saw Gatland's men look like they might nick it.
But the men in red failed to add any more points and with that, their 100 per cent record went too.
Tries: Kuridrani; Penalties: Mogg (3)
Penalties: Hogg (2), Farrell (2)