There's something strange for sure.
So, in the Fistfight, you are prompted to press W,A,S or D by the yellow letters on the screen - Top, Left, Bottom, Right - respectively. Assuming you press the correct letter in time, you will connect a blow to your opponent. It should only take 4-5 correct blows to down any opponent.
No potions needed.
Many find it easier to sit back a bit from the monitor with your eyes lazily on Geralt. You will see the yellow letters flash on the screen as described above, sometimes in combination.
Are you doing that?
I'm getting the feeling we may have a bit of a PEBCAK error here.