How long between delivery and testing?
Their website says that you've got 2 days to dispute it if the item is faulty or wrong and 7 days to return if you change your mind. I suspect that both of these statements are illegal as there's a convention that you're allowed a reasonable time to inspect the goods that you've purchased and, as long as you can prove that the battery is wrong it's unfit for purpose.
Time to inspect is something I remember from A level law 20 odd years ago, so it's worth checking to see exactly what the current rules are. Fit for purpose is a completely different matter as it's protected by statute. They MUST cancel the contract and put you in a position where you're financially no worse off than when you entered into the contract. They can't use the T&Cs to say this doesn't apply as the Sale Of Goods Act trumps their contract.
Let them know in writing that you'll take them to small claims if they don't collect the battery and refund you everything that you paid. They can't withold postage costs BTW. If it does go to court then claim for your time, any costs that are involved in contacting them, the extra cost (if there is any) in replacing the item from an alternative supplier (Loss Of Bargain is the legal term here) and interest between when the money left your bank to the court date at (last time I looked) 8%! Add court costs and they could well be looking at a couple of hundred quid on top of the cost of the battery.
To give you an idea, Lidl refused to deal with a faulty DVD player I bought about 10 years ago. I was offered a refund but not the additional costs I'd incurred while trying to get it fixed. In the end I took them to small claims and was awarded £157 costs ON TOP of the £60 refund for the player. £40 of that was the difference between the player I bought and the next cheapest one I could find of the same specification.