I wrestled with this problem for ages, everything pointed to the Servo and I changed it hoping to solve the problem, it didn't
In truth it could be any of the following;
Servo, leaking vacuum.
Faulty (or non existant in my case) nonreturn valve in the Vac pipe to the servo
Collapsed rubber pipe which links the two halves of the servo, or the two seperate servo's in a dual circuit car.
Collapsing flexi hoses to front calipers and rear axle
Sticking calipers
BUT and I reckon most likely, faulty master cylinder, yes there is a spring which returns the piston and the piston seals can make it all stick, it needs to be adjusted so that there is some movement of the pedal before it activates the cylinder, if not as the cylinder moves back it takes up all the slack and the pedal will always be dragging the brakes.
If you try adjusting it and it doesn't solve the problem, you might need to look to replacing the cylinder.
How old is the current M/C?, if it's an old one it could be life expired, but don't rule it out if it's new, because there is a faulty batch of pattern cylinders which are giving many people a lot of headaches. I had exactly the same problem and it eventually came down to one of those faulty cylinders, very frustrating.
Good luck, and search about a bit on the forum for 105 brake problems, it's common, too common!!