Unsprung weight can be very important - depending on the vehicle. A motorcycle probably has 15% unsprung weight, so reducing it makes a huge difference to how well the suspension performs (and therefore the bike handles). On a car, the unsprung weight is going to be more like 2% - so reducing it will be naff all difference.
Maybe if you're actually racing, where every fraction of a second might make all the difference, it'd be worth it - but getting hung up on a few kg here and there is a waste of money. Besides, as soon as you take a single passenger you've undone all the good work anyway!