I've had this for a while now - I've managed to improve it but it's never really gone away. The biggest issue with mine is that there is grit etc trapped in the "furry" surface of the door window seal. I really should take the seal off and wash it all out - or maybe buy a new seal. What I've done a couple of times is to try to scrape as much as I can out of the seal. I used an expired credit card as the scraper, angled and pressing it hard on the rubber between the door frame and the glass (outside) moving it along the door moved a load of cr@p the first time I did it. At least it's stopped scratching the glass now.
Hope that helps