Dont want to sound like James May here, but heat is exactly whats wrong with this scenario.
On static lubricated matterials, brass on steel generates 5 times more friction than teflon on steel.
That means that the brass speedo drive itself will generate more heat anyway.
To make things worst, you have the big brakes and exhausts right next to it, and you are racing !
The thing you've got in your favour is that brass's melting point is three times higher than Teflon, but then again with the more friction, I wouldnt know what other damages that may occur on the speedo drive.
If the plastic drive brakes and crumbles inside the gearbox, all that will happen to it is be chewed up by the drive joints.
If the brass drive snaps or comes loose and drops in the gearbox.... that would be a different story.
Been told these from an old retired specialist of Alfas, when I mentioned to him that I wanted to do the same but with different matterials.
Why risk it mate ? Find another couple of plastic ones, they will outlive all of us