Dust shields (more commonly known as splash shields) also serve as heat shields (and debris shields). They block radiant heat from the ball joint, tie rod ends, cv joints, and shock etc. Obviously, the brake temps and heat soak when track driving are significantly higher than when you are just daily driving the car. What I am not sure about is whether I am doing damage to the other parts while running without shields in search of cooler rotor temps (I have the RS3 deflectors and I typically pull the shields off in the summer for better track day cooling). Any automotive engineers here that can chime in?
After spending some time researching this through a few different searches a year or so ago (GRM, various car forums like Rennlist, Miata.net, bimmerforums) I think the vast majority of people believe you get a net benefit (cooler brakes during track sessions) and almost no negative effects (thermally degraded grease or advanced deterioration of rubber pieces). I plan to pull them off again in prep of my first track sessions this year.
I asked my mechanic this exact question the other day, and we had a good discussion about it. FWIW, he's a crew chief for two pro teams, has raced for 30 years, and knows his stuff. He said rip off the shields and don't worry about it, there won't be any real world impact. The effects you listed above are all theory.
From a physics/engineering perspective (I am not technically an engineer, but with six more classes I could have been a triple major including a physics and an engineering discipline), on track the dust shield will ensure the rotor retains heat, and as a result it will get quite hot itself. Since heat can't escape the rotor area, it will continue to build and make the heat shield hotter and hotter. Remember that the shield is also connected to the hub area and will transfer heat by contact. By removing the shields, if the rotor runs cooler, there is less heat overall to radiate into the hub and surrounding areas. <- This is theory, and the only thing that can really be done to prove this is experimentation. Luckily, a lot of people have been ripping off their heatshields with no long term impact to other components, so we can use that as a sample. Anecdotal, sure, but it's some evidence, which is better than none/pure theory.
If you're street driving and want to have cleaner wheels, cleaner wheel well, and you don't wash your car regularly, then the dust shield will serve a very good purpose.