So some good news. I wrapped the circumference of the stock plastic spacer with 22 gauge galvanized steel and epoxied it in place. After this dried I added a layer of Raamat on the surface of the steel.
Installed M5 rivnuts into the door (who pop-rivets speakers in place?) and reinstalled. One note - if you do this you have to grind down the plastic lip on the back side of the spacer around each mounting hole (you'll see what I mean when you look) and then add a new layer of something compressible to seal the speaker to the door since the rivnuts have a ring around them that sits above the metal surface. Was going to use speed clips but I hate those things and then I remembered I had the rivnut kit, that is why the paint is all scratched up.
Stock (unmodified) driver response at 1 ft:
Stiffened stock driver:
On a crap phone screen shot it is subtle, but the unmodified driver has a peak at 1200 Hz where as the modified one is flatter from about 600-1200 Hz. It definitely sounds fuller by ear. Most importantly I was able to get this response from it by changing the head unit low/mid/high settings in less radical ways than needed for the fronts (there is no door panel at this point so the tweeter isn't hooked up, that is why there is a huge drop at 10k).
So this looks like the best way forward without going full replacement. Going to pull the other rear tomorrow and do the same before moving to the front.
Also, if you haven't worked on the rear doors it sucks. Everything is tight because they just don't open very far.