If someone raped someone, and then said on trial, "I was raped and now I think rape is a positive experience", would that be justified? Of course not! No sane person would excuse their actions, regardless of their difficulties.
It's the same thing.
Someone who was sexually abused is not in the right for then condoning sexual assault of others, even if they were psychologically damaged by their experience.
Sure you can have sympathy for his struggle (I personally can't given his atrocious track record of bullying), but he still said things that are inexcusable.
Being a victim of a crime does not give him immunity to commit or even encourage others to commit crime.
If someone like, say Wayne DuMond or Albert DeSalvo, were abused and and had a fucked up upbringing (which they did, ESPECIALLY DeSalvo), that would not give them any leeway in their behavior. And it doesn't put anyone else in the wrong for deciding not to associate with them.
So he made comments people from the left and the right didn't like. As a result, the response was clear: deciding not to invite him to speak, not publishing his book, and not wanting him to write for their website anymore shows that they do not want have anything to do with him.
They all have free will and utilized it by choosing to distance themselves from someone saying atrocious things.
He can backpeddle all he wants, but when not even Stephan Molyneux of all people is willing to defend Milo, there's nothing to stand on there. He received his well-deserved pariah status.
On another note, if Milo was caught sexually abusing his family members I'm sure Mike Huckabee would have came to his defense. After all, ask Josh Duggar on that one.