As a trans person and hence someone who has experienced both male and female toilets, I would like to say that I would never want the toilets to ever not be separated by gender. Women's toilets are just cleaner and nicer, I would hate for them to become even half the bio-hazard that men's toilets were like. (this is not to say that women's toilets are clean, its just they are not terrifyingly disgusting like the men's ones are.)
As for when gender or sex matters, I don't think there is a single good answer. A persons gender is based on their own sense of self, and how they express that gender is important to them, even when it is not important to others. A simplistic example can be seen as a woman who chooses only to wear dresses because to her expressing her femininity like that is important to her, even though now days women can and do wear anything.
Unfortunately when things like being trans is thrown into the mix things quickly become complicated. As stated before the personal expression of ones gender is important to the person. However, feminists tend to dislike trans-women because when trans-women express their gender the feminists feel they are re-enforcing gender stereotypes. The gender equality argument is all good and well, and it sounds good to say things like gender roles and stereotypes are meaningless and shouldn't matter, but reality is far more nuanced. There are people who want to do those gender stereotypical things, like how there are women who want to be stay at home housewives even now in the era of women's liberation. They are not brainwashed or being coerced, they are choosing to act like that because it is what they want and important to them.