I always figured it was something his followers made up after the fact to make him look holier.
Within the story itself, I prefer the interpretation that Joseph loved Mary enough that he didn't care.
I do like that approach. The question is, who is the father after all? I see four possibilities:
1. Mary was raped. In terms of what makes the best story, I like this one the best. Mary remains innocent, and Joseph gets points for not rejecting her.
2. Mary cheated. Works better for Mary, but then she's a sinner as well. Although Joseph then gets mongo points for forgiving Mary.
3. Joseph himself - they did it before the wedding, and used the virgin birth story to cover it up.
4. The virgin birth story was made up after Jesus died. Jesus was conceived within wedlock after all.