I used to wear real ones because I needed them for a school project and they were effing expensive at the surplus store. Granted, I never really learned the "proper" way to wear them (orientation, location on clothes, etc.), which probably made it clear that I didn't really know what they were for. My vet relatives understood that I didn't mean any disrespect by it, and I only ever had one problem with a hothead marine who seemed to think the sole reason for my existence was to piss him off.
I don't see what the big deal is.