Ah, no, not necessarily.
Being aroused is a physical thing. The body responds to stimulation of such even without the urging of the mind. Just because you have no sexual attraction doesn't mean your body doesn't react to stimuli with arousal.
Logic like yours is often used to defend the notion that men can't be raped, because "penetration has to be involved, and men don't get hard if they aren't wanting to have sex." Which is biologically wrong.