I didn't ask whether that would make you a particularly skillful worker. Not looking at the whole picture here. I asked why lack of ability in one area invalidates ability in a completely unrelated area.
Well, you got the answer, didn't you?
Skill doesn't mean much if you can't apply it.
You can certainly apply brute strength to some things. But over the long run, without academic abilities, it won't be able to sustain you.
Thus, if you had to pick one, you should go with math. It gives you drastically more options.