What's wrong with being a casual gamer?
Genuinely nothing. Problem is, the term carries with it certain connotations of, well... it'd be difficult for me to explain, but "casual" is pretty much the Dinkelburg that people cry when a game doesn't match their tastes 100%.
They complain that the video game industry is being ruined because it's "catering to casual gamers" (as Ravynous just mocked) and absolutely hate the idea of a game including an optional mode that might make it more available to the common person (See: People who were raging at a rumor that difficulty levels might be added to Dark Souls 2)
As my signature puts it, "they want to be an exclusive club while simultaneously demanding public acceptance." They want all of the benefits of being part of the mainstream without actually being part of the mainstream.
I generally hate these kinds of gamers more than I hate twihards, people obsessed with Justin Bieber, and so forth.