As some have suggested, I think one big mistake is looking at this as though evolution has been occurring at a steady rate. Nothing could be further from the truth. One significant event that occurred was the Cambrian Explosion, which began around 580 million years ago. At first it was thought that only simple, single celled creatures were present prior to this occurrence, but evidence now indicates that there were some more complex animals present prior to 580 million years. However, the diversity of life beginning at about that time was so markedly extreme, that even Darwin thought it was the single most compelling argument against his theory of evolution to the point that he devoted a chapter to it in his book
The Origin of Species. The causes behind the explosion have not been completely discovered, but a couple of causes thought to have caused it are a significant increase in oxygen levels in the atmosphere and an increase of calcium in seawater. Regardless, a significant and rapid diversity of life occurred. You can read more about it
here.
We must also take into account that there have been several mass extinction events in Earth's history that severely curtailed life, one as recently as 65 million years ago. There have been bottlenecks caused by these events that on occasion nearly wiped out all life on earth. Yet life persisted and flourished.
To me, saying there simply hasn't been enough time for mankind to evolve from a specific early ancestor is to totally ignore what we do know about Earth's history. This is nothing more than misdirection and grandstanding by creationists who so desperately want (and need) their little book of goofy explanations to be true.