We have it pretty good these days. From Gene Expression:
Geneticists have long known that the ancestors of modern humans numbered as few as 10,000 at some time in the last 100,000 years. The critically low number suggested that some catastrophe, like disease or climate change induced by a volcano, had brought humans close to the brink of extinction.
If the new estimate is correct, however, human population size has been small and fairly constant throughout most of the last million years, ruling out the need to look for a catastrophe.
Assuming an average census size on the order of 50,000, it seems as if our species stumbled onto a rather "risky" strategy of avoiding extinction. From what I recall conservation biologists start to worry about random stochastic events (e.g., a virulent disease) driving a species to extinction once its census size reaches 1,000. I suppose the fact that we were spread out over multiple continents would have mitigated the risk, but still.... It also brings me back to my post from yesterday, it seems that for most of human history we are a miserable species on the margins of extinction. For the past 10,000 years we were a miserable species. And now a substantial proportion of us are no long miserable (it seems life is actually much improved from pre-modern Malthusianism outside of Africa and South Asia). If only Leibniz could have seen it!