Of our 23,000 protein-coding gene base pairs, we get 11,500 from each of our parents, 5,750 from our grandparents, 2,875 from our great-grandparents, and so on.
That repeated division means that by the fifteenth generation - which is only a few centuries ago - your average ancestor (assuming zero inbreeding) is contributing, on average, less than a single gene to your current genome. Go back a thousand years to the 30th generation, and the average genetic contribution is effectively zero.
While it isn't really accurate to say that we're all inbred, at least not in a genetic sense,
it might actually be fair to say that we're all the descendants of inbred people. Numerous theories have been put forward about a huge decrease in the human population tens of thousands of years ago - one particularly extreme version suggests the human population in sub-Saharan Africa remained under 2,000 for as much as 100,000 years,
while more moderate hypotheses suggest a population bottleneck of about 15,000 that occurred about 70,000 years ago. Either population bottleneck would most likely necessitate fairly extensive inbreeding, and that's backed up today by the relatively low level of genetic variation within humans.