Did Agriculture Make People Worse Off?

According to Jared Diamond (in the course of a whine about the agricultural revolution):

If one could choose between being a peasant farmer in Ethiopia or a bushman gatherer in the Kalahari, which do you think would be the better choice?

Is the choice between existence as a peasant farmer and existence as a hunter-gatherer? Or is it between existence as a peasant farmer and non-existence? There was, after all, a large in population.

As far as I can tell, the upper-class population in agricultural societies was about the same as the hunter-gatherer population. For example, in Medieval England, there were 200 men in the upper aristocracy and 1000 knights. If we assume that a typical aristocratic family included a Lord, a Lady and couple of children, the upper class would be 4800 people. According to Jared Diamond, hunter gatherers had a population density of $\frac{1}{10}$ person per square mile, which means England's 50,000 square miles could support 5000 of them, about the same number as in the agricultural upper class. The advantages of being a hunter-gatherer also applied to the upper class. People in the upper class did not spend all day shoveling manure and had a diet with adequate protein.

In other words, the agricultural revolution did not take hunter-gatherers and turn them into peasants but added a peasant population.

Applying the above to a hypothetical society consisting largely of “ems” will be left as an exercise for the reader.