Over Population
Too much population usually leads to an over supply of laborers. This in turn leads to lower wages and higher unemployment. For 99% of the population this situation is not desirable and often leads to misery. For the employers this situation is good. As they profit from cheap labor.
The beginning of rapid population increases is often the beginning of a golden age for the elites. While for the rest of the population their misery starts.
At this golden age the elites can extract the most amount of wealth. At this stage more and more people can be uplifted into the elites. The more the population reaches it carrying capacity the more it reduces the surplus. As the surplus of production reduces the population growth slows down and elites cannot extract the same amount of surplus anymore.
Because the elites grown in size the average surplus extracted per elite decreases, but because the overall surplus generated by the population also decreases as its reaches their carrying capacity less and less there is for the elites. These two factors greatly decreases the average elite income.
This will result in the matthew effect where some elites will get richer even though their average income falls. The same happens for commoners.