Shrinking population will kind of stabilise things, I think. You precisely named "guaranteed growth" here. Most gambling and pyramids are based on the "guaranteed growth", more precisely on "promise of guaranteed growth". Without guaranteed growth it will be much harder to gamble, pyramid, usure and rentier. Meanwhile it could be one of the reasons of all that immigration flood that was created by elites - to continue profiting from promises of guaranteed growth.

Of course, shrinking population is a bad thing, a sign of a degradation of society, but it could play well against elites if immigration flood will be stopped. And as soon as things become better, then people will began to breed again. And there will be a huge challenge to not allow elites to hijack economy again.