I’m inclined to believe they meant what they said, when they all called this group/race “giants” - but admittedly the notion that their large stature was simply “in the eye of the beholder” or somehow otherwise a metaphorical statement can’t be ruled out.

My best understanding is that, in the past, society/civilization has risen and fallen. The most coherent explanation for what appears to be a “common cultural ancestor” is exactly that - a group who was forced to spread out in diaspora (ecological disaster/warfare induced migration). Then, because of this disaster and diaspora, these different locations were unable to contact eachother, and thus what started as a common cultural influence gradually changed as different local factors impacted the development of these new civilizations [made up of small numbers of diaspora members and relatively far larger numbers of “primitive” (aka “suited to life after a disaster) locals.]