I don't think this is a conspiracy. Technology can be lost to time just like any information we have. The theory that the universe is slowly moving from order to chaos (entropy) supports the idea.

The other thought I have on this is the people who crafted these blocks probably spent a lot of time crafting each piece that we just wouldn't find acceptable after the industrial revolution. Since every block had to be cut and crafted by hand anyway, they had the time to make them fit perfectly, but buildings and temples took decades to build instead of months. That kind of construction just isn't done anymore - it's not acceptable to anyone to commission a building and wait 40 years for its completion.