Maggie's Farm

OneGuy: The hidden "problem" is why are there going to be over 1000 of these humongous data centers? Take a look at a 2 gig micro SD card and realize these are not the state of the art in miniaturization and efficiency. The data centers will be using a faster, smaller, more efficient storage system and computers to access and process data.

This is worth pointing out. Reading a book is one thing. Modeling the interconnections within the book is another. Consider The Lord of the Rings: how the characters interact, how their personalities lead them to act in certain ways, how events in one part of the story affect later parts, why does Gandalf refuse to take the Ring himself despite knowing he could use it more effectively than Frodo, and how themes and relationships recur across hundreds of pages.

Now extend that from one book to a library of a million books containing about a terabyte of text data (10^12). Storing that library is easy; a standard hard drive can hold it. What AI does, however, is train on the data.

Suppose we train a model with 100 billion parameters (10^11). Training involves repeatedly processing the text: predicting the next token, computing the error, propagating corrections backward through the network, and updating the parameters. The total computation for a training run can reach roughly: ≈10^24 flops. That level of computation can require thousands of GPUs consuming several megawatts of power continuously for weeks or months. The computational demand grows superlinearly with both dataset and model size. The challenge is not storing humanity’s books; it is performing the immense amount of computation required to model the statistical relationships within and across them. And because larger models often produce more capable systems, companies continue building ever larger data centers. If you don't build them, someone else will.

(By the way, a million books is nothing. Nearly everyone with a computer now stores the equivalent of a million books.)

OneGuy: I have worked with the worlds physically largest computer.

Sure, but the world is changing. A supercomputer can run at 10^18 flops. But modern AI systems have on the order of trillions of parameters 10^12 to 10^13 and are trained on tens of trillions of tokens 10^13 to 10^14. Training cost scales roughly with the product of these two quantities, producing total compute requirements on the order of 10^25 to 10^27 operations. This is why training requires massive distributed supercomputing clusters and dedicated data centers, and why costs continue to rise as both model size and dataset size increase.

That does not discount your justified concern about the growth of large data centers and their impact on resources, including electricity, pollution, global warming, and water. But a mere supercomputer is nowhere near powerful enough for modern AI.

OneGuy: Your entire first paragraph was AI generated gobbledygook.

It was neither AI nor gobbledygook.

OneGuy: you can put the world's fastest most powerful super computer inside a building no larger than a typical city high school and have enough space left over to house the high speed memory large enough to store all the worlds known data.

As explained above, memory is not the issue, but computation. The world’s fastest computer would take years to perform the necessary computations to train a model of 10^13 parameters over a data-set of 10^14 tokens, ~10^27 flops. That’s why they use clusters of thousands of what are essentially custom-designed supercomputers working in tandem. It’s not like a Google search, which grows arithmetically with size, but multiplicative growth.

(See Nvidia)