A Building the Size of a Soccer Field Could Hold Humanity's Accumulated Knowledge
The amount of information is immense, yet surprisingly small.
As a young scientist I remember my boss musing that, in R&D, our only lasting output is technical reports. If we extend that to all of humanity, what are we really leaving to the ages? It is the written word, artwork, photos, audio, and video. Most everything else will eventually decay, even disappear.
One of the things that makes humans special is our ability to pass down information from generation to generation. By teaching of course, but also by leaving a legacy of recorded work.
How much information is there?
For centuries, idea storage was in the form of physical media, tablets, scrolls, books, vinyl records, films, DVDs, you get the idea. But today nearly everything can be digitized. This is an incredible miniaturizer. The size of the human record can now be measured in bytes.
But how much is there? It depends what we want to keep, and at what resolution. But any way you look at it the amount of digital storage required is huge.
In very rough numbers, for publicly available published material of all types, we are looking at multiple exabytes (one exabyte is a million terabytes). Written works, photos, and artwork can each be estimated at about 1 exabyte. Video is larger, perhaps five times higher, and all these categories are growing.
The table below gives the estimated requirement for a single copy of the data. These numbers are very rough but the order of magnitude should be about right. Also for a safe permanent record several copies at least should be saved.
| Category | Estimated storage for one copy |
|---|---|
| Written material | ~1 EB |
| Audio | ~50 PB |
| Photography | ~1 EB |
| Video | ~5 EB |
| Scanned art | ~1 EB |
| Total | ~8 EB |
Note: Written material includes books, newspapers, periodicals, catalogs, maps, blueprints, ephemera, government reports at about 1 MB per page. Audio includes music, radio recordings, podcasts, audiobooks, speeches and similar recordings. Photography excludes private albums and phone camera rolls. Video excludes home video, surveillance video, and user uploaded social video. Scanned art includes museums, galleries, murals, monuments and sculptures with 2D or 3D scans as needed.
For scale: 1 TB = 1000 GB, 1 PB = 1000 TB, and 1 EB = 1000 PB, or 1 million TB
The numbers are huge. I don’t believe I have ever spoken of exabytes before. However, to me the shocking thing is not how large those numbers are, but how small they are. The quantity of storage needed is within the range of human engineering capability. The largest hyperscale storage facilities currently in existence are in this range. A serious archive with backups in several different locations would require perhaps 10 giant storage facilities or the equivalent in smaller facilities.
Another interesting comparison is to AI data centers, which can be several times larger and much more energy intensive than the storage archive envisioned here.
A well-run project, might cost on the order of $20 billion to build and then perhaps $2 billion a year to maintain. This is well within the capability of a large nation’s budget.
Historical trends
It is informative to look at how storage requirements increase over time. The graph below shows storage needs growing approximately exponentially.
Rough order of magnitude estimate for one digital preservation copy. Note: vertical axis is logarithmic, each step is ten times the one below it. Video includes films, television, news, training, and institutional productions, but excludes home videos, surveillance video, and user uploaded social video.
But storage capacity technology is also growing exponentially and the price per byte dropping. This suggests storage needs in terms of investment required will be reduced over time. A few decades ago a project like this was impossible. Now it is feasible, and in the future becomes even more affordable.
Why there is some urgency
But this does not mean we should wait, because information is constantly being lost unless it is proactively saved. At the same time the record is increasingly being mixed with AI generated material. To separate legitimate from AI-enhanced, or completely fabricated material requires curation and expert archival judgement. A quick search on YouTube reveals fanciful videos of made up historical situations. Whether created for fun, or for more nefarious reasons it will become harder over time to distinguish real from the made up.
Would anyone use it?
A short time ago I was looking up something in one of my hometown newspapers from a few decades ago. Newspapers.com has scans of many historical newspapers online for a nominal fee. This is the modern equivalent of the space-saving microfilm files that libraries used to store copies of old newspapers and magazines. On the website there is a counter showing how many people have accessed each page. In my case, I was the first person to access the page. I snooped around other pages, in nearly all cases, I was the first. It is striking to see the quantity of well-written articles just sitting there unread.
In fact, unless there is some compelling reason there is a good chance no human will ever look at these pages again. And the same is likely true for the majority of archived material.
Most items are not important, but a few could be very important. The problem is many times we do not know in advance which items those will be.
This is completely to be expected. We spend most of our time reading and consuming current information, only rarely going back to look at older material. It is a natural consequence of our ability to save and store vast amounts of information.
What are we saving now?
Much of the historical output is already being saved in a variety of places: libraries, museums, online archives, cloud storage, private archives and countless other nooks and crannies of the world. Organizations like the Library of Congress, the Internet Archive and others are doing important work and without those efforts far more would already be lost.
Different organizations have different goals, budgets, restrictions and problems. They may be incomplete, or not even intended for long-term preservation or public use. Sites can shut down, formats become obsolete, and there are many ways the information can become lost again. Humanity has learned this painful lesson in the past beginning with perhaps the Library of Alexandria. In large part our current process is leaving a lot up to chance.
Indexing and other complications
To be useful an archive should not simply be a data dump repository. It should be actively curated by trained archivists, essentially the ultimate archivist challenge. But this is already a studied field, so curating the entire output of world history is more a matter of scale than inventing something new.
There are many issues to be dealt with beyond storing raw data, like copyright (perhaps make submissions private for 50 years). Also, privacy, national secrets, and so on. But is it feasible? Absolutely. Does society want to do this? Only the future will know.
In a way, this is the modern-day equivalent of humans retelling legends around a campfire. Passing information from one generation to the next. But if you accept the premise, that fundamentally this is a major part of what humans produce, then taking a more direct approach to saving it might be worth considering. Our descendants may thank us someday.
Related ideas:
The Math Is Right, but Something Doesn't Add Up
The World's Largest R&D Lab