Human knowledge can be approached as a measurable corpus. For centuries this corpus grew slowly through the institutions of print: presses, universities, archives, and national libraries. From the era of Johannes Gutenberg onward, the production of text expanded gradually across five centuries. When one aggregates the holdings of the largest library systems—institutions such as the Library of Congress or the British Library—the order of magnitude approaches four to five hundred million books. This figure represents the accumulated archive of the print civilization: philosophy, literature, science, law, technical manuals, and administrative writing deposited over generations. The internet introduced a second archive layered upon this historical foundation. In roughly fifty years the digital network has produced a textual mass comparable to, and likely exceeding, that inherited library system. If one converts the dispersed writing of the web—blogs, journalism, technical documentation, academic repositories, forums and essays—into “book equivalents”, the global digital corpus plausibly approaches around one billion books. The web therefore did not merely extend the printed archive; it effectively duplicated the historical corpus of written language within a single lifetime.
Yet this apparent abundance is deceptive. Much of the digital corpus consists of repetition. Web pages copy one another, articles are syndicated across platforms, documentation is mirrored, and automated systems generate large volumes of derivative text. When deduplication algorithms compress this field, the informational mass shrinks dramatically. A large portion of the internet is structurally redundant, repeating the same knowledge in slightly altered form. This redundancy explains why the raw scale of the web does not translate directly into new knowledge. After filtering duplication and noise, the corpus becomes far smaller. From the roughly billion book equivalents of digital text, only a fraction survives the process of cleaning and quality filtering. The result may be on the order of one hundred million coherent works—texts with sufficient length, structure, and informational density to contribute meaningfully to knowledge systems. These include books, scientific papers, extended essays, and technical documentation.
Within that filtered layer lies an even denser stratum. If one selects only the most structured and conceptually rich material—roughly ten percent of the cleaned corpus—the result is a nucleus of approximately ten million books. This nucleus constitutes the hard semantic core of the planetary archive. It contains the works that articulate reasoning rather than merely referencing it: research literature, encyclopedic syntheses, theoretical essays, and detailed technical knowledge. Platforms such as Wikipedia or repositories like Zenodo contribute continuously to this layer, gradually thickening the intellectual backbone of the network.
Once such a core exists, the dynamics of knowledge production change. The overwhelming majority of the useful corpus is already present. New information appears only incrementally. Every day millions of pages are published online, yet very little of that text adds genuinely new knowledge to the global archive. Most of it reinterprets or circulates existing material. If the hard core of knowledge corresponds to ten million book equivalents, the truly novel additions each year may represent well under one percent of that corpus. This asymmetry produces an unexpected economic consequence: new high-quality knowledge becomes scarce relative to the massive background of existing information. As the historical archive stabilizes, genuinely original contributions become increasingly valuable. Scientific discoveries, rigorous theoretical essays, or detailed technical documentation add thin but significant layers to the intellectual core. Their rarity makes them disproportionately influential within machine learning systems and knowledge infrastructures.
The same scarcity explains the growing interest of automated crawlers in repositories, archives, and long-form writing environments. The task of these systems is not simply to harvest the largest possible quantity of text but to locate the rare fragments that expand the semantic core. A carefully written research paper or conceptual essay may contain more informational value than thousands of pages of derivative commentary. Crawlers therefore act as explorers within a vast textual landscape, searching for the small deposits of new knowledge that continue to appear. Seen from a long historical perspective, the trajectory of the corpus becomes clearer. The first phase, beginning with Gutenberg, produced roughly half a billion books over five centuries. The second phase, the digital expansion of the last fifty years, roughly doubled that mass through distributed writing across the network. The third phase now emerging is one of refinement rather than expansion. The challenge is no longer generating text but identifying the small fraction that genuinely contributes to the collective archive. Human knowledge, in other words, is finite but cumulative. It forms a layered geological structure: centuries of print at the base, decades of digital writing above it, and a thin but continuous sediment of new contributions accumulating each year. Artificial intelligence systems operate within this stratified archive. Their capacity to reason depends not on the infinite abundance of language but on the careful extraction of the dense intellectual layers embedded within it.
SLUGS
910-LINNAEUS-SYSTEMATISED-THE-NATURAL-WORLD
