In the world of data centres, vast collections of text, known within artificial intelligence circles as corpora, are crucial in teaching machines to understand and generate human language. These corpora underpin AI chatbots and virtual assistants that have fascinated the public in recent years.
The scale of modern datasets used for training AI is immense. For instance, when OpenAI introduced its GPT-3 model in 2020, the sheer magnitude of its training data, which comprised approximately 500 billion tokens, was a significant milestone. This number has since been outstripped; Google's PaLM model, unveiled in 2022, utilised a dataset of over 780 billion tokens. Researchers have observed that increasing the size of these datasets often correlates with enhanced model performance, perpetuating the drive for ever-larger corpora.
However, it's not just the quantity of data that matters. The diversity of these datasets is equally pivotal. For instance, Common Crawl, a non-profit that archives web pages, provides snapshots of the internet that many AI researchers use. These archives, updated monthly, encapsulate around 3 billion web pages, offering an extensive reservoir of human communication in various forms.
The task of building such datasets is monumental. Algorithms designed for web scraping sift through billions of web pages, eliminating spam and low-quality content. Following this, teams of linguists and data scientists undertake the laborious process of cleaning and pre-processing the text, transforming raw data into a format suitable for AI models.
Certain specialised domains necessitate even greater curation. The MIMIC-III database, for instance, is tailored for medical AI research and includes de-identified health data from over 40,000 critical care patients. This dataset maintains patient privacy while preserving vital medical information. In the legal sphere, the Case Law Access Project, developed by Harvard Law School, provides a comprehensive corpus of over 6.5 million cases, covering all official, book-published United States case law from 1658 to 2018. This resource is invaluable for AI systems aiding legal research and analysis.
The creation of these vast datasets is fraught with challenges. Web scraping can inadvertently capture copyrighted material, raising legal issues. Furthermore, the internet may not always accurately reflect human knowledge or values, presenting additional hurdles.
In response, research teams are developing more advanced filtering techniques. Some are even turning to synthetic datasets, generated by other AI models, to fill in gaps or create more balanced representations of specific topics.
Preparing data for AI training involves several key steps:
- Tokenisation breaks text into individual words or subwords.
- Normalisation standardises text by converting it to lowercase, removing extra spaces, and handling special characters.
- Sentence segmentation identifies sentence boundaries, essential for many natural language processing tasks.
- Part-of-speech tagging labels words with their grammatical categories.
- Named entity recognition identifies and categorises names of people, organisations, and locations.
- Dependency parsing analyses the grammatical structure of sentences.
While text-based corpora are foundational, researchers aim to expand beyond text to include images, audio, and sensor data, leading to more comprehensive AI systems. The Pile, a dataset created by EleutherAI, exemplifies this approach by incorporating English text from 22 diverse sources such as academic papers, coding repositories, and web content. This diversity enables AI models to assimilate a broader spectrum of human knowledge and communication styles.
For multilingual models, datasets like mC4, used to train Google’s mT5 model, and OSCAR (Open Super-large Crawled Almanac) provide texts in over 100 languages. These resources are pivotal for training AI systems capable of understanding and generating text in multiple languages.
The future of AI corpora may involve real-time updates, allowing models to assimilate the latest information dynamically, without necessitating full retraining. Additionally, researchers are exploring ways to enhance training efficiency, potentially diminishing the required data volume while maintaining or improving model performance.
As AI progresses, the development and refinement of these text collections will remain a significant challenge. The size, diversity, and quality of today’s corpora will shape the capabilities of tomorrow’s AI systems, influencing how these digital entities comprehend and interact with the world.
Source: Noah Wire Services