The Technology Innovation Institute (TII), supported by the UAE government, has introduced Falcon 3, a series of open-source small language models (SLMs) designed for effective operation on lightweight, single GPU-based infrastructures. This launch marks a significant advancement in the accessibility of AI capabilities for developers, researchers, and businesses.
Falcon 3 is available in four sizes: 1 billion (1B), 3 billion (3B), 7 billion (7B), and 10 billion (10B) parameters, featuring both base and instruct variants. The models have already shown promising performance, either outperforming or closely matching other popular open-source models, such as Meta’s Llama and the leading model Qwen-2.5, according to the Hugging Face leaderboard. The growing demand for SLMs, which are less resource-intensive compared to larger language models (LLMs), is driven by their efficiency and affordability, making them suitable for usage in sectors such as customer service, healthcare, mobile applications, and Internet of Things (IoT) devices.
Industry forecasts predict that the market for SLMs will expand significantly, with an annual growth rate (CAGR) of nearly 18% anticipated over the next five years, as reported by Valuates Reports.
The Falcon 3 family is trained on an extensive dataset of 14 trillion tokens, which is more than double that of its predecessor, Falcon 2. The models utilize a decoder-only architecture featuring grouped query attention, which optimizes memory usage and enhances operational speeds for various text-based tasks. These models support four primary languages – English, French, Spanish, and Portuguese – and are equipped with a 32,000 token context window, thereby enabling the processing of lengthy inputs such as complex documents.
TII has stated, “Falcon 3 is versatile, designed for both general-purpose and specialized tasks, providing immense flexibility to users. Its base model is perfect for generative applications, while the instruct variant excels in conversational tasks like customer service or virtual assistants.” This adaptability underscores the models’ suitability for a range of specific use cases.
Performance-wise, Falcon 3's 7B and 10B models have achieved state-of-the-art results in categories such as reasoning, language understanding, instruction following, and complex tasks in coding and mathematics. Particularly noteworthy is their ability to surpass competitors, including Google’s Gemma 2-9B, Meta’s Llama 3.1-8B, and others in the under 13B-parameter category, with the exception of the MMLU benchmark, which evaluates language comprehension capabilities.
With the deployment of Falcon 3 now active on Hugging Face, TII aims to enable a wide array of users to implement cost-effective AI solutions without facing significant computational constraints. The models are tailored to deliver speedy processing for targeted tasks, thereby enhancing various applications across multiple industries, including but not limited to customer service chatbots, personalised recommendation systems, data analysis, fraud detection, healthcare diagnostics, supply chain optimisation, and educational tools.
TII’s future plans include expanding the Falcon family by introducing models with multimodal capabilities, set to launch in January 2025. Additionally, all Falcon 3 models have been released under the TII Falcon License 2.0, a permissive framework that promotes responsible AI engagement and development. To facilitate user interaction with the new models, TII has unveiled a Falcon Playground, a testing environment where researchers and developers can experiment with Falcon 3 before integrating these models into their systems.
Source: Noah Wire Services