The Allen Institute for AI (Ai2) Rolls Out Open-Source Multimodal AI Models

Seattle, WA—25th September, 2024 – The Allen Institute for AI (Ai2) has launched Molmo, a groundbreaking family of open-source multimodal AI models. These state-of-the-art models outperform several top proprietary AI systems, including OpenAI’s GPT-4o, Anthropic’s Claude 3.5 Sonnet, and Google’s Gemini 1.5, across various third-party benchmarks.

Advanced Capabilities in Image Analysis

Molmo models accept and analyse user-uploaded imagery, matching the capabilities of leading proprietary systems. Ai2 highlights that Molmo relies on "1000x less data" than its competitors, achieved through innovative new training methodologies detailed in their latest technical report.

Open Research Commitment

Ai2, founded by the late Paul Allen and led by Ali Farhadi, underscores its commitment to open research by providing high-performing models with open weights and data. This initiative aims to benefit the broader research community and companies seeking customisable AI solutions.

The Molmo family consists of four primary models:

  1. Molmo-72B: The flagship model featuring 72 billion parameters, based on Alibaba Cloud’s Qwen2-72B.
  2. Molmo-7B-D: A demonstration model derived from Alibaba’s Qwen2-7B.
  3. Molmo-7B-O: Based on Ai2’s OLMo-7B.
  4. MolmoE-1B: A mixture-of-experts model that nearly rivals GPT-4V in both academic benchmarks and user preferences.

High Performance and Open Licensing

The models have achieved leading performance on several third-party benchmarks, surpassing various proprietary alternatives. Available under the permissive Apache 2.0 license, these models can be used for a wide range of applications, including enterprise-grade solutions.

Molmo-72B, in particular, has excelled in academic evaluations, clinching the top spot on 11 key benchmarks and ranking second in user preference, just behind GPT-4o. Vaibhav Srivastav, a machine learning developer advocate engineer at Hugging Face, praised Molmo on X, describing it as a formidable alternative to closed systems, setting a new standard in open multimodal AI.

Innovations in Visual Grounding and Robotics

The inclusion of pointing data in Molmo has caught the attention of researchers, including Google DeepMind’s Ted Xiao. He noted that this feature makes Molmo particularly effective for visual grounding in robotics, outperforming other multimodal models in tasks involving complex visual explanations and interaction with physical environments.

Architectural and Training Innovations

Molmo models leverage sophisticated architectures to optimise efficiency and performance. They utilise OpenAI’s ViT-L/14 336px CLIP model as the vision encoder, which processes multi-scale, multi-crop images into vision tokens. These tokens are then projected into the language model’s input space via a multi-layer perceptron (MLP) connector for dimensionality reduction.

The training of Molmo involves two critical stages:

  1. Multimodal Pre-training: Models are trained on generating captions from high-quality image descriptions provided by human annotators, compiled into the PixMo dataset.
  2. Supervised Fine-Tuning: The models are fine-tuned using a diverse set of academic benchmarks and newly created datasets, enabling them to perform complex real-world tasks like document reading and visual reasoning.

Benchmark Performance

Molmo models have demonstrated outstanding performance across various benchmarks. For example, Molmo-72B scored 96.3 on DocVQA and 85.5 on TextVQA, outperforming Google’s Gemini 1.5 Pro and Anthropic’s Claude 3.5 Sonnet in these areas. The model also surpassed GPT-4o on Ai2’s proprietary benchmark, AI2D, designed to evaluate complex visual reasoning.

Future Directions and Accessibility

Ai2 is committed to making these models and accompanying datasets available publicly on its Hugging Face space, ensuring compatibility with popular AI frameworks like Transformers. Over the coming months, Ai2 plans to release additional models, training codes, and an expanded technical report.

Researchers and developers eager to explore Molmo’s capabilities can access a public demo and several model checkpoints via Molmo’s official page.

Conclusion

The release of Molmo by Ai2 marks a significant step forward in the landscape of open-source AI, offering high performance and innovative capabilities that rival leading proprietary systems. Through continued open access and collaborative efforts, Ai2 aims to drive further advancements and applications in the AI community.

Source: Noah Wire Services