DataPelago Emerges from Stealth to Revolutionise Data Processing Virtualisation
Mountain View, CA – October 2023 – DataPelago today announced its emergence from stealth mode, unveiling a groundbreaking virtualisation layer designed to revolutionise the handling of AI, data analytics, and ETL workloads. This innovative technology allows users to run their data-intensive tasks on any physical processor without necessitating code changes, potentially delivering substantial efficiency and performance improvements across data science, analytics, engineering, and high-performance computing (HPC).
The rise of generative AI has led to a significant demand for high-performance processors capable of managing the extensive computational demands of large language models (LLMs). Simultaneously, organisations are eager to maximise efficiency from existing compute resources for advanced analytics and big data, while continuously managing the proliferation of data in varied formats.
DataPelago's response to these market dynamics is a universal data processing engine, aimed at decoupling data workloads from specific compute infrastructures. This flexibility allows users to operate big data, advanced analytics, AI, and HPC tasks on any available cloud or on-premises system that aligns with their price/performance targets.
"Much like Sun built the Java Virtual Machine or VMware innovated the hypervisor, we're developing a software-based virtualisation layer," said Rajan Goyal, Co-founder and CEO of DataPelago. "This layer provides a clean abstraction for anything above it."
The virtualisation layer, positioned between the query engines (such as Spark, Trino, Flink, and SQL) and the underlying hardware (including CPUs, GPUs, TPUs, and FPGAs), enables users to submit jobs without concern for the compute infrastructure. DataPelago's system then automatically routes and executes these jobs on the optimal processor to meet the set availability or cost/performance goals.
Technologically, when a job is executed, the processing engine converts it into a plan. DataPelago employs an open-source layer, like Apache Gluten, to transform this plan into an Intermediate Representation (IR) using standards such as Substrait or Velox. This IR is then sent to a worker node within the DataOS component of the DataPelago platform, where it converts into an executable Data Flow Graph (DFG). DataVM then evaluates the DFG nodes and dynamically allocates them to the appropriate processing element, enhancing performance and efficiency.
The company's software optimally matches workloads to processors' sweet spots, bridging performance gaps often encountered with accelerated compute engines. Goyal emphasised that while DataPelago's solution may not surpass the peak performance of native code developed specifically for processors, it closely approximates such performance while simplifying the complexity involved and detaching users from middleware dependencies.
“There is a significant gap between expected and actual performance of GPUs in applications, a gap that DataPelago bridges by targeting optimal processor usage and reducing I/O overheads,” explained Goyal. "Our architecture leverages operators fusion and streaming execution to minimise data movement, achieving 80-90% of GPU peak performance."
DataPelago targets a range of data-intensive tasks, including SQL analytics, ETL processes, and streaming workloads. Generative AI applications benefit substantially, particularly in LLM training and runtime operations, due to accelerated retrieval augmented generation (RAG), fine-tuning, and vector embeddings creation capabilities.
The platform is versatile, operating on-premises or in the cloud. For cloud lakehouses such as AWS EMR or Google Cloud’s DataProc, DataPelago claims it can reduce the required node count by 90%, offering a substantial performance boost and a twofold improvement in total cost of ownership without necessitating code alterations.
Benchmark comparisons have shown that DataPelago outperforms established platforms significantly, with a 3x to 4x improvement over Databricks Photon, which is considered a market leader.
Goyal highlighted the potential for DataPelago’s technology to enhance scientific computing and modelling tasks on HPC setups, offering a simplified abstraction that optimises low-level code and kernel orchestration for efficient operations.
Currently, DataPelago has a mix of pilot and production-phase customers, ahead of a planned full-scale launch in the first quarter of 2025. The company also announced it has secured $47 million in funding, thanks to investors including Eclipse, Taiwania Capital, Qualcomm Ventures, Alter Venture Partners, Nautilus Venture Partners, and Silicon Valley Bank, a division of First Citizens Bank.
DataPelago’s innovative approach promises to significantly enhance the efficiency and performance of data processing workloads, providing substantial value to organisations striving to navigate the complex landscape of advanced computing and analytics.
Source: Noah Wire Services