Grab, a leading Southeast Asian technology company, recently made significant advancements in its internal data discovery processes by integrating advanced Generative AI (GenAI) technologies into its Hubble data discovery tool. The primary motivation behind this development was to address the complexities and inefficiencies of managing over 200,000 data tables, sourced from its expansive data operations encompassing data lakes, Kafka streams, production databases, and machine learning features.
Historically, finding the appropriate datasets for new applications posed a significant challenge, often reliant on "tribal knowledge" among employees. Approximately 18% of data searches were abandoned, leaving many data consumers without the needed insights. The company identified that data discovery processes could take several days, hampering progress and efficiency across data-driven projects.
Responding to these challenges, Grab’s team, led by product manager Shreyas Parbat, set out with an ambition to automate the data discovery process and effectively remove human intervention through the integration of Large Language Models (LLMs). The goal was to streamline the process to seconds, utilising technology that generates and structures database documentation proactively, thus liberating users from the traditional methods of inquiry.
Central to improving Hubble, Grab invested in enhancing ElasticSearch’s ability to manage and interpret data metadata effectively. This effort included user interviews to optimise the search tool, leading to implementations such as concealing obsolete tables, demoting outdated entries, and promoting relevant and certified datasets through strategic tagging. These changes significantly increased user engagement as evidenced by a 12% rise in the search click-through rate.
Another notable innovation involves the use of GPT-4 to automate documentation creation. By generating documentation from table structures and sample data automatically, Grab has markedly increased documentation coverage from a scant 20% to an impressive 90%. Nearly all users—95%—acknowledged the usefulness of these AI-generated documentations. This feature was incorporated into the Hubble UI, empowering data producers to further refine or create table-level documentation seamlessly.
To further aid data consumers, the Hubble team designed a Slack bot leveraging Glean technology. This integration allows users to access data-lake table documentation within the Slack platform, enhancing accessibility and reducing the time required for data discovery. The HubbleIQ Slack bot, constructed using Glean Apps, connects with Hubble search functionality, presenting another leap in user convenience.
Looking ahead, Grab is not halting its momentum. Plans are in place to enrich the GenAI functionality further, particularly by adding more context to the documentation generator, thereby improving the generated documentation’s depth and relevance. The team also envisions features that would allow analysts to update documentation automatically based on discussions within Slack threads. Additionally, Grab intends to incorporate Reflexion technology, a move expected to elevate the quality and utility of the documentation still further.
These improvements signal Grab’s commitment to harnessing artificial intelligence to revolutionise data management and discovery, potentially setting a new standard for business intelligence processes within the industry. As the company continues to refine and enhance its tech solutions, the impact on productivity and efficiency across its operations is anticipated to be profound.
Source: Noah Wire Services