Lightdash takes on Looker with an open source BI platform built for dbt

Lightdash, an open source business intelligence (BI) platform that’s setting out to challenge proprietary incumbents such as Looker, is officially launching its core commercial product to the public today backed by $8.4 million in funding. The seed round was led by Accel, with participation from Moonfire, Y Combinator (YC), Snyk founder Guy Podjarny, and Gitlab CMO Ashley Kramer.

In its original guise, Lightdash was known as Hubble when it graduated from YC’s S20 batch, with a focus on helping companies run tests on their data warehouse to identify issues with data quality. These data quality metrics, as it transpired, were most useful inside BI tools, which cofounder and CEO Hamzah Chaudhary says no BI tools on the market supported. And so they pivoted the product to Lightdash and began working on the new project full-time in May 2021 to solve a big pain point for data analysts. 

“Modern data analysts are becoming increasingly like software developers, but they’re stuck with enterprise user-interfaces that lock in business logic and slow them down,” Chaudhary explained to TechCrunch. “Lightdash gives productivity tools to analysts, which allows them to deploy BI at enterprise scales with substantially less effort.”

Lightdash is built specifically for dbt, a command line-based data transformation tool that allows analysts to transform raw data in their warehouse using SQL and their usual text editor. Dbt is the “t” in “extract, load, transform (ELT), and Lightdash in turn transforms any dbt project into a “full-stack BI platform.”

It’s worth noting that Lightdash is both a front-end and a back-end. So for regular business users who might not be conversant in SQL (e.g. marketing or finance), Lightdash serves as the visual layer for dbt, while in the back-end data analysts and other more technical users can build custom workflows and define all the business logics for metrics and KPIs, essentially “abstracting the complexity” away of how they’re all calculated.

“Lightdash is focused on giving data analysts tools to enable true self-serve BI for the rest of their company,” Chaudhary said.

Lightdash dashboard Image Credits: Lightdash

BI basics

Business intelligence, for the uninitiated, is the process of mining, integrating, and organizing disparate data sets to inform decision-making. Big data insights is the name of the game, helping analysts draw meaningful conclusions, identify and visualize patterns, and predict future outcomes (e.g. sales forecasts).

The BI market is big business, pegged as a $23 billion industry in 2020 and predicted to surpass $33 billion by 2025, which is likely why Google doled out more than $2 billion to acquire Looker back in 2020, and Salesforce snapped up Tableau for more than $15 billion before that.

Lightdash, for its part, serves as the main gateway for companies looking to explore their data, sporting native integrations with many of the tools that constitute the modern data stack, from dbt through Snowflake, Airbyte and Fivetran.

“Lightdash is built to be open and integrated with the modern data stack, not a closed system,” Chaudhary said. 

Lightdash in action Image Credits: Lightdash

The genesis of Lightdash can be traced back to Chaudhary and his CTO cofounder Oliver Laslett’s time together at UK insurtech Cytora, where they were charged with scaling the company’s data analytics output. 

“We saw the huge chasm in the quality of tools available to our data teams versus our software engineers, even though the jobs being asked of our data analysts were equally technical,” Chaudhary said.”When we left Cytora, we knew we wanted to empower data analysts and data teams by giving them tools that were more fit for purpose and up-to-date. We [then] worked as data consultants helping companies set up their data stacks, and eventually figured out that the weakest link in the workflow was the BI layer, because BI tools don’t integrate well with the rest of the data stack, don’t support any data developer workflows, and make it difficult for data teams to collaborate effectively.”

And this, perhaps, gets to the crux of what Lightdash is trying to do: it’s about helping data analysts and analytics engineers use their existing tools, such as code editors, and allowing teams to collaborate at scale. “A platform built to integrate with other tools,” as Chaudhary puts it.

The open source factor

While the main Lightdash project is open source, the company launched a fully-managed and hosted Lightdash Cloud service in beta back in January, preceding the launch of its free self-hosted Community edition in June. Today signals the public beta launch of its core Cloud product, which has so far amassed a waitlist of some 600 companies.

“We had always planned to have a commercial version of Lightdash, but also wanted to ensure that the open source product was usable too — which is why almost the entire product feature set is available in the self-hosted open source version,” Chaudhary said.

Looker and its ilk are the obvious comparisons here, but Lightdash’s open source credentials are one of its key differentiators, going some way toward ingratiating itself to SMEs and larger enterprises. Open source is a major selling point for security-conscious companies in particular, as it means that they have full visibility into how their data is being handled. It also means they can start off small-scale by deploying Lightdash to just one or two teams to test-drive it before expanding further into their stack if they like what they see.

“This is much more efficient compared to proprietary BI tools where you often have to go through a lengthy sales and procurement process before you get hands-on with the product itself,” Chaudhary said. “For a lot of enterprises and startups alike, this is the preferred method to get started with new tools — it has the added benefit that being open source, it’s already built to deploy on-premise if needed, which is often a requirement for large enterprises.”

This is a model that has yielded success for so many startups before it: an open source foundation for organizations that need full control and flexibility, with a commercial layer that removes much of the complexity and spadework for those that need it.

A quick peek across the competitive landscape reveals a few other players in the commercial open source BI space, including Metabase which raised a $30 million tranche of funding last year, and Preset which raised around $36 million to commercialize the Apache Superset project. So it’s clear there is real demand not only for BI, but open source BI backed by a fully supported commercial service.

Lightdash is a remote-first company, with its founders based in or around London and the rest of its 8-person team spread across Europe, though the company is incorporated both in the U.S. and the U.K. With $8.5 million in the bank, which includes a hitherto unannounced $2.4 million pre-seed round led by Moonfire, the company said it’s now well-financed to expedite its hiring particularly in its product team, as well as expand an educational program called Lightdash University, which is designed to “upskill” BI teams. 

Lightdash takes on Looker with an open source BI platform built for dbt by Paul Sawers originally published on TechCrunch