To further strengthen our commitment to providing industry-leading coverage of data technology, VentureBeat is excited to welcome Andrew Brust and Tony Baer as regular contributors. Watch for their articles in the Data Pipeline.
After Snowflake and MongoDB’s product fireworks a couple of weeks ago, Databricks joined the party. At its ongoing Data and AI summit, the San Francisco-headquartered data lakehouse company has made a number of notable announcements, starting from Project Lightspeed aimed at improving streaming data processing to a more open Delta Lake and improved MLFlow.
However, the summit hasn’t just been about platform improvements from Databricks. Multiple players forming a part of the modern data stack have also announced new and improved integrations to help their customers get the most out of their lakehouse investment.
Below is a rundown of key new integrations.
Data observability provider Monte Carlo first announced quick, no-code integrations to help enterprise users get end-to-end data observability for Databricks data pipelines. The company said it will let enterprises plug Monte Carlo into Databricks meta-stores, unity catalog or delta lake and use them to gain out-of-the-box visibility into data freshness, volume, distribution, schema and lineage – and the anomalies associated with them. This way, teams will be able to quickly detect structured and unstructured data incidents, starting from ingestion in Databricks down to the business intelligence (BI) layer, and resolve them well before they affect downstream users.
Acceldata, Monte Carlo’s competitor in the data observability space, also announced an integration for end-to-end data pipeline visibility. This solution will track pipeline quality inside and outside Databricks to flag incidents and also include performance optimization capabilities such as automated stability tra...