Python Based Time Series Analytics on Databricks

Databricks · Intermediate ·📊 Data Analytics & Business Intelligence ·2mo ago

Key Takeaways

The video demonstrates Impulse, an open-source framework for Python-based time series analytics at scale, built by Databricks and AVL to handle petabyte-scale telemetry data from proprietary binary formats.

Full Transcript

Cars are becoming software-defined, autonomous, AI-defined, and every one of these changes generates exponentially more data. A single car test today produces petabyte-scale telemetry. For decades, engineers analyzed all this data using desktop tools that don't scale, working with hundreds of proprietary file formats that don't talk to each other. That's why Databricks partnered with AVL to build an open-source declarative time series analytics framework, Impulse. This project introduces a new way to query petabyte-scale measurement data in plain Python. First, raw sensor data lands in Databricks Lakehouse. Impulse standardizes it into a silver layer data model. Now, instead of digging through proprietary files on a laptop, an engineer can ask a question on petabytes of measurement data in 10 lines of Python. The framework handles the scale, the governance, and the file formats automatically. What used to take days now takes minutes. And ultimately, this means safer, smarter, better vehicles built faster and validated continuously. Find the resources to get started in the captions and follow for more data and AI stories.

Original Description

Analyzing petabyte-scale telemetry locked in hundreds of proprietary binary formats has been an engineering challenge for years. That's why AVL partnered with Databricks to build the open-source framework Impulse: Python-based time-series analytics at scale. The framework handles scale, governance, and format translation automatically. 🔗 GitHub repo: https://bit.ly/impulse-repo Share with others!
☆ Save to playlist Watch on YouTube ↗
Sign in to unlock AI tutor explanation · ⚡30

The video introduces Impulse, a Python-based time series analytics framework that handles petabyte-scale telemetry data from proprietary binary formats, allowing engineers to analyze data in minutes instead of days.

Key Takeaways
  1. Land raw sensor data in Databricks Lakehouse
  2. Standardize data into a silver layer data model using Impulse
  3. Write Python code to query petabyte-scale measurement data
  4. Use Impulse to handle scale, governance, and file formats automatically
  5. Analyze data and gain insights in minutes instead of days
💡 Impulse framework allows engineers to analyze petabyte-scale telemetry data in minutes instead of days, enabling safer, smarter, and better vehicles to be built faster and validated continuously.

Related Reads

Up next
The Test Is Right 99% of the Time
DataMListic
Watch →