Unity Catalog Fine-Grained Access Controls on External Engines

Databricks · Beginner ·🔄 Data Engineering ·3w ago

Skills: ML Pipelines70%Security Basics60%

Key Takeaways

Demonstrates Unity Catalog fine-grained access controls on external engines using Databricks and Apache Spark

Original Description

See how Unity Catalog enforces row filters and column masks identically, whether you're querying from Databricks or from open-source Apache Spark running on a VM outside your workspace. This demo walks through defining ABAC policies once in UC, verifying enforcement on Databricks serverless SQL, then running the same query from OSS Delta-Spark on EC2 to show identical results. Finally, we demonstrate external engines creating and writing to managed tables with catalog-managed commits, fully governed and automatically optimized. Learn more about Catalog Commits: https://www.databricks.com/blog/convergence-open-table-formats-and-open-catalogs-catalog-commits-generally-available

Watch on YouTube ↗ (saves to browser)

Sign in to unlock AI tutor explanation · ⚡30

More on: ML Pipelines

View skill →

Building a Dog Breed Identifier App from scratch - DogNet

Building a Dog Breed Identifier App from scratch - DogNet

Aladdin Persson

Complete Dockers For Data Science Tutorial In One Shot

Complete Dockers For Data Science Tutorial In One Shot

Part 6 | Deploy ML Model on Kubernetes | Auto-Scaling with HPA and Monitoring with Prometheus

Part 6 | Deploy ML Model on Kubernetes | Auto-Scaling with HPA and Monitoring with Prometheus

Abonia Sojasingarayar

MLOps Tutorial: Build a Full ML Pipeline with MLflow, DVC & Deploy on AWS

MLOps Tutorial: Build a Full ML Pipeline with MLflow, DVC & Deploy on AWS

Analytics Vidhya

Vertex Pipelines: Qwik Start

Coding a Multimodal (Vision) Language Model from scratch in PyTorch with full explanation

Coding a Multimodal (Vision) Language Model from scratch in PyTorch with full explanation

Related Reads

I Built My Second ETL Pipeline. This Time, I Started Thinking Like a Data Engineer

Learn how to build a production-ready ETL pipeline with Python, Docker, PostgreSQL, and Kestra by thinking like a data engineer

Towards Data Science

JuiceFS Sync for PB-Scale Data Transfers: Resumable Sync, Encryption, and Bandwidth Control

Learn how to efficiently transfer large volumes of data using JuiceFS Sync, which offers resumable sync, encryption, and bandwidth control, ideal for PB-scale data transfers.

How Airflow is using AI to make data engineering more resilient, not more complex

Airflow uses AI to make data engineering more resilient by detecting data drift, resuming failed pipelines, and fixing issues automatically, reducing complexity and improving reliability.

What Can We Do When Memory Becomes the New Bottleneck in Data Engineering?

Learn how to overcome memory bottlenecks in data engineering using Pandas chunking, Dask, and Polars, and why it matters for processing large datasets

Towards Data Science

A Moment Frozen in Time | Arnav Iyengar | TEDxJenks Youth