Streaming to Kafka Just Got Easier with DLT Pipelines

Key Takeaways

thousands of organizations rely on Delta life tables or DLT to build reliable data pipelines DLT makes pipeline development declarative providing built-in data quality checks error recovery and efficient scaling on serverless compute whether you working with files Landing in cloud storage or realtime data streams from apachi kfka DT has a unified programming model for both batch and streaming data processing but how can we publish data back to Kafka from a data pipeline Delta life tables now introduces native Kafka syn support you simply Supply a Kafka sync using the create Sync API and then use a pen flow to stream your data to this sync this isn't creating another DLT table it's establishing a continuous flow to Kafka okay then let's look at the code I'm setting the bootstrap and the topic for the Kafka server here and then I'm using confluent Cloud so I need this API key and I need the secret the confluent secret and both of them I get from a data brick secret scope that I set up with the CLI and I retrieve those values and then I create the Kafka sync and the Kafka sync is actually a simple create sync I give it a name I give it a type which is Kafka and then I provide the properties now the properties depend on the kafa flavor and the Kafka server that I'm using here I need to specify those sassle and SSL parameters but as I said that depends on your Kafka server so the command is a simple create sync and then what I do is I read a sales transaction data stream now this data is actually coming from uh Marketplace it's a public data set everyone of you can get that for free and it's a table which is OV So it's b house sales transaction and I'm reading this as a stream and then I'm also filtering I I only want the high uh value payments it's all about cookies it's the cookies data set from uh from the summit and the way I read it here is in a streaming table which is defined here DLT table that makes it a streaming table so we have a streaming table um that gets us access to the sales transaction and then all I need to do is since I have the sync already all I need to do is append flow I specify the sync that I want to use here as a Target and then I read from my streaming table I write back to that sync ad Chason structure which is written as a value if I've run this it looks like this that's the pipeline actually the the data set on Marketplace is like 3,300 uh transactions but because of the filter the streaming table is only raing like 696 and all of this goes to the Kafka sync and uh that's it basically so you can get the data set from Marketplace it's a one click it gives it gets you access it's then available in unity catalog as a shared data set and if you're looking for the source code of this example it's available on GitHub go to GitHub data braks TMM TMM that's the thing you need to remember and then you get to the source code you can play with it yourself one more thing did the data really make it into cfom well this is the broker site and if I click on messages here you see exactly 696 messages are ingested and you see the messages here and they already look like what I was sending but if I drill into that you see exactly the same Json structure um that I published to that topic as a value well thank you for joining us today go and build something amazing and let us know about it