Big Data Visualization Using Datashader in Python | How does Datashader work and why is it so fast?

Key Takeaways

i'm trying to plot 11 million data points on my laptop it looks like a big blob and i can't extract any meaningful information from it what should i do let me just google it big data visualization python okay looks like i should use data shader let me give it a try wow oh my god it works like magic it is so fast and the plot is interactive let me show you what data shooter is and how data shooter works data shooter was developed by my anaconda colleagues philip rudiger john stevens and banner it is one of the seven libraries in the hall of face ecosystem if you'd like to learn more about holobas check out my previous video on how and why i love hallows so why is big data visualization so hard there are two main obstacles to visualize big data first is b if you were to plot big data using your regular python plotting libraries it could be extremely slow and your jupiter kernel could crash second is image quality even if it doesn't crash and you're willing to wait most plotting libraries will simply keep drawing each new data point as a circle or some other shape on top of each other which will result in over plotting even adding alpha transparency for over plotting points won't always help in the situation data shader provides elegant and seemingly magical solutions to those two obstacles next i will show you an example and peek into the magic involved first let's import needed packages in your contact environment i guess before that if you haven't installed anaconda or minicon already please first install those then you can create a new contact environment activate this environment and then install the packages we need now we can open up a jupyter notebook or jupyter lab and import the packages we need next let's reading some data from parquet files using the pandas read parquet function here we'll load two columns representing taxi drop-off locations for the very largest files you'll want to use a distributed processing library like desk with data shader but here we only have a prokaryote file with only 11 million records which datasheeter can easily handle on a laptop using pandas without any special computing resources then we can plot our data using data shader i only took four lines of code where we use the datasheet function to data shade all the points alternatively instead of using the datasheet function we can use hp plot with rasterize equals true to apply rasterization using data shader i highly recommend using hp plot it is super easy how does data shader work exactly data shader turns your data into a plot using a five step pipeline projection aggregation transformation color mapping and embedding i'm going to break down my example into those small steps so that you can see exactly what data shader is doing under the hood let's first install the underlying data shooter functions so that we can run through the individual steps the first step is projection we define a 2d canvas with width and height for the data to be projected onto the canvas defines how many pixels we would like to see in the final image but for speed each point is actually projected during the aggregation step next the second step is aggregation after we define the projected canvas we project each point into the two-dimensional output grid and aggregate the results per pixel data shader supports many options for such aggregation but in this exam we simply count how many data points are projected into each pixel by iterating through the data points and incrementing the pixels where the point lands the result in this case is a two-dimensional histogram counting drop-offs per pixel the third step is transformation once the data is in this grid we can do any kinds of transformations we like such as selecting on a certain range of accounts masking the data based on the results of other datasets or values and others here the drop of data ranges from zero in some pixels and tens of thousands in others and if we were to plot the grid directly we would only see a few hotspots to make all the different levels visible as in the image we showed you earlier the data is transformed using the image processing technique histogram equalization to reveal the distribution of the counts rather than their absolute values histogram equalization is actually folded into the color mapping steps but we can do explicit transformations at this stage if we want such as squaring the counts the fourth step is color mapping we can render the bend grid data into the corresponding pixels of an image each bin value is mapped into one of the 256 colors defined in a color map for example here it uses histogram equalization in this example we're using the fire color map from the color set which starts at black for the lowest values and goes up to red for higher values and then yellow for even higher values and finally white for the highest accounts per pixel which is tens of thousands in this case we set the background to black to better visualize this data the final step is embedding as you can see data shader only renders the data not any pixels color bars or similar features you would expect in a full plot to get those features that help you interpret the data we can embed this image generated by data shooter into a plot the easiest way is to use holoviews which is a high-level plotting api that provides the flexibility to use either matlab lib bokeh or poly as a back end here is an example of using hover views to define a points objects and then data shading all the points here we demonstrate an alternative method rasterize instead of data shade so that bokeh is in charge of the transformation and color mapping steps and allows hover and color bars to work you can also use rasterize with hp plot just like we showed in in the example earlier why is data shader crazy fast first we need to talk about the original data format data shooter is so fast that reading in data is usually the slowest step particularly if your original data is a bunch of json files or csv files the parquet data format is usually a good choice for the columnar data like the drop-off points we have here because it's compact quick do loading efficiently reads in only the columns and ranges you need and supports distributed and out of core operation when appropriate second with the right input file formatting we can investigate the next most expensive task which is the combined projection plus aggregation step the step requires calculating values for each of the millions of data points while all the subsequent calculations using the final fixed size grid and are that's much faster so what does data shooter do to make this step fast first data shaders aggregation calculations are written in python but then just in time compiled into the wicked fast machine code using number second data shooter plus number also supports cuda qdf data frames as a dropping replacement for pandas data frame that runs even faster if you have a gpu third datasheeter can also parallelize its pipelines so that it can make use of all the computing cores you have available scaling to even larger data sets and even faster results because datasheeter is so fast we can actually visualize big data interactively dynamically rejoin whenever we zoom or pan here is an example where you can view the nyc taxi data interactively in a panel dashboard my favorite example on ship traffic illustrates that even though all you see is the pixelated image the data shader renders you can still inspect individual data points and understand what they're saying the other examples at pivx.org show datasheeter for much larger files up to billions of points on an ordinary laptop if you'd like to learn more about data shader i highly recommend checking out the many great examples at datasheeter.org and our examples using datasheeter in a panel app in awesomepanel.org thank you hope you enjoy this video and i'll see you next time bye