Building a Data Pipeline from Scratch
Building a Data Pipeline from Scratch
medium.com Building a Modern Data Pipeline: A Journey from API to Insight
The buzz of ‘Big Data’ has passed. Terabytes of data is the new normal, and efficiently managing and processing data is more critical than…
I’ve written a series of Medium articles on creating a Data Pipeline from scratch, using Polars and DeltaTables. The first (linked) is an overview with link to the GitHub repository and each of the deeper dive articles. I then go into the next level of detail, walking through each component.
The articles are paywalled (it took time to build and document), but the link provided is the ‘family & friends’ link which bypasses the paywall for the Lemmy community.
I hope some of you may find this helpful.
0
comments