Designing a metadata-driven processing framework for Azure Synapse and Azure Purview

In the blogpost Modern Data Pipelines, you learned about data pipelines and some of their difficulties. You also read up on an approach by binding Azure Synapse Analytics and Azure Purview in order to build a framework for intelligently processing data.

In this article I want to take the data processing approach one step further by making data pipelines metadata-driven. A metadata-driven framework allows for more scalability. It speeds up development, provides better maintainability…