Notes from Industry
Design considerations for building a data mesh architecture
All organizations I work with understand the importance of data and are either interested in or planning their next generation of a modern data platform. Their aim is to move away from tightly coupled data interfaces and varying data flows towards an architecture that allows eco-system connectivity: a cloud distributed data mesh, which allows domain-specific data and treats “data-as-a-product,” enabling each domain to handle its own data pipelines. …
In my other posts you learned how ABN AMRO makes data available in a data mesh style architecture. In this blogpost you will learn about how to break big data monoliths apart.
Data-driven decision-making shift
In the years since data warehouses became a commodity, much has changed. Distributed systems have gained great popularity, data is larger and more diverse, new database designs have popped up, and the advent of cloud has separated compute and storage for increased scalability and elasticity. …
Azure Purview is a unified data governance service that helps organizations to manage and govern their data estate. It provides a holistic, up-to-date map of your data landscape with automated data discovery, sensitive data classification, and end-to-end data lineage.
One of the great things of Azure Purview is its openness. Azure Purview uses the Apache Atlas Open API ecosystem, with some enhancements and additions by Microsoft. In this tutorial, you learn how to use these APIs by creating custom lineage.
Prerequisites
To get started, you must have an existing Azure Purview account. …
The last couple of weeks I used my evening time trying to hack around VS Code to create a consistent workflow environment. While at first glance this seems to be easy, there are a number of considerations you need to take into account when working on multi-component applications. The project, which I used is hosted here: https://github.com/pietheinstrengholt/rssmonster/
Installation
This guide combines Docker and Visual Studio Code for creating consistency across your development environment. Both are required to be installed to use this guide. …
Azure Machine Learning is a cloud service for accelerating and managing the machine learning project lifecycle. It enables you to create models or use a model built from an open-source platform, such as Pytorch, TensorFlow, or scikit-learn. Azure ML is complimented with additional MLOps tools, which help you monitor, retrain, and redeploy models.
In this blogpost I want to demonstrate how to build, train and deploy a model using an Azure Machine Learning notebook and compute instance for development, a cluster for training, and a container website for production. For this demo we will use the Titanic passenger data (Titanic.csv)…
With this blogpost it’s my pleasure to offer you the chance to have a look at the cool initiatives that we do at ABN AMRO. I want to emphasize that our Data Integration Architecture pertains to a common problem area. The ideas that are discussed in this article are not restricted to only ABN AMRO and are non-industry specific. The audience is anyone with an interest in data: data architects, data engineers, solution designers, data professionals, etc.
The architecture that we have developed helps our architects, engineers and solution designers to pick the right building blocks for delivering value for…
At ABN AMRO, we are quite familiar with a data mesh. As a matter of fact, we’ve been working on this new type of architecture for quite some time. 4 years ago, our data journey started. With this blogpost it is my pleasure to let you peek deeper inside.
The term data mesh was originally coined by Zhamak Dehghani. It describes a paradigm shift from traditional data architectures, like enterprise data warehouses and data lakes, towards a modern distributed architecture, using concepts like Domain-Driven Design, platform- and self-service thinking, and treating data as products.
One of the biggest problems many…
This article consolidates a large amount of content from the book Data Management at Scale. This content didn’t make it to the final book, because various external reviewers found it unchallenging. So, if you believe you have a good understanding of data management already, the content discussed in this article might sound familiar. However, I have noticed that less mature data professionals find the data integration part and context perspectives, difficult to digest. Therefore I decided to make this content freely available to all of my followers.
Data Integration is considered to be part of data management, but since data…
The content of this blogpost builds up on my previous article: Data Integration and Data Modelling demystified and consolidates a large amount of content from the book Data Management at Scale. This content didn’t make it to the final book, because various external reviewers found it unchallenging. So, if you believe you have a good understanding of data and database management already, the content discussed in this article might sound familiar.
A crucial application component that I haven’t dived deep into the previous article is the application data layer, or database layer, of the application. The values or data of…
Data warehousing and business intelligence play an important role in many, if not, all of the large sized organizations working on turning data into insights that drive meaningful business value. Data warehousing is process for collecting and managing data from varied sources to provide meaningful business insights. Business intelligence comprises the strategies and technologies used for providing these insights. Both areas can be considered part of data management. They are heavily intertwined with the other data management areas and depend a lot on data integration.
Data warehousing became popular during the nineties and started as a common practice to collect…
