What are the Data Mesh common challenges? What are the people involved with Data Mesh saying about it?
Many people are talking about Data Mesh. But there are even more people asking for references on the topic. I have been involved in many Data Mesh engagements and I will be sharing more about it in future articles. In this short article, I share the result of a Data Mesh retrospective with 50 people from different organisations.
In October 2022 I had the pleasure to be with Zhamak Dehghani in a 20 hour Data Mesh training. It was not the first time I helped her with the Data Mesh training, and, most likely, not the last one. I will try adjusting my agenda so I can be with her on her future trainings. She has been doing it once every other month as she is super busy with many engagements. More info on the next trainings here.
There are quite a few amazing benefits for me to be part of the training:
- listen again and again about Data Mesh from its creator,
- share a little on my experience with Data Mesh on-boarding and inceptions (very little as her training is already full of amazing contents; at least, I share a few hints and links to help the participants)
- listen to the avid participants as they share conversations, questions, challenges, learnings and valuable tips about Data Mesh in their organisations.
At the last training, I did a quick retrospective for the participants. The result of the retrospective reveals important information for all of the training participants. But I realised these revelations are useful for many people getting started in Data Mesh. So I decided to share the results in this post.
I used two activities to gather the information I am sharing with you. They are:
- Slider on the Data Mesh principles , and
- Anchors and engine for data mesh in your organisation
Slider on the Data Mesh principles
I started by asking the group to have a quick recap about all the teachings Zhamak shared on the four principles of Data Mesh: Domain Ownership, Data as a Product, Self-serve Data Platform and Federated Computational Governance.
Then I asked each participant to vote on the mostly challenging principle for the Data Mesh transformation in their organisation.
The image below is the result for the voting session.
Domain ownership and Federated computational governance are considered to be the most challenging principles. Then, Data as a product and Self serve data platform are considered to be challenging, but a little less challenging.
The result from this group (about 50 people from different organisations) is very representative. I have seen many groups going through the same activity. In general, people are having more challenges in Domain ownership and Federated computational governance than Data as a product and Self serve data platform.
One very important aspect: this activity is only for raising awareness and starting important conversations. These principles are not to be worked on in isolation.
Anchors and engine for Data Mesh in your organisation
I asked two very straight-forward questions: In the context of Data Mesh, What are the things pushing you forward? What are the things holding you back?
Below is the image, result of the Data Mesh Anchor and Engine activity.
Please note the colour coding for each of the Data Mesh principles (same color used in the previous image activity).
Below is the text from all the notes.
Engine, the things moving you forward:
- Clear separation of the concerns, ownership, responsibilities
- Perception of the data from different perspective, multiple ways of use cases
- There is a lot of technical challenges that engineers will enjoy
- Creation of pizza teams
- Seeing data as value and product
- Transformation to product teams
- Flexibility, scalability
- Setting standards and SLO’s on data will be very happily accepted
- Having teams organised in Value Streams, or something similar and already have DevOps maturity to enable automation.
- Shared responsibility over common goal against centralised committees and processes
Anchor, the things holding you back:
- In our landscape the distance from operational domain to data domain is very big
- Organisational Mindset and culture
- Getting a common understanding of domains in the organisation
- No ready-made tools to choose yet, building complete infrastructure will take huge efforts.
- Convince org to change when not organised around domain today
- Cross company initiative against Data team activity
- Required mind shift in business regarding ownership
- Defining the products
- No familiarity, nowhere to show a working example
- Create Data Products that are valuable on their own
- I have to sell it without complicated words..too many new words in this workshop
- Centralised thinking of existing data teams
- No existing products yet
- DevOps maturity
- How to justify required setup / resources
- Maturity of data platform, mastering of the technologies
- Organisational politics & mindset
- Expectation of centralised governance and approaches which is not happening
- Data office not in line with decentralisation of governance
This short article has no intention to tell you how to handle each of these things holding you back, or to share how to take advantage of the forces pulling you forward. My main intention is to share with you how other people pursuing Data mesh are feeling.
As Zhamak says so clearly:
“Data M;esh is in its infancy. Please stay open to and aware of many advances in this area. More important than any specific practice or tool is to adhere to the Data Mesh principles.”
