principles
The foundational principles behind DataOps Management are:
1. Business focus
2. Design for team productivity
3. Design for constant change.
Principle 1:
business focus
The first question that should always be answered is what business problems you want to solve and what business outcomes you aim to achieve and why. Instead, the focus of data teams is too much on the technology stack and reinventing the wheel, again and again. Having too many tools and complicated technology architectures can easily distract attention from the essentials.
Business people can rarely articulate their needs right in the first go, so you need to have ability to implement things iteratively and incrementally. The better you are at this, the more satisfied and engaged the business stakeholders will be, and more value you get from the data. The quality of outcomes are better, when the validation happens early in the process and feedback loop is short. And business people don't fade out as the they do not have to wait long for changes and improvements.
It is also important to think what technical capabilities you should build yourself and what you should buy instead. If you need to use more time on fiddling with technology stuff than on working on the business problem, something is wrong. What is valuable use of time for your business?
Principle 2:
design for team productivity
Nowadays data and analytics solution development is rarely technology-constrained, but it is mainly constrained by availability of skilled human resources. In the current job market the turnover of people is high. You cannot rely on certain hero individuals, as they rarely stay long with you to maintain the system.
Short time-to-value should be an outcome of built-in productivity of the whole team and adaptability of the data platform, instead of “quick and dirty / cut the corners / we fix it later” mindset. You should tune the processes and practices so that the whole team (or multiple teams) can maximize their productivity in the longer term. Sometimes an individual professional can be seemingly very productive, but from the perspective of the whole their productivity can be even negative in the worst case. This can be easily missed, if you only look for a short term productivity.
Three ways to increase the productivity of your team:
Increase reusability
The more reusable content you have, the more easy and quick it should be able to answer business data needs.
Focus on improving the flow of work
Doing implementation in small batches will naturally increase the flow of work, so you should use incremental and iterative approach for solving the business requirements. The ultimate level in this is continuous delivery, which is more of a system capability than just a way of breaking work in smaller pieces. Integrating and deploying changes early and often in small batches, instead of large deployments, can dramatically increase the capability of delivering value to customers.
Do continuous improvement systematically
There will inevitably be some technical debt, as it is not often possible to have all the best information available or sometimes you just need to cut corners. You should just be aware of this, put this work on your backlog and regularly reserve resources for doing the refactoring and cleaning work. When you do a mistake, make sure the team does not do the same mistake again. Too often learnings remain on individual level, and it is not shared with the team. Team learning needs to be systematic.
Principle 3:
Design for constant change
A data platform is a big investment and therefore its lifecycle should be as long as possible, to get a return on the investment. A key factor in this is how well a cloud data platform stands the test of time in serving the needs of the business. The wrong approach in trying to achieve this is to try to isolate the data platform from change - this will lead to its inevitable death. The right approach is to design it to be productive and adaptable to change.
You should have a mindset that everything will change all the time and your data solutions and platform will be affected by this change. Changes in business needs, source systems, technologies, legislation and people. Person locks are often worse than (software) vendor locks.
Four ways to design change into the system:
Adopt continuous delivery approach
When you have a fine-tuned continuous integration and deployment capability, it speeds up the time to value for all changes and makes your data warehouse adaptive to changes.
Utilize the loose coupling design principle
Do this both in data architecture and technology architecture design. This enables you to build and improve business solutions without a constant need for refactoring the whole end-to-end system.
Automate repetitive tasks as much as you can
Before automating, make sure that working practices are standardized enough. If you do not have systematic processes and repeatable procedures, trying to automate them, is a waste of time.
Choose and utilize a solid and proven data modeling methodology
You should consider carefully, if there are real reasons why you should not use a standardized modeling methodology that is designed to adapt to changes and enable automation. There might be a bit higher learning curve in the beginning, but it will pay off the many times in the future, as you don't need to reinvent a solution for every problem yourself.