Risk matrix for automating data solutions

Rationale

A major driver behind the development of QUIPU4 is the aim of helping organisations to reduce enterprise risk when developing data solutions. Access to well-structured data is becoming more important with each passing day. As a result, underlying data solutions can increasingly be classified as critical applications. Data solutions people can trust. That are easy to modify and expand. Solutions that can be easily transferred and, moreover, that can be properly monitored and audited.

Unfortunately, today, too many solutions are still for the most part developed manually by people in changing project teams. People who each have their own beliefs about what the best architecture or technology is, which then becomes visible like a patchwork quilt in the realised solution. In addition, the documentation is often inadequate or missing altogether. The result is that the solutions developed only have a limited lifespan. The business case for replacing earlier developed, suboptimal solutions is often difficult, which means that legacy systems often continue to exist for too long. In the long run, these systems are kept alive in the intensive care unit of IT departments, consuming up precious resources. Looking back, what we see is a graveyard full of good intentions. The costs of further development are no longer in proportion to the defined business cases.

Model-to-model transformation

QUIPU4 is a platform that supports model-to-model transformation. This is how the product sets itself apart from other commonly accepted ETL tools and other data warehouse automation tools. Although such tools usually work well as far as the transformation and manipulation of data at the entity or attribute level is concerned, they lack the support needed to define and develop transformations at the model level. It is at this higher level that patterns can be found and applied, and therefore can be generated. While at the same time, exceptions to these patterns can still be specified at the entity or attribute level. According to QOSQO, this results in a much more efficient way of developing new data solutions quickly.

Risk matrix

The team behind QUIPU4 has a vision. An ultimate goal. We have tried to incorporate this into our matrix, set down below. Our aim is for the matrix to serve as a guideline for the further development of the product.

The vertical axis shows the various aspects that are characteristic of data-solution development. Things like data models, load routines, transformations (business rules), process control (load process), metadata use and, finally, the way in which the new developments will be rolled out.

The horizontal axis shows the degree of growth in maturity. A shift away from an approach that is high risk due to things like manually designing data models and manually coding solutions, not using metadata intelligently and tool dependency, to the absence of risk because data models and load code are generated. Development is then completely data driven based on a single central repository. In such a development process deployment is fully automated. An adaptive solution that itself detects changes in the environment and can apply and roll them out automatically.

Currently, QUIPU4 mainly supports the 3rd column: metadata-driven development. But we have already taken the first steps towards supporting change handling. For example, in QUIPU4, a module is already available that enables model changes to be detected and reported. The logical next step is the automatic implementation of these changes in the data solutions generated, as the first step towards adaptive solutions. Artificial intelligence and machine learning technologies will be essential to translating the impact of changes in the environment into sensible adjustments to the data chain.