Auxiliary timelines and partitioned loading
January 8, 2020Lineage visualisation
June 8, 2020
Risk matrix for automating data solutions
Rationale
A major driver behind the development of QUIPU4 is the aim of helping organisations to reduce
enterprise risk when developing data solutions. Access to well-structured data is becoming more
important with each passing day. As a result, underlying data solutions can increasingly be classified
as critical applications. Data solutions people can trust. That are easy to modify and expand.
Solutions that can be easily transferred and, moreover, that can be properly monitored and audited.
Unfortunately, today, too many solutions are still for the most part developed manually by people in
changing project teams. People who each have their own beliefs about what the best architecture or
technology is, which then becomes visible like a patchwork quilt in the realised solution. In addition,
the documentation is often inadequate or missing altogether. The result is that the solutions
developed only have a limited lifespan. The business case for replacing earlier developed, suboptimal solutions is often difficult, which means that legacy systems often continue to exist for too
long. In the long run, these systems are kept alive in the intensive care unit of IT departments,
consuming up precious resources. Looking back, what we see is a graveyard full of good intentions.
The costs of further development are no longer in proportion to the defined business cases.
Model-to-model transformation
QUIPU4 is a platform that supports model-to-model transformation. This is how the product sets
itself apart from other commonly accepted ETL tools and other data warehouse automation tools.
Although such tools usually work well as far as the transformation and manipulation of data at the
entity or attribute level is concerned, they lack the support needed to define and develop
transformations at the model level. It is at this higher level that patterns can be found and applied,
and therefore can be generated. While at the same time, exceptions to these patterns can still be
specified at the entity or attribute level. According to QOSQO, this results in a much more efficient
way of developing new data solutions quickly.
Risk matrix
The team behind QUIPU4 has a vision. An ultimate goal. We have tried to incorporate this into our
matrix, set down below. Our aim is for the matrix to serve as a guideline for the further development
of the product.
Figure 1. Risk matrix for automating data solutions
The vertical axis shows the various aspects that are characteristic of data-solution development.
Things like data models, load routines, transformations (business rules), process control (load
process), metadata use and, finally, the way in which the new developments will be rolled out.
The horizontal axis shows the degree of growth in maturity. A shift away from an approach that is
high risk due to things like manually designing data models and manually coding solutions, not using
metadata intelligently and tool dependency, to the absence of risk because data models and load
code are generated. Development is then completely data driven based on a single central
repository. In such a development process deployment is fully automated. An adaptive solution that
itself detects changes in the environment and can apply and roll them out automatically.
Currently, QUIPU4 mainly supports the 3rd column: metadata-driven development. But we have
already taken the first steps towards supporting change handling.
For example, in QUIPU4, a module is already available that enables model changes to be detected
and reported. The logical next step is the automatic implementation of these changes in the data
solutions generated, as the first step towards adaptive solutions. Artificial intelligence and machine
learning technologies will be essential to translating the impact of changes in the environment into
sensible adjustments to the data chain.