v4.4.0 Overview

Introduction to QUIPU

QUIPU: a next generation data management automation tool

Following up on its successful predecessor we are happy to announce the release of QUIPU. We’re taking things a step further by introducing the next level in data management automation using patterns as guiding principle. Making data warehouse automation, data migration, big data applications and similar projects much faster and easier.
We strongly believe in leveraging the knowledge of data engineers and data architects. By automating the straightforward stuff and thus killing tedious, boring jobs we bring back the fun. Allowing them to focus on the complex tasks ahead when data transformation is concerned and to spend more time satisfying business needs and truly participate in creating value for the company.
The previous version of QUIPU already offered software and services that generate data warehouse solutions based on data vault. QUIPU takes generation to the next level and will generate more generic data management solutions based on standard patterns wherever that is possible. Including data warehouses, data migrations and big data applications.
QUIPU will also support specification of exceptions as well where standard patterns do not suffice. This way, regeneration will apply the same patterns again but will also re-apply the exceptions, resulting in the expected outcome.
If you require the more complex data vault model for your source or business data warehouse layer we refer you to QUIPU version 3.4 that is still available and supported whilst we transfer functionality to QUIPU version 4.

Model to Model Transformation

QUIPU is primarily a model-to-model conversion tool

QUIPU is primarily a model-to-model conversion tool. And this is precisely where it distinguishes itself from existing ETL tools and other data warehouse automation tools. These tools do a good job in transforming and manipulating data on entity or attribute level but lack the support for defining and developing transformations on a model level. It is at this higher level that patterns can be found and applied, and thus can be generated.
This is a far more efficient way of developing data management and data warehousing solutions. QUIPU can for instance take an entire staging data model as input and generate a model that stores data historically.
QUIPU not only generates modified data models, but also keeps track of the mappings between source and target models. These mappings allow QUIPU to generate the ETL (SQL) code -or views, if that is possible and preferred- to move the data in the production environment.
It will generate all output using templates, so the output can be easily adjusted for specific database platforms or customer standards.

Modular Approach

If there is a pattern, we can automate it

QUIPU provides maximum flexibility in designing and automating whatever flow of architectural layers are required. The new web based graphical user interface provides a canvas where one can select the desired building blocks to achieve this. New building blocks required can be developed by us in a short period of time.
If there’s a pattern, we can automate it!
Some examples:
A user could decide to extract source data directly into a generated historical data archive (HDA) and from there straight into a couple of star schema modelled data marts;
Another flow could be including a multitude of source data vaults all feeding one single business data vault;
Or just a simple flow to offload data from a soon-to-be-replaced ERP system into an archive system;
Or …

Dealing with Exceptions

QUIPU will keep your manual changes between generation actions

The unique approach of QUIPU will save you tremendous amounts of time by applying patterns to transform models. But what about the exceptions to the pattern?
Well, you could easily modify the code produced by QUIPU manually to deal with those exceptions. But that would reduce the benefit of having QUIPU to a single run. Because QUIPU would overwrite all of your changes on the next run.
Of course you could save your changes in a separate file and compare the newly generated QUIPU code with your manual changed code to find and reapply your changes. Obviously that is a very tedious and error prone process.
QUIPU has automated this approach. QUIPU will always keep generated models (and mappings) apart from any changes you make (manually via the user interface or by uploading files). Every time you re-generate a model a new version of the generated model is stored and your previous modifications are re-applied automatically.
NOTE: It remains your responsibility to validate that the manual changes you made are still valid for the data model that has now changed due to the re-generation.
NOTE: At this point you have no direct access to the different versions of the generated models nor the manual changes you have made. Giving you access is high on our priority list and will assist you in managing your changing environment.

Access to QUIPU

Introduction

User management is only used to record some user preferences

User management is not a very critical aspect of the application at this point. The application is typically used by very few persons in a client environment: the architect or lead developer of the data warehouse environment.
For this reason we have not given user management a high priority at this stage.

Security Considerations

Security is initially not incorporated into the application, but instead relies on existing facilities in the server environment

Security is initially based on access control: the server components should be installed on a server that is only accessible to the intended users of the application.
Your IT support staff should make any changes necessary to the internal firewalls so the appropriate people have access. But you could also install QUIPU on a single windows PC or laptop as its operating requirements are modest.
The installation manual clearly states the ports the QUIPU server uses and explains how these can be changed.
Make sure you have authorisation to access at least your source systems via JDBC connectors for QUIPU to read the meta-data of your sources. If QUIPU has access to your target systems it can generate code to a database table for easy processing from a scheduler.

Scripting

Most operations available in the GUI can also be executed from a script

A powerful feature of QUIPU is the option to execute most of its functions directly via the REST API. This opens the possibility to script QUIPU actions and expand the QUIPU capabilities beyond what we can offer.