A QUIPU Dataflow can be composed visually on the Diagram. A Diagram shows Models (circle symbol), Data Containers (database or file symbols) or Modules (rounded square symbols). The Toolbox in the panel on the right shows all available Modules, whereas the Explorer on the left gives an overview of all configured components (Models, Modules, Data Containers) that exist on all Diagrams.
Flexibility by design
In QUIPU you define flows to interact with datamodels. Each interaction is managed via a variety of modules. The process usually starts with importing model metadata Modules are the flexible components (represented by the blue squares on the diagram) that each define and enable a specific function. All modules are fully configurable.
JDBC model generator
The JDBC Model Generator Module uses a JDBC connector to retrieve model meta-data and store it in the QUIPU repository, as a Data Model of type ‘Source’. The advantage of using this module to retrieve the meta-data is the abundant availability of drivers for all sorts of databases, and even structured files. It thus provides a very powerful way to bring source models into QUIPU.
Model delta detector
Compares the last two versions of a model and reports the differences. The module reports deltas for entities, attributes, constraints and relations. For each model element, the following deltas are reported. Which model elements are added. Which model elements are removed. Which model elements have properties that have changed.
Generates code based on a model and a template. This code can be written to a data container (database or file), specified by the user. QUIPU is able to generate code directly into a target database. For SQL Server targets, QUIPU offers a standard solution. To let QUIPU generate code into tables in other target databases several queries can be specified.
OData model generator
The OData model generator module reads a single schema from an OData $metadata XML file and generates a data model of type 'Source'. Currently, relations are not supported. The OData model generator module reads and stores entities metadata, attribuites metadata and constraints metadata.
HDA model generator
Generates a Data Model of type HDA (Historic Data Archive) based on a Data Model of type Staging
Staging model generator
Generates a Data Model of type Staging based on a Data Model of type Source.
The transformation generator module transforms an input model by filtering, adding, converting or renaming of objects. A transformation module contains three optional parts: 1 - Pre filters. These filters are used to filter out source model objects. All objects can be filtered using include and exclude logic. 2 - Action groups. Action groups contain actions and possibly a filter. 3 - Post filters. These filters are used to filter out (possibly created) target model objects.
Data set model generator
Generates data set entities that are specified by QUIPU users. The entities in the resulting model represent data sets that are constructed by joining source entities based on child-parent relations defined in the source model.
A Data Container is a physical instance that can either be input for Import Modules or output for Export Modules. Data containers can be databases or files.
Other distinguishable features of the QUIPU framework
A powerful feature of QUIPU is the option to execute most of its functions directly via the REST API. This opens the possibility to script QUIPU actions and expand the QUIPU capabilities beyond what we can offer. A very useful scenario is where you build your solution by configuring modules and stringing them together using the powerful graphical user interface of QUIPU, test it and then create a script to automatically execute all the steps specified on the diagram for future runs when changes come to your environment. Automating as much manual work as possible, since changes will happen.
QUIPU functions are exposed in web services that can be accessed via a REST API QUIPU uses the standard web REST API mechanism to expose its services. The QUIPU front end uses this API to offer you a user friendly application to use the QUIPU system, but it is this same mechanism that allows scripts to address QUIPU services directly. We will make a detailed description of the API available in the near future. For now, use the built-in script generator that will create a script for any diagram you have drawn as a quick start for your own scripts.
QUIPU uses JSON for the configuration of its modules. JSON is a de-facto standard on the internet for exchange of data between web services. The Editor tabs in several Property panels in QUIPU use a JSON Editor.