Task Chaining in Qlik Sense Enterprise SaaS

December 16, 2020

Today, QSE SaaS doesn't do task chaining. Instead you have two options:

Use a scheduled reload task for every app, and try to make sure the start time for task 2 will always be later than task 1
Use Qlik SaaS REST APIs to trigger the reload

This post covers a simple approach (i.e. Qlik script only) for triggering reloads of downstream apps on completion of an app reload in a way that doesn't require much maintenance or any external systems or integrations.

With the APIs, we can reload as frequently as we wish - to some disastrous results. Be careful.

For those who haven't looked into the QCS APIs in the past, there's a lot of information on the Qlik.dev site; and specifically this video on the topic of task chaining with Qlik-CLI.

The theory

Using the reloads API, we need just one thing to trigger a reload - an app ID. So, all we need to do after a reload is make a call to this API to trigger the next app. This means we only need to put a scheduled reload task on the first app in the chain, with every following app triggered by the proceeding application.

To make this work, we need:

A REST connection to the /reloads endpoint
A data file connection to store the logs in
A parent application (app 1 - this will be triggered by QSE SaaS and a scheduled task)
A child application (app 2 - this will be triggered by the load script of app 1)

The reason for doing this in the load script of a normal app (and not as a separate "task manager" app) is because scheduling and running a series of task manager apps is tedious, and setting up a large number of these apps may not be supported by Qlik in the QCS environment.

Note: Fair warning, the reloads API is experimental and may change.

If your reloads suddenly stop working, it could be the API.

The REST connection

We are going to be using the /api/v1/reloads endpoint and hitting it with a POST request. To set this connection up, you'll need the following details:

URL: https://..qlikcloud.com/api/v1/reloads
Type: POST
Body: {"appId":""}
Header: Authorization: Bearer

The subroutine

The inputs and outputs of this sub are listed in the script. You should put this as the last script to execute in your load script.

Sub sTriggerReload(sub_appID,sub_connAPI,sub_connLog)

	/* 
    
    This subroutine triggers the reload of a QCS application (directly, not using scheduled tasks)
    
    INPUTS:
    * sub_appID = the GUID for the app to be reloaded
    * sub_connAPI = a REST data connection that can access the tenant APIs with appropriate privileges
    * sub_connLog = a folder data connection for storing reload trigger log files (these will be stored as "ReloadLog_<AppID>_<ReloadID>.qvd")
    
    OUTPUTS:
    * Send a POST message the task API to trigger the relevant app reload
    * Store a log file to record the reload trigger to assist with finding this event in audit logs if needed
    
    REST CONNECTION CONFIG
    * URL: https://<tenantname>.<region>.qlikcloud.com/api/v1/reloads
    * Type: POST
    * Body: {"appId":"<valid app GUID>"}
    * Header: Authorization: Bearer <API key>
    
    */
    
    // Connect to the REST connection
    LIB CONNECT TO '$(sub_connAPI)';
    
    LET sub_QueryBody = '{""appId"":""$(sub_appID)""}';

    // Collect data from the response for logging
    // Configure app ID for reload
    RestConnectorMasterTable:
    SQL SELECT 
        "id",
        "appId",
        "tenantId",
        "userId",
        "type",
        "status",
        "creationTime",
        "__KEY_root"
    FROM JSON (wrap on) "root" PK "__KEY_root"
    WITH CONNECTION (BODY "$(sub_QueryBody)");

    ReloadLog:
    LOAD DISTINCT	
    	[id] 			AS [Reload ID],
        [appId] 		AS [Reload App ID],
        [tenantId] 		AS [Reload Tenant ID],
        [userId] 		AS [Reload User ID],
        [type] 			AS [Reload Type],
        [status] 		AS [Reload Status],
        [creationTime] 		AS [Reload Creation Time],
        DocumentName()		AS [Reload Trigger App ID],
        DocumentTitle()		AS [Reload Trigger App Name]
    RESIDENT RestConnectorMasterTable
    WHERE NOT IsNull([__KEY_root]);
    
    // Set variables to produce log filenames
    LET sub_ReloadTime 	= Timestamp(Peek('Reload Creation Time',0),'YYYYMMDDhhmmss');
    LET sub_ReloadID 	= Peek('Reload ID',0);
    
    // Check to see if the reload request returned rows, and the variables carry data. If not, fail this reload
    If (NoOfRows('ReloadLog') <> 1) OR ('$(sub_ReloadTime)' = '') OR ('$(sub_ReloadID)' = '') THEN
    	// Fail with an error for the log
        Call Error('An unexpected number of rows was returned by the reloads API, or invalid data was found.');
    END IF;
    
    TRACE >>> Returned reload $(sub_ReloadID) at $(sub_ReloadTime);
    
    // Store logs and clear model
    STORE ReloadLog INTO [lib://$(sub_connLog)/ReloadLog_$(sub_appID)_$(sub_ReloadID)_$(sub_ReloadTime).qvd] (qvd);
    DROP TABLE ReloadLog;
    DROP TABLE RestConnectorMasterTable;
    
End Sub;

// Call - pass in the app ID, the REST connection name, the folder connection name
Call sTriggerReload('ab77b40d-4a30-46d9-9d2b-2943c6b82902','<rest connection name>','DataFiles');

The latest version of this script can be found in this GitHub repo.

What's missing from this?

If you want to take this forward, you'll want a few things:

Error handling - for when you get a weird response from the API
An app to blend the log files this produces with the reload event logs to track duration and errors
To store this script as a text file and load it as an include so it doesn't exist in a million different apps
To consider whether you want to trigger reloads by app ID or by app and space name