QVD read/write performance comparison in Qlik Sense Enterprise with QVD and QVF encryption

Following on my previous post, I had a look at the performance impact of enabling QVD and QVF encryption in Qlik Sense.

In this test, I’m using Qlik Sense Enterprise November 2019 release on an Azure B4ms (4 vCPU, 16GB RAM) VM running Windows Server 2019. Qlik Sense encryption settings were left at default.

A sneak peak of the results for the largest data set tested

The questions

I’ll prepare a follow up post running through the questions and findings, this post summarises the test structure and high level findings.

The tests & source data

The data I’m loading is one of the freely available data sets on archive.org from StackExchange (in this case, a 618MB serverfault 7z archive).

Stack Exchange makes a huge amount of anonymized data available via Archive.org

Uncompressed, it’s 3.13GB, or 2.5GB for just the XML files I’m running tests against.

The three test subjects totalled around 2.5GB uncompressed

Each of the tests below was run a minimum of three times, on XML based data sets of three different sizes (PostHistory, Posts and Badges – in order of decreasing size).

The following tests were run:

  1. Load from XML (no transformation)
  2. Store loaded XML data into QVD (no transformation)
  3. Load from QVD using optimised load
  4. Store loaded QVD data into a second QVD (no transformation)
  5. Load from QVD using unoptimised load and perform transformations (using a wide range of functions)
  6. Store transformed QVD data into a third QVD
  7. Load from QVD using unoptimised load and perform transformation/where reduction (only two functions)
  8. Store transformed QVD data into a fourth QVD
  9. Load from QVD using optimised load, then resident to perform matching transformation to #5
  10. Store transformed QVD data into a fifth QVD

The QVF file and load scripts to run these tests are available on GitHub.

Test results

The results (when assessing ONLY PostHistory – the largest input file), with the exception of tests 6, 8 and 10 (all store operations on data originally loaded from a QVD), show that enabling encryption for QVDs increases load time, and enabling both QVD and QVF encryption increases this further.

No surprises there.

Average test duration grouped by test and test mode

I’ll look into this in more depth in a follow up post.

Observation on QVD file size

There was no noticeable increase in QVD file size following encryption – see screenshots of before and after below.

File sizes without QVD encryption (excluding the 0 value for the first file as this was being written while the screenshot was taken)
File sizes with encryption – at most a few KB out

Considerations for next time

  • Instead of using a burstable instance (B4ms), I should have used a general instance such as a DS3 to ensure a baseline level of performance
  • The server size was likely too large for the smallest data set I used, meaning that operations completed too quickly for any variation to be meaningful, while Posts and PostHistory were more suitable
  • This time, I used Azure files for the primary read/write location. While we should assume performance remains consistent over time, testing with a provisioned disk attached to the VM would be a better test to remove any potential variability
  • Services were not restarted between every test, only between test modes (i.e. encrypted, unencrypted) – it would be a better control to begin all tests following a restart of at least the engine service
  • The initial test created the QVD files which were then overwritten by all following tests – ideally these would have been deleted between tests (incidentally, no obvious variation appeared between tests 1 and 2)
  • There was no system monitoring set up – this would provide insights as to CPU and IO utilisation throughout and would be a useful addition to the time statistics

Qlik Sense QVD / QVF Encryption (Qlik Sense Enterprise on Windows)

From Qlik Sense Enterprise on Windows September 2019 (the option became configurable via the QMC in November 2019), users have the option of enabling one or both of QVD and QVF encryption – adding at-rest protection from people who shouldn’t have access to your valuable files. This is enabled server-wide and applies to all content on the environment.

The big issue that I see is that any security office is going to require that any and all encryption is enabled – without understanding the issues this could cause.

Here’s the new DATA ENCRYPTION section under Service cluster in the QMC
Continue reading “Qlik Sense QVD / QVF Encryption (Qlik Sense Enterprise on Windows)”

Remove (destroy) all variables in a Qlik Sense application

In QlikView, you had the ability to select multiple variables and delete them en-mass. In Qlik Sense, the UI permits you to undertake this activity one-by-one, and each takes two or three clicks.

This bookmarklet (Chrome only this time, not tried elsewhere) removes ALL variables from a Qlik Sense application by cheating the APIs used by the Qlik Sense client.

Demo showing addition of bookmark and destruction of variables in a Qlik Sense app
Continue reading “Remove (destroy) all variables in a Qlik Sense application”

Generating consistent Qlik Sense app icons

Application icons are prominently displayed throughout the Qlik Sense hub, and they are usually either the default blue, or some horrendously stretched icon that has no business being on an enterprise system.

This simple tool (packaged as an extension and accessible as a mashup) helps users generate consistent, appropriate app icons for use in their apps.

Without peering at the text, the default icons are rather generic
Continue reading “Generating consistent Qlik Sense app icons”

Get Qlik Sense Object IDs quickly

If you’re doing anything but vanilla Qlik Sense development, it’s likely you’ll need to get to the object IDs at some point. You can do this by appending /options/developer to the end of the page URL and clicking on each object in turn, or using dev tools – but that’s slow.

This bookmarklet makes getting the object IDs fast in Chrome (and, begrudgingly, IE11).

Animated demo showing adding of bookmarklet to chrome and displaying of object IDs
Continue reading “Get Qlik Sense Object IDs quickly”

An example of embedding Qlik Sense (mashup and APIs)

I’m often asked how to create mashups with Qlik Sense, and I strongly believe that it’s both easy and intuitive to leverage Qlik Sense APIs to build mashups…when you understand the options available to you.

To help new developers, I’ve put together a basic mashup using the Material Design Lite template. This example connects to a provided app and demonstrates several different ways of embedding Qlik Sense into a HTML site using just a little Javascript.

The mashup has four pages, one based on the default template and the other three focused on content
Continue reading “An example of embedding Qlik Sense (mashup and APIs)”

Comparing Autonumber, Autonumberhash128, Autonumberhash256, Hash128, Hash160 and Hash256 outputs in Qlik Sense and QlikView

There’s often a discussion about what each of these autonumber/hash functions does in Qlik. We commonly see these functions used for creating integer key fields, anonymising data (for example, names in demo apps), and maintaining long string fields for later comparison (as the hashes are shorter than the strings they replace).

Sample outputs from the random number generator, with all the functions present

To do this, I’m using the script below. I’m also keen to show outputs from QlikView vs Qlik Sense, and results of running the same script on another machine.

My observations are the following:
AutoNumber/AutoNumberHash128/256 – different output per load as the value is typically based on the load order of the source data
Hash128/160/256 – the same output, across every load. Stays the same between Qlik Sense and QlikView, and also between different machines

Continue reading “Comparing Autonumber, Autonumberhash128, Autonumberhash256, Hash128, Hash160 and Hash256 outputs in Qlik Sense and QlikView”

Qlik load performance with RecNo() and RowNo()

Using RecNo() or RowNo() will impart a performance impact on your load script. I discussed these functions in a previous post where I looked at the output of RecNo vs RowNo.

I recently spotted an unexpected slow-down in a load script, which was caused by using one of these functions. In summary:
– Using RowNo() in a simple load script is considerably slower than RecNo()
– If you must use RecNo(), it may be faster to do this in a direct load
– If you must use RowNo(), it may be faster to do this in a resident load

Example script for one of the tests – load data from disk and add the RowNo

 

Continue reading “Qlik load performance with RecNo() and RowNo()”

Qlik Counter Functions and their outputs – RecNo() and RowNo()

In this post I explore the outputs of RecNo() and RowNo() to demonstrate the difference visually.

These two fields are often used interchangeably, but they provide different output. In summary:
– RecNo() is a record number based on source table(s)
– RowNo() is a record number based on the resulting table

As a result, RowNo will always provide a unique integer per row in an output table, while RecNo is only guaranteed to be unique when a single source is loaded (RecNo is based on each single source table, interpreted individually rather than collectively).

A snapshot of the test output

Continue reading “Qlik Counter Functions and their outputs – RecNo() and RowNo()”

Version sheets and tabs in the load script for Qlik Sense and QlikView

In the absence of a source control system like SVN or GIT, a quick and easy way of capturing changes in a app is to update a version control tab before passing the app back to other developers, or onto testers.

This is a very low-tech solution, but the format below works well in shared environments. The first two tabs of your application should be:
– Version (explain the app and list changes you’ve made)
– Config (set the configuration values like paths, dates, etc used in your application’s data load and UI)

Version and Config sheets will become a familiar sight
Version and Config sheets should become a familiar sight

Continue reading “Version sheets and tabs in the load script for Qlik Sense and QlikView”