QVD read/write performance comparison in Qlik Sense Enterprise with QVD and QVF encryption

Following on my previous post, I had a look at the performance impact of enabling QVD and QVF encryption in Qlik Sense.

In this test, I’m using Qlik Sense Enterprise November 2019 release on an Azure B4ms (4 vCPU, 16GB RAM) VM running Windows Server 2019. Qlik Sense encryption settings were left at default.

A sneak peak of the results for the largest data set tested

The questions

I’ll prepare a follow up post running through the questions and findings, this post summarises the test structure and high level findings.

The tests & source data

The data I’m loading is one of the freely available data sets on archive.org from StackExchange (in this case, a 618MB serverfault 7z archive).

Stack Exchange makes a huge amount of anonymized data available via Archive.org

Uncompressed, it’s 3.13GB, or 2.5GB for just the XML files I’m running tests against.

The three test subjects totalled around 2.5GB uncompressed

Each of the tests below was run a minimum of three times, on XML based data sets of three different sizes (PostHistory, Posts and Badges – in order of decreasing size).

The following tests were run:

  1. Load from XML (no transformation)
  2. Store loaded XML data into QVD (no transformation)
  3. Load from QVD using optimised load
  4. Store loaded QVD data into a second QVD (no transformation)
  5. Load from QVD using unoptimised load and perform transformations (using a wide range of functions)
  6. Store transformed QVD data into a third QVD
  7. Load from QVD using unoptimised load and perform transformation/where reduction (only two functions)
  8. Store transformed QVD data into a fourth QVD
  9. Load from QVD using optimised load, then resident to perform matching transformation to #5
  10. Store transformed QVD data into a fifth QVD

The QVF file and load scripts to run these tests are available on GitHub.

Test results

The results (when assessing ONLY PostHistory – the largest input file), with the exception of tests 6, 8 and 10 (all store operations on data originally loaded from a QVD), show that enabling encryption for QVDs increases load time, and enabling both QVD and QVF encryption increases this further.

No surprises there.

Average test duration grouped by test and test mode

I’ll look into this in more depth in a follow up post.

Observation on QVD file size

There was no noticeable increase in QVD file size following encryption – see screenshots of before and after below.

File sizes without QVD encryption (excluding the 0 value for the first file as this was being written while the screenshot was taken)
File sizes with encryption – at most a few KB out

Considerations for next time

  • Instead of using a burstable instance (B4ms), I should have used a general instance such as a DS3 to ensure a baseline level of performance
  • The server size was likely too large for the smallest data set I used, meaning that operations completed too quickly for any variation to be meaningful, while Posts and PostHistory were more suitable
  • This time, I used Azure files for the primary read/write location. While we should assume performance remains consistent over time, testing with a provisioned disk attached to the VM would be a better test to remove any potential variability
  • Services were not restarted between every test, only between test modes (i.e. encrypted, unencrypted) – it would be a better control to begin all tests following a restart of at least the engine service
  • The initial test created the QVD files which were then overwritten by all following tests – ideally these would have been deleted between tests (incidentally, no obvious variation appeared between tests 1 and 2)
  • There was no system monitoring set up – this would provide insights as to CPU and IO utilisation throughout and would be a useful addition to the time statistics

Qlik Sense QVD / QVF Encryption (Qlik Sense Enterprise on Windows)

From Qlik Sense Enterprise on Windows September 2019 (the option became configurable via the QMC in November 2019), users have the option of enabling one or both of QVD and QVF encryption – adding at-rest protection from people who shouldn’t have access to your valuable files. This is enabled server-wide and applies to all content on the environment.

The big issue that I see is that any security office is going to require that any and all encryption is enabled – without understanding the issues this could cause.

Here’s the new DATA ENCRYPTION section under Service cluster in the QMC
Continue reading “Qlik Sense QVD / QVF Encryption (Qlik Sense Enterprise on Windows)”

AWS Lightsail snapshots – now automatic for Linux

As an update to my previous post on using lambda to automate snaphots, I’ve finally logged back into the Lightsail console and seen they have added a new feature – automatic snapshots.

There’s now an automatic snapshot section – note that the manual snapshots above are the 14 days dutifully maintained by the lambda script

It looks like the feature was added in October, based on the help site.

You’re able to change the time the snapshot is taken, but not the amount of history maintained

I won’t be switching from the lambda function as I require 14 days on Lightsail boxes, but if you’re happy with the following constraints, then it’s a great option:

  • Pre-set to 7 days of snapshot retention (no more, no less)
  • Must be set to run at a specific time (no triggers)
  • Available on Linux boxes only (no Windows as yet)

Remove (destroy) all variables in a Qlik Sense application

In QlikView, you had the ability to select multiple variables and delete them en-mass. In Qlik Sense, the UI permits you to undertake this activity one-by-one, and each takes two or three clicks.

This bookmarklet (Chrome only this time, not tried elsewhere) removes ALL variables from a Qlik Sense application by cheating the APIs used by the Qlik Sense client.

Demo showing addition of bookmark and destruction of variables in a Qlik Sense app
Continue reading “Remove (destroy) all variables in a Qlik Sense application”

Generating consistent Qlik Sense app icons

Application icons are prominently displayed throughout the Qlik Sense hub, and they are usually either the default blue, or some horrendously stretched icon that has no business being on an enterprise system.

This simple tool (packaged as an extension and accessible as a mashup) helps users generate consistent, appropriate app icons for use in their apps.

Without peering at the text, the default icons are rather generic
Continue reading “Generating consistent Qlik Sense app icons”

Get Qlik Sense Object IDs quickly

If you’re doing anything but vanilla Qlik Sense development, it’s likely you’ll need to get to the object IDs at some point. You can do this by appending /options/developer to the end of the page URL and clicking on each object in turn, or using dev tools – but that’s slow.

This bookmarklet makes getting the object IDs fast in Chrome (and, begrudgingly, IE11).

Animated demo showing adding of bookmarklet to chrome and displaying of object IDs
Continue reading “Get Qlik Sense Object IDs quickly”

Adding CentOS Web Panel to a fresh CentOS 7 box

Revisiting an earlier post on configuring a LAMP server, and on a recent post on installing Webmin, this time I’m dropping CentOS Web Panel (CWP) onto a new CentOS 7 box.

So, on a fresh CentOS 7 1901-01 box on AWS, let’s begin.

# Set the hostname correctly before we start off, otherwise CWP will pick up the wrong name
hostname sub.host.com
# Update existing packages
sudo yum -y update
# Install wget and nano (as I like nano and wget is needed for the CWP install)
sudo yum -y install wget nano
# Reboot the box to sort out any updates requiring reboot
sudo reboot
Continue reading “Adding CentOS Web Panel to a fresh CentOS 7 box”

Adding Webmin to a fresh CentOS 7 Box

As a quick refresh from an earlier post, I’m going to use a new CentOS image with Webmin as the control panel of choice.

Happily, the only thing that’s changed in the documentation is the way the gpg key is imported – documentation is here: https://doxfer.webmin.com/Webmin/Installation

So, on a fresh CentOS 7 1901-01 box on AWS, let’s begin.

# Update all base packages
sudo yum -y update
# Install nano (my preferred text editor)
sudo yum -y install nano
# Edit the /etc/yum.repos.d/webmin.repo file
sudo nano /etc/yum.repos.d/webmin.repo
Continue reading “Adding Webmin to a fresh CentOS 7 Box”

An example of embedding Qlik Sense (mashup and APIs)

I’m often asked how to create mashups with Qlik Sense, and I strongly believe that it’s both easy and intuitive to leverage Qlik Sense APIs to build mashups…when you understand the options available to you.

To help new developers, I’ve put together a basic mashup using the Material Design Lite template. This example connects to a provided app and demonstrates several different ways of embedding Qlik Sense into a HTML site using just a little Javascript.

The mashup has four pages, one based on the default template and the other three focused on content
Continue reading “An example of embedding Qlik Sense (mashup and APIs)”