Automating AWS Lightsail backups using snapshots and Lambda

Some of the most glaring omissions from Lightsail are scheduled tasks or triggers – which would provide the ability to automate backups. Competitors in this space like DigitalOcean are all set, as they offer a backup option, whereas for AWS I’m assuming they hope you’ll shift over to EC2 as fast as possible to get the extra bells and whistles.

Of course you can manually create snapshots – just log in and hit the button. It’s just the scheduling that’s missing.

I have one Lightsail server that’s been running for 6 months now, and it’s all been rosy. Except – I had been using a combination of first AWS-CLI automated backups (which wasn’t ideal as it needed a machine to run them), and then some GUI automation via Skeddly. However – while Skeddly works just fine, I’d rather DIY this problem using Lambda and keep everything in cloud native functions.

Continue reading “Automating AWS Lightsail backups using snapshots and Lambda”

Using the Cloudflare API to provide free Dynamic DNS with Windows and Powershell

This post details my switch over to using Powershell and Cloudflare to update a DNS record to a server’s current IP. This effectively emulates dyndns for this host – except it’s free.

There are a load of other options out there, which even include some simple-but-quite-clunky apps for domain registrars like NameCheap; but installing third party software is not the route I want to take.

I previously had my target domain (let’s call it targetdomain.com) hosted on a Linux box, and used SSH to update the DNS settings via a Windows server. This worked well for three years without a blip – but was clunky. I was using a scheduled task to start a bat file, which then ran Putty to run the shell script…to update a config on a server which was only hosting the domain to serve this purpose.

Although the scheduler>bat>shell tasks have been running well for years, it’s time to simplify!

I’ve been using Cloudflare for years, and set aside time to write a script to use their service for this purpose. As it turns out, people have done this for years – so I’ve taken one off the shelf.

Continue reading “Using the Cloudflare API to provide free Dynamic DNS with Windows and Powershell”

Migrating a private repository from Bitbucket to GitHub with Git

As GitHub private repositories have just become free, I’m jumping on the bandwagon and shipping over a few of the repos I have on Bitbucket to reduce the number of providers I store code with.

The end result – a private GitHub repository with all the metadata from the old Bitbucket repository – note we have maintained the last commit from 10 months ago

The option below uses the shell to migrate the repo, but you can also use the GitHub importer if you prefer an automated solution (you’ll just have to wait while it does it).

Continue reading “Migrating a private repository from Bitbucket to GitHub with Git”

Deleting AWS Glacier Vaults via AWS CLI using a Lightsail Instance

Amazon Web Services (AWS) offers some very affordable archive storage via it’s S3 Glacier service. I’ve used this on a backup account in the past to store archives, and have decided it’s time to clear down this account (oh, and save $0.32 a month in doing so).

The main challenge with doing this, is that unlike S3, S3 Glacier (objects stored directly there rather than using the Glacier storage tier within S3) objects can only be deleted via the AWS CLI. And to delete a Glacier Vault, you’ve got to delete all of the objects.

This account has some wild spending. $4.90 a month!

In this post I’ll spin up a Lightsail box and wipe out the pesky Glacier objects through the AWS CLI. This doesn’t require any changes on your local PC, but will require some patience.

Continue reading “Deleting AWS Glacier Vaults via AWS CLI using a Lightsail Instance”

A quick performance comparison with Qlik Sense – AWS EC2 vs Azure Virtual Machines

Previously, I tested the performance of a load script while using RecNo() and RowNo() functions. This conveniently gave me a script which consumes up to 25GB of RAM, along with considerable CPU power.

So, what about testing it on two cloud boxes? I’ve chosen a machine from both AWS and Azure, loaded them with Qlik Sense September 2018 and run the load script.

Total Test Duration by Host

The summary: The AWS box was approx 8% faster than the Azure box.

Continue reading “A quick performance comparison with Qlik Sense – AWS EC2 vs Azure Virtual Machines”

Comparing Autonumber, Autonumberhash128, Autonumberhash256, Hash128, Hash160 and Hash256 outputs in Qlik Sense and QlikView

There’s often a discussion about what each of these autonumber/hash functions does in Qlik. We commonly see these functions used for creating integer key fields, anonymising data (for example, names in demo apps), and maintaining long string fields for later comparison (as the hashes are shorter than the strings they replace).

Sample outputs from the random number generator, with all the functions present

To do this, I’m using the script below. I’m also keen to show outputs from QlikView vs Qlik Sense, and results of running the same script on another machine.

My observations are the following:
AutoNumber/AutoNumberHash128/256 – different output per load as the value is typically based on the load order of the source data
Hash128/160/256 – the same output, across every load. Stays the same between Qlik Sense and QlikView, and also between different machines

Continue reading “Comparing Autonumber, Autonumberhash128, Autonumberhash256, Hash128, Hash160 and Hash256 outputs in Qlik Sense and QlikView”

Qlik load performance with RecNo() and RowNo()

Using RecNo() or RowNo() will impart a performance impact on your load script. I discussed these functions in a previous post where I looked at the output of RecNo vs RowNo.

I recently spotted an unexpected slow-down in a load script, which was caused by using one of these functions. In summary:
– Using RowNo() in a simple load script is considerably slower than RecNo()
– If you must use RecNo(), it may be faster to do this in a direct load
– If you must use RowNo(), it may be faster to do this in a resident load

Example script for one of the tests – load data from disk and add the RowNo

 

Continue reading “Qlik load performance with RecNo() and RowNo()”

VirtualBox on Windows 10: This 64-bit application couldn’t load because your PC doesn’t have a 64-bit processor

Sometimes VirtualBox doesn’t behave when it imports Virtual Machines (appliances).

I exported a Windows 10 VM from one machine to another, imported and ran it to receive a recovery message on launch.

My W10 VM isn’t happy

Continue reading “VirtualBox on Windows 10: This 64-bit application couldn’t load because your PC doesn’t have a 64-bit processor”

Compacting Windows 10 VMs and shrinking their VDI/VMDK disk images on VirtualBox host

If you’ve got a couple of VMs, then trimming them occasionally will help with storage management on the host. Disk files (like VDI and VMDK images) will grow over time if the VM was configured to expand the disk as needed – but they will never shrink on their own.

This was tested on a Windows 10 VM managed by VirtualBox, the starting file size was over 40GB, as it had been used for various roles and grown over time.

Our test VM - starting size
Our test VM – starting size

Continue reading “Compacting Windows 10 VMs and shrinking their VDI/VMDK disk images on VirtualBox host”

Qlik Counter Functions and their outputs – RecNo() and RowNo()

In this post I explore the outputs of RecNo() and RowNo() to demonstrate the difference visually.

These two fields are often used interchangeably, but they provide different output. In summary:
– RecNo() is a record number based on source table(s)
– RowNo() is a record number based on the resulting table

As a result, RowNo will always provide a unique integer per row in an output table, while RecNo is only guaranteed to be unique when a single source is loaded (RecNo is based on each single source table, interpreted individually rather than collectively).

A snapshot of the test output

Continue reading “Qlik Counter Functions and their outputs – RecNo() and RowNo()”