by Brook Queree, Senior Consultant – Altis QLD
So your organisation has moved its data into the cloud, now what?
As businesses realise the benefits behind cloud data platforms such as Snowflake, more and more use cases discovered that would never have been possible before. This is the power of the cloud, adding accessibility and scalability to vast amounts of data.
However, without careful management, it’s easy to allow costs to rise as your data platform grows.
Follow along the Altis webinar on a dive into how to make the most of snowflake while striving towards more data, at a lower latency, for less money.
Below is a small highlight from our webinar
Due to the scaling properties of the snowflake MPP architecture, it is often possible to increase warehouse-size on fixed-workload tasks with minimal increase in cost. This is because while the cost rate doubles with each size, so do the compute resource available to process your data.
What ends up happening is that even though you are paying twice the rate (credits/second) for a larger instance, your query will complete in roughly half as much time.
We can see this effect below when we chart out both the execution time and total cost in credits for an example data transformation query over 1.5 billion rows.
In this example, moving from the extra-small instance to the extra-large instance gives a decrease in execution time from ~670 seconds down to ~50 seconds.
However, when we want to analyse total cost, it’s clear to see the extra-small instance requires a much greater up-time. This results in a total cost on the extra-small instance of 0.19 credits. When looking instead at the extra-large instance, it shuts down much earlier. This results in a total of 0.23 credits to run the same workload.
What this means overall is that for this example workload, you could decrease your data latency from ~12 minutes to below 1 minute. This substantial latency decrease comes while only paying 1.2x the cost for total compute usage. The great thing about this scaling is that it also comes with near-zero engineering effort involved, as altering a warehouse’s size is a simple button click away.
Tune in to the full webinar to get more insights into managing costs on continuous data workloads, as well as a deep dive into the snowflake query profiler as we examine how it can be used to optimise expensive queries in your snowflake account.
Interested in watching more of our past webinars? View our previous Free Data & Analytics webinar recordings here.
