Webinar: Serverless ETL Using ALTIS DLA and AWS

By Manan Patel, Senior Consultant – Altis Sydney

Last month, I presented a webinar that explained the Altis DLA and how it can be integrated with AWS services to create a Serverless ETL architecture. The architecture utilises AWS Step Functions, AWS Lambda Functions, AWS S3 and AWS Redshift to build an automated, metadata-driven data platform. This blog covers some of the content I presented, though a recording of the webinar is available on our YouTube channel:

Altis and AWS

Altis has been an AWS partner since 2013 and established a specific practice in 2016. In 2018 we became an AWS Cloud Warrior and in early 2021, Altis achieved the AWS Data and Analytics Competency. Altis has over 40 AWS customers across Australia and New Zealand, supported by over 20 accredited and certified consultants.

What is DLA?  

The Altis Data Load Accelerator (DLA) is a tool used to accelerate the development of an ETL process. The DLA simplifies the data loading process using pre-set code and guided patterns. The DLA is driven by metadata and is split into two sections:

Extract – Data from source systems is brought into the staging layer of the data platform

Load – Data is transformed and loaded from the staging layer into the presentation layers of the data platform

In this webinar, I explained the DLA + Serverless ETL architecture in more detail, and discussed some of the key benefits of using DLA and AWS:

1. ETL is Centralised to one Platform
   • All services and components reside inside the AWS Cloud
   • Makes the data platform easier to maintain in a centralised location
2. Simplified ETL process
   • AWS Step Functions and AWS Lambda functions are generic and reusable
   • DLA in Redshift is flexible and easy to use
3. Serverless ETL orchestration
   • No provisioning or maintenance of servers
   • Pay only for what you use

Case Study

I also present in the webinar a case study that explains a recent project Altis completed using this architecture. The project involved implementing the new architecture to replace an existing ETL process, which relied on a third-party ETL tool in addition to AWS Redshift and AWS EC2 Servers. The Altis team completed the following activities in this project:

• Implementation of AWS Step Functions, AWS Lambda functions and DLA code in Redshift
• Analysis and migration of old data pipelines into the Redshift DLA
• Integration of 2 new data sources into the Data Warehouse using the new process

The main outcomes for the client were:

1. Data in AWS Redshift now loaded using new Serverless ETL and Altis DLA with no interruption to downstream processes
2. 2 New data sources were successfully delivered as part of the migration
3. 20+ old data pipelines decommissioned and streamlined into 3 new master pipelines
4. AWS EC2 Servers used by old ETL process decommissioned

Do get in touch with us to learn how we can implement a data platform for you. You can also talk to us about leveraging AWS POC and exploratory engagement funding for Redshift and Serverless architectures.