Richard Roose – Principal Consultant, Sydney
What makes a Modern Data & Analytics (D&A) platform? Well, it turns out that much of what defined a traditional Data Warehouse is still required for a Modern Data & Analytics Platform. We still extract, clean, merge and load data but modern times have resulted in more variety of data sources and formats like web api’s, json and unstructured data. The velocity of data and the turnaround time for insights has also changed, narrowing the margins between ingestion and showing results.
These “additional capabilities” that a few years ago were out of reach for most organisations are becoming more prevalent each day.
Below is an example of a Modern Data and Analytics Platform high-level architecture.
NOTE: The various toolsets mentioned in the diagram are “indicative/representative” and should not be taken as “recommended”. A proper toolset selection process should be undertaken to determine the “best fit” for your organisation.
Key to making these capabilities available is the cost/performance/scalability/elasticity of Cloud offerings. Whilst the traditional workloads of ERP, CRM and other systems of record are relatively predictable and easy to size, the ever-changing demands of today’s Data & Analytics workloads makes an on-premise solution expensive and inefficient.
Modern data and analytics – keys to success
Key #1 – “don’t buy the hype”, get over the on-premise security hype and incorporate the cloud for D&A workloads. If we can bank on-line, then surely we can do D&A in those same secure cloud environments.
Key #2 – “don’t boil the ocean”, D&A is an iterative process so start small and get some high visibility use cases into the hands of Execs or the field via mobile or new insights via some discovery/predictive/prescriptive analytics. Low cost/high visibility minimal viable products increases your chances of continued funding. The cloud is particularly well-suited to this iterative style as you just “spin up” new services as required. You don’t need to migrate everything in one go.
Key #3 – “a single copy of the truth”, don’t try and be the “single source of truth” because that leads to arguments with the people responsible for systems of record (e.g. ERP, CRM, etc). You want 1 copy that everybody in the D&A world can re-use so that results are consistent or differences are explainable.
Key #4 – “avoid low-level coding”, where possible use drag & drop tools and automate as much data preparation as you can. Spend the time and make your Data Ingestion/Transformation/Quality processes “bullet proof” and easily supported. Chasing errors through often undocumented 3GL code and/or complex integration patterns can take up a lot of time and leads to user distrust in the data. Avoid batch for data ingestion where possible as these are often the ones with complex integration patterns (e.g. export -> ftp -> upload) with multiple breakpoints. When talking data ingestion, if you have a CDC option then use that over batch.
Key #5 – “don’t govern everything”, governance of Business Metadata is critical to good D&A results and general usage/trust by the masses (e.g. known questions and known data like financial/KPI reporting). Onboarding new data sets, data wrangling, data science, data profiling, self-serve and general “discovery” should be encouraged/supported through light weight governance, if any. A sandpit with a “delete after x days” rule should suffice as anything of value will be put up as a candidate for automation (see Key #4).
Key #6 – “make your DQ /Business Metadata visible”, displaying the completeness/freshness of your data gives your users an added level of trust in using a given data set/dashboard. Hover tool-tips displaying “certified” data definitions is another way to promote self-serve and avoid unnecessary support calls.
Key #7 – “be an expert at data storytelling”, time to insight is everything in today’s world. If people have to spend time digesting your visualisations, then that’s wasted time and possibly even wasted opportunity. Make sure you understand the human brain and how it processes information vs. just displaying “data gaga”.
Key #8 – “fit for purpose”, modern data & analytics brings to the table a set of concepts like Real Time Streaming and Data Lakes. These components can add great value to your existing D&A environment if used correctly with previously unobtainable insights.
Contact us to find out more on how you can successfully move to a Modern Data Platform.
