AI without hallucination: A future built on purpose, not hype

We are a couple of years into our latest hype cycle, Agentic Artificial Intelligence (AI). AI’s potential has increased from the previous days of Machine Learning, but organisations are not seeing the expected Return On Investment (ROI).

Industry statistics from Adapt Research, covering the Australia and New Zealand region, shows that:  

• 80% of AI deployments are still finding their footing at scale 
• 72% of Chief Data Analytic Officers (CDAOs) are working to align initial investments with Return on Investment (ROI) 

A key factor impacting these initiatives is the data used in AI projects. ADAPT’s CDAO and CIO Edge Survey found that while Data and AI are core to an organisation’s strategy, less than 24% of surveyed leaders believe their data is “AI-ready”. Despite many organisations already having invested in modern data platforms and talented teams, 68% still report only minimal data integration across sources.

When AI systems are trained or prompted using outdated, siloed, or poorly tagged information, they can hallucinate and make erroneous predictions. They return confident-sounding but misleading answers, eroding user trust and damaging operational outcomes.

How do you get data that is AI-Ready for AI projects?

Here’s what I look for when searching for data for AI projects.

Five essential characteristics of AI-ready data  

• Accessible and unified: Data must be centralised and queryable. Teams shouldn't need to submit tickets or wait days to access the information they need. 
• Quality: Data should be clean, validated, and consistent. This means implementing automated quality checks and clear ownership of data domains. 
• Rich in metadata: While AI can work with various data formats, it performs best with metadata rich modelled data. This includes both structured data warehouses and well-organised data lakes (unstructured and semi-structured).
• Governance: Clear policies around data access, privacy, and usage ensure compliance and build trust in AI outputs. 
• Observability: You need visibility into data pipelines, transformations, and quality metrics. When something breaks, you must know immediately.  

Making your data AI-ready

There are 3 key areas that yield well-modelled, well-governed data necessary for AI projects.

1. Activating data governance

Data governance isn't optional for AI-ready data, its foundational. Effective modern data governance ensures: 

• Data quality standards: Clear definitions of what "good data" looks like for different use cases, with automated validation to enforce those standards. 
• Security and compliance: Proper controls around sensitive data, especially when AI models may expose or infer information about individuals. 
• Data cataloguing: Searchable metadata that helps teams find and understand available data. This accelerates AI development by reducing time spent hunting for information. 
• Ownership and accountability: Designated stewards responsible for data quality, documentation, and access policies within their domains. 

Without governance, organisations struggle with compliance risks, duplicate efforts, and erosion of trust in data-driven decisions. 

For a practical guide about reframing data governance and where to get started, read “Reframing the data governance conversation.”  

2. Establishing metadata

Making analytic data AI-ready also requires that data carries the necessary metadata that improves the meaningful insights generated by AI.

This metadata is resident in five distinct categories:

• Descriptive and schema metadata
• Operational metadata
• Trust and governance metadata
• Semantic and contextual metadata
• AI specific metadata

The first 3 types of metadata are the bare minimum pre-requisites for making data AI-ready. The last 2 types of metadata improve the accuracy and precision of AI models and reduce the likelihood of hallucinations and erroneous predictions. Embedding the metadata into data destined for AI models requires robust data engineering and the activation of policies and culture that render the five essential characteristics of AI-ready data.  

3. Strengthening data engineering

AI models work with the data they are provided. So having strong data pipelines with trusted data mean that the AI models can start from a point of trust. Data engineering bridges the gap between raw data sources and AI-ready data for AI models and analytics. It involves: 

• Creating reusable frameworks and patterns
• Building robust data pipelines that handle scale
• Infusing necessary metadata into data and data pipelines
• Implementing proper data modelling and transformation logic
• Establishing monitoring and alerting for data quality issues and trends
• Managing data processing costs efficiently

Data engineering practices can assure requisite metadata is provided to ML/AI models to improve their accuracy and precision. Without strong data engineering practices, organisations face a painful cycle: build AI models that fail due to data issues, spend months fixing data problems, then discover new issues when deploying the next model. 

Data engineering teams that use metadata-driven frameworks can have an edge on those that don’t in terms of making data AI-ready. Metadata-driven frameworks automatically embed the first three essential layers of metadata - descriptive, operational, and governance - directly into the data pipelines built with them. Because metadata is used as input parameters in metadata-driven frameworks, the data products produced by them retains the descriptive and schema metadata.

Likewise, the templates and patterns used in these frameworks generate operational and governance metadata as the raw data is processed and refined through the data pipelines. With the frameworks providing the first three types of metadata, data engineering teams can focus their expertise on high-value business logic and the contextual enrichments necessary for high-quality, trustworthy AI predictions. 

Next Steps

The foundations for successful AI Projects are likely already within your organisation. If not, or you need to strengthen them to do more, the good news is that you don’t need to solve everything before starting. A simple next step isn't a total overhaul but perhaps implementing a metadata-driven framework that automates the "heavy lifting" of data engineering and metadata management.  Whether you build this internally or adopt an existing platform such as Altida from Altis, the goal is the same: create a scalable foundation that makes AI achievable, not aspirational.

Would you like to know how your organisation measure up? We can help you assess your data foundations to see how quickly you can move from being data-heavy to AI-ready in weeks, not months.

Get in touch for a quick assessment of your AI-readiness. 

Learn more about Altida, our metadata-driven data engineering framework.