Big data is pervasive today but many organisations are still struggling to understand where it fits into their technology and business landscape and to define a suitable use case to get started with Big Data.
At Altis, we’ve been exploring different use cases for Big Data and our team have built a Sentiment Analysis tool using our Yammer collaboration platform. In a matter of weeks, the team built an end to end solution to demonstrate how Big Data concepts and techniques can be applied in a broader context. Big data used to be defined by the 3 Vs (volume, variety and velocity). Even without large volumes of data, the variety and velocity at which data is changing is shifting our expectations from batch or overnight processing to near-real time analysis for decision making. The wealth of new tools now available to make analytic and machine learning accessible is empowering new insights for a competitive edge.
Some other Big Data uses cases we are helping clients with include:
- A large New Zealand financial institution moved to a cloud Hadoop platform recently and reduced their disk space requirements by 16 times and improved the timeliness of forecasting by reducing their process from three weeks to two days.
- An Australian water utility is utilising their big data platform for real-time analysis and management of their network to be more responsive to pipe and equipment failures.
- A New Zealand transport operator is utilising big data to improve timetabling and resource management so their buses spend less time “off grid” through an improved understanding of journeys and times, which will reduce costs and maintenance.
Why did Altis pick Sentiment Analysis for our proof of concept?
There are many established solutions for sentiment analysis in the market, but these often focus on external feeds or social media and come with a hefty price tag. We looked internally at the data we already have available and the variety of potential data sources including team engagement surveys, our collaboration platform Yammer, Office 365 and Sharepoint. As each of these are rapidly changing and come in different formats, it was an excellent way to practice and demonstrate the technology and techniques that are needed to ingest, transform and present big data.
We developed our concept Sentiment Analysis tool using the Microsoft Cortana suite, including:
- HDInsight (Data Lake service) as a data repository for structured and unstructured data
- Apache Hive to transform and manipulate the data
- PowerBI to present the findings in an interactive dashboard
Insights from this proof of concept:
- Adding in a layer of word mapping to positive and negative, before pushing through Cortana, allowed for greater flexibility and customisations for industry verticals.
- Privacy is an important consideration, in particular when dealing with detailed data from internal sources. We ensured privacy was maintained by anonymising the data on extract.
How can Altis help you on your Big Data journey?
- Architecting data in today’s world
- Building Big Data solutions as needed
- Helping organisations understand the value of data
- Delivering training programs
- Building solutions that can maximise your investment in Big Data
Contact us at connect@altis.com.au or get in touch with your local Regional Manager.