As any initiative is about value and money generated, next sequence reflects money generation process based on Company value added chain (VAC):
Raw Data -> Smart Processing -> Actionable insight -> Changing/Optimizing VAC -> Money
As I mentioned before, Actionable Insights can be done only by people. It’s true so far just because there is no viable approach to teach machine about business concept and common sense. It can’t be changed, no matter of amount of information gathered, level of “smartness” of modern algorithms and power of hardware.
It’s long way to totally different type of Artificial Systems, understanding world, reasons, passions over facts and optimization. Most of current algorithms are based on idea of optimization and it makes them weak in complex and fast changing reality. Model, which is “optimal” now, can be totally useless in couple days because some parameter changed beyond limits. Example? New local tax, or regulation. It can change behaviour of customers and Company will need new Actionable Insight from totally new model/data.
So, it makes sense to add human efforts to picture: Raw Data-> Cleansing and Preprocessing -> Specialist/Analyst Efforts->Actionable insight -> Changing/Optimizing VAC -> Money
It makes clear that Actionable Insights depend on Analysts qualification. Technology has role of power tools rather then decision making authority.
Due to uniqueness of every company, generating money based on Big Data can be challenging task, and, in most cases, it doesn’t work without engineering appropriate solution/processes. Today, vendors don’t have generic solutions in the area, so every company is doing own customization using tools available on market. It means the need of significant R&D and availability of people with inventive mind, not just coder or product consultants.
To emphasise role of qualification for Actionable Insights, here are several use cases.
Example 1. IT system malfunctioning. After some growth of number of users, Enterprise Asset Management application started failing to distribute users between servers. As a result, system worked slowly with crushes as everyone was sitting on the same Web Front End. System produced millions of lines of log messages, but IT department couldn’t make any sense of it. So, there were no actionable insights so far. The same log data where provided to Consulting Company specialized on the application support. They got the same actionable insight as IT department; which means nothing certain. Then vendor provided one hour of developer, who designed balancing feature. This time, millions of records made sense and generated actionable insight. It was a bug happening with specific combination of Network Adapter/Driver/OS/version of middleware. Bug was fixed in one day, after month+ of struggles.
So, in short: millions of logs records + IT Department + Consulting Company = 0 insights.
Same millions of logs records + 1 vendor developer = actionable insight = problem fixed
Example 2. Product Quality Analysis. Company had billions of recorded PLC instruction/signals, reports, and there were no signs that something is going wrong with equipment. Only concern was raised by quality control about suspicious weight variance of product (let’s say bottle of water). Although it was inside limits, the manager wanted explanation.
After trying different ideas, Consultant decided to analysed operators’ activities and found weird pattern. Operators regularly switch process manually when one of tanks still was not empty. After investigation, they found some minor defect in tank which prevented normal flow for last quarter of liquid. Long time ago, operators found how to bypath the problem, so they just did it as part of internal procedure. They routinely “prevented” problem which was about to happen every shift.
Finding the flow in tank is not real actionable insight in the use case. It’s rather knowledge that operators can have some common tricks and do them regularly. So, here, I would apply power of Machine Learning solutions: having all manufacturing process data, raise alert if pattern of manual intervention inconsistent between plants. This way, company can directly invest into stable quality, early signs of problems with personal and equipment. Money coming from confidence in product and fact-based warranty, as well as problems prevention, which is common cost saving tool for manufacturing.
To be fare, in some cases, generic tools and approaches work without too much expertise
Example 3. Sales prediction. Company routinely gather geo-tagged sales information by communities with all available demographic detail, such as income, education level, as well as weather and economical parameter of regions. Data are enriched with location/number of partners, competitors’ activities and sales representatives’ current activities. Altogether, it’s pretty big volume of detailed information. After data cleansing and massaging, standard BI tools (including pivot tables in Excel) can easily show anomalies (like drop or spike in sales in similar communities), correlations, and predictions.
In last example, money value of Actionable Insights is coming from directing local representatives, changing frequency and type of sales activities, introducing new product and initiatives to contractors covering specific communities. Month or even week later, Company can compare results of initiatives with forecast and correct predictive model/initiatives.
Does the example provide any competitive advantage if every single company uses similar tools/data? Only if Company can learn quickly and react faster than competitors. We will discuss some approaches to it in next topic about Machine Learning.
Roofing Industry. Big Data and Actionable Insights.
It’s common practice to mix in one presentation Actionable Insights and Big Data (see top process on picture below). This way, marketing people are trying to connect Big Data with some value directly. I’d like to show that this connection is not that simple. Value and money are not coming just from gathering more data and getting some automatic calculations.
Actionable Insights and Data Processing are the same old as data itself. Examples of actionable Insights can be found as far as 6000 years back in Babylonian records talking about Sun and Moon cycles or Egyptian’s records about levels of Nile and prediction of harvest. Some observational person discovered dependency or correlation using previous experience and records. Then the bright guess was supported with new data. It shows that Actionable Insight is very personal business related to intentions, skills and sometimes ambitions.
“Big Data” is a modern approach to collect, store and process data, which is relevant to current computers hardware and software. “New” part here is implementation of parallel processing on fault tolerant infrastructure. It means that system consists of many nodes and any job is done on many nodes simultaneously. In theory, due to redundancy in processing and storage, failing of one of several nodes doesn’t influence result. It means, that a lot of data can be processed faster and on cheap replaceable hardware. It’s often promoted as ideal tool to make Actionable Insights.
From Data perspective, only limited number of Actionable Insights need Big Data. But it can be very valuable Insights, which is practical to dig out only if Big Data technologies available. Pattern analysis is most common way to get value, and it works well only when there is an ocean of similar data.
From project management perspective, Big Data help to fit time and budget limitations. When data are in one big storage (Data Lake), it’s easy to apply new requirements, build reports, calculate statistics and prove hypotheses.
From volume and processing perspective database severs and file storages can’t compete with Big Data. But other aspects require more attention as Big Data is not answer to every problem. For example, most of available Big Data components don’t provide transactional integrity, they have problems with multi-user queries, and there are many limitations for data updates.
Comparing Big Data on premises vs Big Data on Cloud is another very important point. For roofing industry with dozens of plants, I would say that Cloud could be preferred option for encrypted or anonymized manufacturing data. Using Amazon Glacier option with rare access, is cheaper than paying for dozens of servers, space, maintenance, electricity and saturate bandwidth of internal network with low priority traffic. You need data analysis on Cloud? Rent small Spark cluster, copy specific data from Glacier, run query/report/ML algorithm, shut down cluster and pay only for used hours. It’s nice, cheap and is not killing performance of internal Data Lake!
The key element of Actionable Insight is human expertise in specific domain and/or company’s processes. Volume of data and sophisticated tools can’t replace understanding of what is missed in processes and inconsistent in data gathering. If data missed, there is no way to come to right results using only calculations. So, internal bright and motivated people is a requirement for successful Big Data Actionable Insights!
In addition to generic diagram, I put specific example of Actionable Insights for roofing industry using Big Data and Enterprise Data Warehouse. It illustrates an idea how to extract relevant information from diverse data sources to support profit-generation activity. I intentionally choose use case where it is hardly possible to avoid using of Big Data.
Is it difficult to architect similar Actionable Insights for some other task? It requires combination of Business understanding, IT systems Engineering, and Data Analysis. It is not the same as installation of modern tools and connecting several data flows in one place. And, in most cases, it’s impossible to outsource because it’s based on core information streams of business, available only to insiders.