Data warehousing applications in organizations are usually viewed as focusing on either operational or analytical applications. Operational applications focus on providing decision makers the information they need to monitor and control their organization. Analytical applications, which include data mining, allow the use of sophisticated statistical and other analytical software to help develop insights about customers, processes, and markets.1 Several analytical applications are discussed in Technology Application 10.1.
Data warehouses can be a massive effort for a company. For instance, Wal-Mart’s worldwide data warehouse is the largest in the world with over 16 terabytes of data in a single data warehouse.2 For many companies, such integration of corporate-wide data is a taxing process that requires years of development. This complexity is raised another magnitude as companies increasingly try to use data warehousing tools in contemporary enterprise systems to merge data captured through processing with other types of data desired in a data warehouse.
Using this massive array of data from which customer buying habits, characteristics, and addresses can be analyzed and linked, marketing departments can undertake extensive studies. Researchers armed with neural networks (as discussed in Business Intelligence and Knowledge Management Systems), comprehensive statistical analysis packages, and graphical presentation software can rapidly begin to develop insights about relationships within the marketing information. Of course, as demonstrated in Technology Excerpt 10.2, users of the data warehouse need to consider carefully what the outputs of such analyses really mean.
TECHNOLOGY APPLICATION 10.1
Applications of Data Mining
National Australia Bank recently implemented data mining tools from the SAS Institute to aid particularly in the area of predictive marketing. The tools are used to extract and analyze data in the bank’s Oracle database. Specific applications currently focus on assessing how competitors’ initiatives are affecting the bank’s bottom line. The data mining tools are used to generate market analysis models from historical data recorded in event-level form. The addition of data mining tools is one more step in a strategic set of initiatives focusing on the development of a comprehensive data warehouse. National Australia Bank considers the data warehousing initiatives to be crucial to maintaining an edge in the increasingly competitive financial services marketplace.
National Data Corporation/Health Information Services (NDC/HIS) has found a way to leverage its extensive database of pharmaceutical firm data into a new market of information services through the use of data mining tools. A long-time supplier of information to the pharmaceutical industry, NDC/HIS recently released a new subscription service, Intellect Q&A, that provides in-depth information that subscribers can mine for key data. One happy client is Lowe McAdams Health Care, a Manhattan advertising agency. It recently won a $20 million contract with a client after mining through NDC/HIS data and developing detailed financial information on the client that the client’s own internal reporting systems had yet to compile. The client was so impressed by Lowe’s depth of knowledge that they signed the contract. NDC/HIS was also thrilled as it created an excellent example for demonstration in selling access to their data warehouse to other potential clients.
FAI Insurance group also recently implemented a large data warehouse. Insurance companies are broadly recognized to be heavily reliant on demographic data that support assessment of insurance risk. FAI Insurance decided to use its data warehouse to reassess the relationship between historical risk from insurance policies and the pricing structure used by its underwriters. The data analysis capabilities should allow FAI to better serve its customers by more accurately assessing the insurance risk associated with a customer request. Developers of the data warehouse note that the models that can be built through data mining are much larger than those the company could previously develop. Through the use of neural networks and linear statistics, the analysts comb the data for trends and relationships. Out of 25 users currently on the research system linked to the data warehouse, 12 researchers work almost full-time on mining the data in the system. The strong relationship between the information technology group and the researchers has resulted in what FAI believes to be the most effective data-mining and data-warehousing approach in the industry.
Sources: Iain Ferguson, “Data Mining Lifts Competitive Edge,” Computerworld (February 6, 1998): 18; Linda Wilson, “Data Mining Strikes Gold,” Computerworld (August 8, 1997): 34; Merri Mack, “Data Quality and Analysis are FAI’s Secret Weapons,” Computerworld (April 4, 1997): 24.
Technology Excerpt 10.2
The Surprising Discoveries in Data Mining
All This Mining Can Be a Dirty Business by Shaku Atre
FRAMINGHAM, MASSACHUSETTS—First, they told us to go work in a data warehouse, forklifting loads of data and doing lots of heavy lifting. We were told to establish total control over the company’s data inventory so end users could order any combination of items at any time. Before long, we were in the big boss’s office, requesting the kind of hardware and staffing used at NASA to calculate interplanetary space shots.
And we wondered whether we would need a UN peacekeeping team to help departments agree on what the terms “sale” and “customer” meant, so we could model the enterprise. When the boss said we might be happier in outer space and put us on a stricter allowance, we looked into data marts. That was a way in which each user department could create a decision-support solution that would stand alone as a beautiful, tropical island of information.
Now they’re sending us to the mines. The data mines. We’re supposed to dig out diamonds of information. A diamond mine is the kind of place where—in a prison movie—they send lifers.
Couldn’t we just strip-mine for coal and avoid a bowels-of-the-earth excavation?
If the metaphor holds true, we’re in for some very dirty business, laboured breathing and low pay, in an effort to harvest baubles. In a famous data mining example, a large retailer discovered that beer and nappies often wind up in the same shopping cart at convenience stores on Friday nights.
The analysts theorise that Mum sends Dad out for nappies, and he picks up a six-pack while he’s at the store. But the analysts aren’t positive. Maybe Mum is going out for a six-pack, and the baby starts to cry. That’s where the results of data mining can get tricky. Armed with the market-basket correlation of beer and nappies, what action do we take? The experts say we can stock beer and nappies side by side. Or place those items at opposite ends of the store, so Dad or Mum will have maximum exposure to the temptation of impulse shopping when travelling between beer and nappies.
But why stop there? We can stock nappies in the refrigeration unit and put warm beer next to the baby formula. We can put them in the same package. If Dad or Mum pays by credit card, we know where they live, and we can manipulate the information. For starters, we can mail them discount coupons. Then, we can sell their information so that bags full of catalogues and special offers clog their mailbox, and telemarketers ruin their dinner hour. From there, the sky’s the limit.
As a technologist, I recognise that data mining offers great value to businesses. It can improve efficiency and increase precision in supplying what customers want.
But as a person on the receiving end of data mining’s insights, I feel a growing dread as marketers and others sharpen their aim in trying to influence and manipulate me. Who wants to shop in a store that runs you around like a rat between special offers? Who wants to order some clothes by telephone and, as a result, come to the attention of hundreds of companies that want to sell you something or change your behaviour? Who wants to be denied credit, a job or insurance—because you fall inadvertently into a pattern found by a computer? In short, who wants to feel that whatever move you make is captured and engenders the attempts to influence your next move?
Source: Shaku Atre, “All This Mining Can Be a Dirty Business,” Computerworld (August 22, 1997): 24. Reprinted with permission.
Note: For our American readers, we should probably note that a nappie is more commonly referred to as a diaper in the United States.