Mike Lampa on BI

BI Appliances: Why does it have to be so hard?

2010-08-21T21:59:00.001-05:00

There's lots of buzz these days on appliances in the Business Intelligence and Data Warehouse space. Twenty-eight years ago, Teradata established the first reference pattern of a database machine, later to become known as a data warehouse appliance when "data warehouse" caught on. Netezza made the term "appliance" popular in 2000 with it's marketing campaign. Now several database vendors (Greenplum, Asterdata, Vertica, Paraccel, Dataupia to name a few) offer their product as an appliance either with their own DBMS software or combined with a pre-defined hardware configuration from a hardware partner. According to Gartner, roughly 50% of of the vendors on the 2010 Data Warehouse Database Management Systems Magic Quadrant are considered appliances.

So today's concept of an "appliance" in the data warehouse & business intelligence space appears to be centric to the database management function. That's a great start, but how do I get the data into these database appliances and how do I get it out for consumption and to take action? What about the data integration, data visualization, data analytics and operational performance management functions? Will we see appliances pop up in the data integration, visualization, analytics and performance management functional areas as stand alone appliances or will we see the database management appliances expand their scope to include these other mission critical functions? Will we find ourselves inter-connecting functional-silo appliances or will we see an evolution towards bundled "BI in a Rack" multi-functional appliances? Will the multi-function appliances contain all of the compute, storage and network services necessary to seamlessly enable the inter-operability across the data integration, data management, visualization, analytics and performance management functions? Can I fill up cages with "BI in a Rack" functional appliances to facilitate “sprawl” (multi-node grid processing across these functions)?

Presuming for a moment that the short answer to my barrage of speculative questions is a resounding yes, yes, yes, let's ponder the next question. Can I simply subscribe to these functional "BI in a Rack" appliances as a set of services? Dare I say it...can I get Business Intelligence as a Service?

I'd like to hear your thoughts.

Entrenched Success-It's a Good Thing Right?

2009-11-12T17:37:00.006-06:00

Two weeks ago I read an article about the higher education system in the US. I was surprised to learn that the US has something like 35 of the world's top 50 Universities and 8 of the top 10. It's been that way for quite a while. "Wow, that's impressive" thought I. The article went on to highlight that in the 50's the US auto makers were the world's best. They stayed that way for quite a while. In both cases these US-based enterprises enjoyed what was termed "entrenched success" in their industries/markets. Things were going along swimmingly well, day after day, quarter over quarter, year after year. "If it ain't broke, don't change it", I mused. Entrenched success is a dangerous thing. It breeds complacency and blindness to the reality that things are changing all around, all the time. The US automakers came to realize that when Toyota took the market over. The US Higher Education system could be heading for the same demise if we don't change with the times.

Last week I attended a course on strategic leadership. The instructor shared the "ladder of inference". Where we naturally use our belief systems and past experiences to hone our ability to look at the right data to come to the right conclusions to take the right actions, right? Wrong! The inherent problem is the reflex loop, which causes us to filter down to only those data points upon which we've come to rely. This results in our being blind to the myriad of other data points that could be relevant and shed some breakthrough insight. Wow! The very experience base that has contributed to success can cause us to become blind to change.

Change is constant. That includes the need to change our personal definition of what breeds success. We need to be aware of the impacts of the reflex loop and force ourselves to take off the filter when dipping into the data pool. Ask other's for input, promote constructive debate, challenge the definition of success, then draw conclusions and take action. We'll produce better results in our roles as family members, work colleagues or members in our communities.

"Where's the Process?"

2009-05-14T17:57:00.002-05:00

Attended an MDM user conference today.

As the MDM Hub was described, I got the visual of an Operational Data Store on "Biz_rule steroids". Thought to myself, "Aren’t those biz rules about biz process? Where's the Process Governance role, the Process Steward role, Process Quality? Where's the definition of valid processes?" The message came out several times that MDM is functionally intense, process intense, a paradigm shifting experience. This is all reminiscent of a few years back at TDWI as the buzz about BPM and BAM whirled about. Lots and lots of implication and rhetoric of things that sound like “process", but we continue to talk and teach Data Governance, Data Steward, Data Quality, Data Integration, Data Models, and Data Profiling. We have even gone so far as coining "Data as a Service". Oh my! Here I always viewed a SERVICE (notice the “verb” feel to the word “Service”) as a logical unit of work; an encapsulation of process and data into a bundled logical unit of work (a little itty bitty OBJECT). My soap box on this topic (for the last 8 years) is data is, by its very essence, simply a by-product of Business Process execution. Without the execution of process, there really is no need for acquiring data and there will certainly be no creation of data. Isn’t the optimization of business process the Holy Grail to optimized business performance? Granted, given today’s technology, data IS our best available empirical means to objectively measure the performance of enterprise process. I do get it. Data is important. We're just missing the boat by doing lip service to process. The business doesn't do data. Business does process. And we wonder why the business looks at us as if we have a 3rd eye when we tell them they need to own Data Governance & Data Stewardship. MDM is tough to sell to the business (just like Data Quality has been tough to sell), wonder why that is? Maybe because we keep talking about data and the business keeps thinking "that's an IT thing or a data entry clerk thing!" We need more enterprise process focus in our teachings. We need more emphasis on the disciplines for driving out the integrated business process models and mapping data to those valid process model objects. I pine for the good old days when Sarson taught us the importance of structured analysis and design at Northern Illinois University and when Martin published the Information Engineering trilogy showing us the importance of an integrated business model (process & data interacting).

Ever Wonder Why?

2008-06-08T23:49:00.006-05:00

I've been doing this data warehouse thing for a few days now. I learned quite a bit during the process. Most of the more important learning came from the school of hard knocks. One of my favorite Data Warehouse Architecture lessons is around key management strategy. The great debate is; should we use Surrogate Keys or Natural Keys? Data Architects are an interesting breed. We will debate this topic with the kind of passion typically reserved for topics of significant social impact. Ever Wonder Why?

This is a simple decision. Ask the following questions to understand the characteristics of your environment.

Question: Can the source applications 100% guarantee there will never be a change in their key management practices? If you cannot get a resounding yes you DO NOT want to use the source application primary keys as your data warehouse architecture keys.
Question: Do you have only one source application that will be feeding data into the subject areas of your data warehouse? If you cannot get a resounding yes you DO NOT want to use the source application primary keys as your data warehouse architecture keys.
I get a lot of rhetoric around the value and promise of using "natural keys". I ask you, what is "natural" about the database management system keys used by any of your source applications? There is nothing natural about an order number that is generated from an Order Management system. It's a number, a code. Sometimes it is cooked up by an application developer as a means to try to ensure each order posted to the database will have a unique index address.
I also get heated opinions around assigning a surrogate key when the source application is already assigning surrogate keys. (Think Oracle Applications) If my concern is well behaved keys, well it's already a surrogate, why put a surrogate on top of a surrogate? Good point, but (behold the underlying truth) what should we do when the single instance of the source application simply cannot handle the workload of the entire enterprise and the source application team decides to clone the application for each major region? Same Code base, different physical instances each using Oracle Sequencers for IDs.
What about source data that is truly an event, once posted will never change, has a decent unique id and has extremely high volumes? Now this is a set of characteristics that make a case for a different PK management strategy reference pattern. I guess you can't ALWAYS say always or never.

Yes, Surrogate Key management requires some engineering. It is a valid and necessary component of your overall Data Warehouse Architecture. No, not in ALL cases will every table need to have a Surrogate key. There are exceptional reference patterns that will exist. Make sure you allow for that in your overall Data Warehouse Architecture.