Intelligent Enterprise

I thought I'd share a CEP-Interest list message on Complex Event Processing (CEP) as a marketing device with Intelligent Enterprise readers. Posted by a long-time member of the event-processing community, Hans Gilde, it's an objective, refreshing market positioning/messaging analysis. I don't completely agree with Hans of course, but I do think his views are insightful. They echo points I wrote about back in 2007, and some are points I puzzled out in writing a sponsored CEP-BI white paper for Coral8. With Hans's permission, the rest of the words in this article are his --

For software vendors, CEP is a marketing device and nothing else. Notice that no two people will agree on what CEP is, but everyone claims to be more CEP than the next guy. It's like asking to define "cool." The only two consistent attributes across all "CEP" products are (1) they are in some way used for soft real-time processing and (2) they are general purpose, rather than coming pre-customized for a very specific purpose.

So the question of the CEP marketplace has two parts: The marketplace for the brand (really a "brand attribute" or whatever) that is the acronym CEP and the marketplace for products with the above two attributes.

"If you want to keep your job, use Spoke," advises recent e-mail from the folks behind "the fastest growing and most up-to-date business network in the U.S." Sounds like something to look into — social / people networks are one of the most important BI assets to have emerged in recent years — and I figured I owe Spoke another chance after panning it back in 2004. Grading Spoke according to the same accuracy, completeness, quality, usefulness, and usability standards I'd apply to other BI tools, I'm afraid I'd grade Spoke a low C. Here's why.

Spock is a people-search engine, currently in beta release. The company uses "a combination of search-engine technologies and user edits to aggregate the world's people information and make it searchable." Think Google meets LinkedIn: Web search with accuracy boosted by allowing individuals to claim, augment, and correct information about themselves. (See the screenshot below, right.)

Spock interface allowing a registered individual to amend his or her profile
(click image for larger view)

Andrew Borthwick is Principal Scientist at Spock Networks. I "met" him on the GATE e-mail list. (GATE is open-source text-mining software.) Andrew kindly fielded a number of questions I had about Spock's use of text analytics to build its search database and about Spock's data-quality efforts.

The rapid pace and high volume of twitter messaging has upped the stakes for BI on content feeds. BI on content feeds: that would be stuff like monitoring and mining sentiment from social media for reputation and brand management, which you can do with text analytics on RSS and Atom feeds and Web pages. I wrote in September on a leading-edge implementation at Thomson Reuters. But twitter messaging is both faster and, given social-network mediation, more focused: instant messaging text gone public. One approach to making sense of the flow is the CEPish application of continuous transformations that the folks behind SQLstream recently showed me.

What to make of Michael Widenus's astounding blog posting, "Oops, we did it again (MySQL 5.1 released as GA with crashing bugs)"? The signs have been there: MySQL, as an independent company and then as a Sun Microsystems subsidiary, has worked for over three years (!) to bring out 5.1. EWeek quoted Zack Urlocker, VP of products for Sun's Database Group, last May as claiming, "This version now has zero bugs," a statement disputed even then by Widenus, who effectively characterized MySQL bug management as a shell game, and GA release did take 7 more months.

It's time for the BI community to treat social networks as the business-intelligence resource they are. (BI is more than reports, dashboards, and OLAP!) The recent "Motrin moms" clamor and response to Mumbai terrorism prove networks' value. Both cases involved twitter, the first as a conduit for advertising-prompted outrage and the second for early and rapid news dissemination. It has become clear that twitter and the rest of a broad set of social networks &mdash as messaging / blogging / microblogging channels and as a means of publishing and finding personal and corporate information — hold immense business value. The value of the information that flows through these networks is indisputable. A deeper challenge is next on the agenda: optimizing that flow by better understanding the networks themselves.

Information Week has published my article on open source business intelligence (OSBI), Open Source BI Still Fighting For Its Share, a title that applies both to the BI software market and to IW column inches. (The article is now also an Intelligent Enterprise feature.) I'll share with readers material I wrote, cut by IW's editors, on open-source data-integration vendor Talend and on Eclipse BIRT, Business Intelligence and Reporting Tools.

I spent a worthwhile day last week at Clarabridge's inaugural user conference. The company is a leading text analytics vendor, and the opportunity to catch up with staff and users and (it turned out) prospects and partners, without having to travel far from home, was too good to pass up.

Clarabridge is different from many other text analytics vendors in its singular focus on customer-experience management (CEM). This isn't to say that you can't license and use Clarabridge's Content Mining Platform (CMP) for other applications, and it's not to say that the company doesn't have strong technical capabilities. It's a matter of market positioning that concentrates on a defined set of business users, across a broad spectrum of business sectors, with market messaging that focuses on business benefits rather than on technology. The approach seems to work. Clarabridge CEO and co-founder Sid Banerjee says "the business impact and deployability/usability of our solutions seem to be resonating with our customers and prospects."

Manya Mayes of SAS has written a helpful introductory paper on audio analytics, "Tune into the Voice of Your Customer with Voice Mining." While the paper, which includes a call-center case study, focuses on customer-feedback audio, the technology and techniques described have applications for e-discovery, intelligence, and rich-media search. Given coverage of distinctive characteristics of speech and of analytical concerns that include BI integration, Manya's paper merits a look for anyone who works with audio data.

Manya's thesis: "Combining voice capture with business intelligence, analytics, and text mining provides valuable customer intelligence for marketing and competitive intelligence business functions." Hers is a marketing paper, but it's motivated by significant technical and business challenges. I've looked at the issues myself and have planned to research them and couldn't agree with her more. (That's why we plan significant coverage of speech mining and audio analytics at the Text Analytics Summit, which I chair, in 2009.)

The New York Times has published another excellent visualization, this one a heatmap, Can a President Tame the Business Cycle? The on-line, interactive version adds highly useful capabilities — a detail window that pops-up as a mouse-over effect, and alternative, bar-chart visualizations of each of the seven data series — that obviously can't be delivered in a static, printed newspaper. I've noted that paper/electronic gap before. My purpose this go-around is to explore, via comparison with a financial-information visualization published by the NASDAQ stock market, how underwhelming a mediocre heatmap can be.

A press release from Lexalytics touts ExecDex, a Web site that features a "business-leader ranking index." I checked it out. Now Lexalytics makes an interesting sentiment-analysis engine, but I thought ExecDex should have been more fully developed before release to the likes of me. It seems Lexalytics CEO Jeff Catlin agreed, but the two of us couldn't have been more wrong. I suspect in the end, we both received a lesson in looking at tech applications through others' eyes.

The Semantic Web was conceptualized almost a decade ago, but despite progress on protocols and publishing tools, it remains far from realization. SW-technologist David Provost doesn't share my pessimism. To the contrary, the premise of his new report, On The Cusp: A Global Review of the Semantic Web Industry, is revealed by the report's title, namely that we're almost there. Yet the report itself, like so much material in the SW world, is itself devoid of semantic mark-up. Yes, semantics are important in boosting information findability and usefulness, but these SW examples — I cited another in a year-ago blog article — only emphasize the gap between SW boosterism and Web reality.

Do you work for a company or project that makes software tools? Now's the time to check out, and consider a nomination for, the 19th annual Jolt Awards for software-development product excellence.

I judge the database and enterprise tools categories. I'm definitely on the look-out myself for "Joltworthy" data management and analysis and application-deployment tools, products that provide an SDK, components, languages or APIs, and/or back-end capabilities for developers. There are 13 categories total. They accommodate a spectrum of software and software-related products.

A question I posed to a LinkedIn group — Is Business Activity Monitoring (BAM) a BI Application? — sparked interesting discussion. I noted and asked, "BAM involves dashboards and analyses for business processes, and BI isn't typically very processy. If not BI, who 'owns' BAM?" There have been 9 responses to date, including two from Howard Dresner, who has done as much as anyone to shape current-day BI. The responses speak to growing interest in operational BI, and they hint at the impact that complex event processing (CEP) will have on enterprise analytics.

BAM displays operational performance indicators in numerical and graphical form, often backed by rules-based alerting capabilities. BAM monitors execution of business processes and is part of operational-performance management solutions. It can be incorporated in line-of-business and operational interfaces, for instance for contact-center management, and in automated control systems. As the speed-of-business accelerates, BAM is more important than ever.

Lyzasoft, Inc. calls its Lyza software a "powerful desktop analytics solution." According to the company, Lyza "enables analysts to synthesize, explore, and visualize data, then to publish compelling presentations and dashboards." Lyza seems worth considering as a personal data-integration tool, but it appears to fall short of greater claims.

Richard Brown of Thomson Reuters delivered an illuminating talk, "News, Blogs, and Full-Tick Logs: Innovative Approaches to Quantitative and Event-Driven Trading," Tuesday at Gartner's Event Processing Summit. The summit and the Event Processing Technical Society symposium now underway feature many such use cases, descriptions of low-latency transformation and analysis of high-volume data and event streams as applied to diverse business problems. Brown's case study, which looked at exploiting information from unstructured sources to support financial-market trading, was of particular interest (to me) due to its combination of events, text sources, and sentiment analysis.

There's more to/about the Infobright open-source announcement than I covered in my Intelligent Enterprise article. I have thoughts to share on Infobright's architecture and limitations of the release. There's more to say about the MySQL data-warehousing context and then there's the puzzle of the significantly delayed MySQL 5.1 general availability (GA) release.

Advanced visualization isn't easy to get right. Simply put, our ability to generate and crunch data, and the palette of visualization options available to us, have outstripped our ability to choose charts and options that are appropriate for our data and effectively communicate important interrelationships. Nonetheless, we all know that visualization done right can really boost BI's value.

The visualization approaches taken by leading BI companies vary significantly. Some have built out their capabilities nicely — notably MicroStrategy and SAS — while others continue to promote more-of-the-same-but-glitzier graphs and charts. In a worst-case scenario, we even have a major analytics vendor that’s fumbled its latest visualization launch.

For all the coverage of the Chrome Web browser announcement, little note has been taken of Google's choice of the ultra-liberal BSD open-source license. The BSD choice accentuates that Chrome will be more (and very likely less) than a conventional Web browser. Chrome will contribute Web rendering to an increasingly comprehensive enterprise software platform and may (further) tie non-Google application developers to the Google stack.

I spend a lot of time on teaching materials on text analytics: articles, presentations, and courses. I've gotten positive feedback about my introductory materials, which I designed for practitioners (like myself) rather than for academics or researchers. There are great resources out there — technical papers and white papers, case studies, software, etc. — but you have to get the basics down first.

Prabhakar Raghavan, head of Yahoo Research, says that Yahoo "will be launching a new generation of search in two to three months... Search is going to move in a completely new direction." The initiative, one would infer from today's Financial Express interview of Raghavan, will build on Yahoo's BOSS (build your own search software) platform, which implements a "self-service Web services model for developers and start-ups."

"Gaining traction" is a good description of Kognitio's Data as a Service message. I spoke to company execs at this week's Data Warehousing Institute Conference, and I reviewed results of a survey they released at TDWI. Signs are that Kognitio's DaaS positioning is helping the company define itself and carve a niche in the crowded and dynamic data warehousing-analytics market. That positioning may need tighter focus, but focus will surely come as the company signs some of the prospects it has gained since its February North American launch.

Back in July 2005, I published a column titled The Limits of Prediction that addressed barriers to the adoption of predictive analytics. I've continued to think about the topic even as politicians come up with yet more wishful-thinking panaceas like off-shore oil drilling to reduce gasoline prices and business — for instance, the real-estate, financial, and automotive sectors — reports huge losses derived from in-hindsight foolish decisions. All those folks spend massive sums on IT and analytics. Why didn't they know better?

The Lexalytics-Infonic merger announced last week creates a company, focused on sentiment analysis, that is poised to compete with larger, established text-technologies vendors. The companies' market presence hasn't been huge — Infonic's text-analytics division booked £600 thousand (about $1.2 million) in 2007 sales, and Lexalytics, a rapidly growing US start-up, earned $1.5 million in 2007 — but the combined company should be worth more than the sum of the parts. Whether it will be worth enough to justify a $40 million valuation of the deal, which exceeds Infonic's $23 million market capitalization, remains to be seen.

Sybase is a DBMS stalwart that gets far less attention than deserved. The company recently beat financial-performance estimates and has raised its 2008 sales estimate to $1.11 billion. Sybase's on-going success — the company's DBMS is much more than the parent of Microsoft SQL Server — earns the company a closer look, in its own right and as an excuse for one last comment on Microsoft's planned acquisition of DATAllegro.

I learned of the Microsoft-DATAllegro deal from DATAllegro e-mail sent at 12:57 pm EDT on July 24. Ten hours later, at 11:11 pm, I thought I'd see what others had to say. The search for views was more illuminating than any additional analyses I found. Take a look... if you don't mind snarky blog articles —

My first thought on learning of Microsoft's plan to acquire DW appliance vendor DATAllegro is that MS is buying the wrong company. Yes, DATAllegro's parallelized database technology will fill a big gap in Microsoft's DW product line, namely that SQL Server doesn't scale to the top end, but the technology isn't compatible and will take years to build into SQL Server. I'd think it's Dataupia that would, without disruption, close the gap. No, this deal is about gutting the DATAllegro appliance. Look at the details —

AIIM, the Association for Information and Image Management, has been bombarding me with e-mail promoting Dan Keldsen's and Carl Frappaolo's sponsored study, Findability: The Art and Science of Making Content Easy to Find. While I wonder what "findability" offers beyond buzzword differentiation for a few search vendors, even after learning that "findability is the art and science of making content findable," I do understand the distinction that "under findability, the burden of intelligent content processing is placed on the content itself." I'm tempted to say that's hokum — in keeping with my amity for automated knowledge discovery in text, a.k.a., text analytics — except that supporting examples are easy to come by. Here are a few, inadvertently provided by a leading enterprise-search vendor.

I've written about the "category error" of looking at open source primarily as targeting end-user replacement of BI applications and established data warehouse platforms. I've long seen that OS's greatest BI/DW has instead been in enabling developers to build BI into line-of-business applications and create specialized analytical tools. I'm more convinced than ever of this assessment, even as OS-BI vendors have launched improvements that target enterprise end users. On the DW front, the launch of Aster nCluster supports my point.

I have created a short survey for users and consultants on Voice of the Customer (VoC) text analytics best practices. There are seven questions plus a comment field. The survey should take less than 5 minutes to complete. If you are involved with VoC text analytics or are looking at solutions for possible adoption, please respond to the survey at --

http://www.surveymonkey.com/s.aspx?sm=HyhmPOYKhh8BcDeC_2b1Im5A_3d_3d

I'll publish results at a later date. Thanks!

E-discovery and auditing are flip sides of a single coin, the one concerned with retention of records and their production in litigation, the other with studying records to verify the correct of execution of corporate business processes and accounting procedures. Extending the metaphor, compliance is the coin standing on edge: neither anticipation and response to litigation (e-discovery) nor historical analysis (auditing) but rather operational rules and monitoring designed to ensure that businesses stay out of legal and accounting trouble.

Others have reported on this year's Text Analytics Summit: the prevalence of Voice of the {customer | Market | Patient} as a theme, the focus on sentiment analysis and on BI integration, the vendor announcements, applications for analysis of social media data, and so on. This commentary is helpful so I'll link to it in this article, and I have additional observations to share, drawn from summit discussions, concerning the evolution of the text-analytics market.

I have to comment on my colleague Doug Henschen's article, "Is Social Networking KM All Over Again?" Doug did right at the Enterprise 2.0 conference to focus on cloud computing, a much more appropriate topic for enterprises than social networking. From the corporate perspective, the "cloud" is a diverse source of information, including all kinds of social and traditional media, out there to be searched and filtered for exploitable enterprise-relevant nuggets. But precipitous enterprise adoption of social networking? That would be foolish, destructive and not just disruptive. Corporations rely on and benefit from hierarchies and restricted lines of communications. Being selectively anti-social, for corporations, is a good thing.

I've been very impressed by ParAccel, a company that has lived up to their Intelligent Enterprise designation as a Company to Watch. They sell a columnar, MPP DBMS that has won strong market visibility in the two years since the company's founding. The industry experience of ParAccel executives, gained at companies including Oracle, Teradata, Netezza, and DATAllegro, surely plays a part in the company's success, and so must two technology antecedents: an earlier columnar DBMS, Clareos, and the PostgreSQL open-source RDBMS.

Some of the most interesting BI innovation of recent days has come from a, well, likely source: insiders itching for another go at BI, a chance to (re-)do it right. Ward Yaternick is a case in point. Ward led Cognos development teams, with lead responsibility for the PowerPlay OLAP engine. He created OLAP@Work, an Excel add-in to access Microsoft OLAP Services that he subsequently sold to Business Objects. Ward has been building a new company/product, nextanalytics, that unquestionably represents a fresh take on BI. Ward believes nextanalytics is now ready for prime time.

Richard Stallman announced the GNU Project in September 1983. Eric S. Raymond published the first version of The Cathedral and the Bazaar in 2000. IDC estimated a year ago that worldwide revenue from standalone open source software reached $1.8 billion in 2006, projecting a compound annual growth rate (CAGR) of 26% from 2006 to 2011. That's revenue, not the presumably much higher avoided cost of closed source alternatives. So why are open-source fundamentals still so widely misunderstood, including in the business intelligence and data warehousing markets?

Case in point: An advisory firm's recent ranking of Top 200 Private Technology Companies misclassifies DATAllegro, a data warehouse appliance vendor, as open source. Now DATAllegro's innovative "direct data streaming" and database parallelization technology is built around the Ingres open-source DBMS, but they haven't opened their own source à la Sun and OpenSPARC.

IBM recently posted a quite nice page on extracting business value from "unstructured" data. The page describes use of IBM's own products and formats to be sure, but it is potentially helpful for anyone who wishes to learn about information extraction from textual sources for data warehousing.

IBM's page starts with a brief text-analytics overview. It then dives into implementation with the OmniFind Analytics Edition for DB2 and its pureXML capabilities. It describes a process flow includes XML tagging of document features and the alternatives of mapping the XML schema to relational database structures or use using the XML structures directly for analyses. This text-analytics workflow, and the choices involved in dealing with text-sourced information, are not specific to IBM's tools, however. So which IBM provides diagrams and code listings and an analysis of the alternative approaches that relate to their own products, the lessons apply much more generally.

The New York Times publishes exceptional visualizations. A couple this week stand out: All of Inflation's Little Parts, graphing the average American's spending by category, and a map of the human "diseasome" that supports the article, Redefining Disease, Genes and All.

What distinguishes these visualizations — the first is a form of treemap, a "space-constrained visualization of hierarchies," and the second a network-connectivity diagram — is their success at communicating relationships along multiple data dimensions.

Cheers to Microsoft Labs for their release of Search Commands, an Office 2007 add-in that "helps you find commands, options, wizards, and galleries in... Word, Excel, and PowerPoint." The embedded Guided Help calls it "a useful complement to the usual method of browsing for commands by clicking tabs on the Ribbon."

I'm all for a way to work around Office ribbons, a set of interface elements introduced in Office 2007 that I characterized last September as "visually unbalanced." Ribbons degrade Office usability. I wrote in September that "they force extra clicking around for routine work and make it hard to find less frequently used functions." Microsoft is now, essentially, pleading guilty. Search Commands' Guided Help, in addition to calling the awkward process of "browsing for commands by clicking tabs" a "usual method," says Search Commands is "especially useful for finding commands that you use less often."

Adoption of text analytics has accelerated in the years I've followed the topic, with growth in expected and unexpected directions both. It wasn't hard to foresee extension of leading data mining workbenches to text, but I'll admit I had thought BI vendors would be much quicker to build handling of "unstructured information" into their technology stacks. (This lag has created partnering opportunities for a number of BI-focused text analytics companies. Business Objects' acquisition of Inxight and SAS's of Teragram show that the BI big guns are closing the gap.) And I didn't anticipate the nature of the solutions, other than semantically enhanced search and expansion in the legal, tax & regulatory (LTR) sector, that would be responsible for the greatest market growth. I'm referring to sectors such as media & publishing and applications including competitive intelligence and Voice of the Customer analytics supporting CRM, product management, and marketing.

BI Scorecard author and IE blogger Cindi Howson writes that she "gasped" on hearing Jim Davis of SAS talk about the commoditization of BI. Yet I'm with Davis — BI softwareis a commodity — the technology, that is, and not BI as a whole. That "as a whole" includes extensions wrapped around the commoditized technology core, extensions that build-out common-place core BI into solutions, extensions that adapt the technology — that package it in suite or application or embedded form — and link it to information sources and presentation and, nowadays, decision management.

I wrote about BI's information angle in an earlier BI-market appraisal. I didn’t bother to take on BI technology as a commodity in that earlier article because the technology’s commodity status was almost self-evident to me. But to take it on now, I’ll cite what I wrote in a look at database management software. Condensing a bit:

So far as that functional core is concerned, 1) there's broad consensus on definitions, 2) basic interface standards are well established, 3) leading products are interchangeable, 4) vendors compete on extended or niche capabilities, and 5) vendors don't compete on software price. I'd further add that with a commodity there is no barrier to user entry-level adoption and there's similarly only a low entry barrier for a would-be technology provider.

This description fits BI technology. It fits reporting and OLAP and ETL from the leading BI firms. It explains why the innovation is outside these core areas, differentiated by (claiming) improvement on the commodity core.

Prediction markets are mathematically based but human powered, a tool for turning collective human insights into forecasts. The approach enables individuals to bet on ideas or events. Twists such as anonymous participation, restriction to experts, and pre-screening for character traits are designed to reduce bias and boost accuracy.

The New York Times reports that "companies use prediction markets to funnel ideas from the work force" (Betting to Improve the Odds, April 9, 2008). Steve Lohr's article provides illuminating examples from companies including Best Buy, InterContinental Hotels, and Hewlett-Packard. Lohr also notes, "for years, public prediction markets have been used for politics, where buyers and sellers bet on which candidate will win a particular race." It’s the kind of article that everyone interested in decision sciences should read, and then follow-up on to understand not only the mechanics of the techniques but also their limitations.

Intelligent Enterprise and TechWeb are not the New York Times, so we get to publish some of the news that's seemingly Not Fit to Print:

Data warehousing powerhouse Teradata has acquired BI/DW solutions provider Claraview.

The news is there on Claraview's Web site — "Claraview is a division of Teradata Corporation (www.teradata.com), the world's largest company solely focused on raising intelligence through data warehousing and enterprise analytics" — but I don't have a clue why Teradata hasn't seen fit to announce the acquisition. There are a few hints of the take-over out on the Web, and an individual in a position to know told me the deal is (or was) "an open secret."

SAS has announced their take-over of text-analytics vendor Teragram. The companies' joint press release states that "the acquisition will enhance SAS's own robust text mining and analytical BI offerings and extend them to enterprise and mobile search." The release cites Teragram's natural language processing (NLP), categorization, and enterprise search technologies, but it leaves much of back-story untold. Here's my take on positioning, technology, and solution considerations that likely motivated what seems like a smart move for both companies.

The essence of the announcement is captured in SAS CEO Jim Goodnight's statement, "Teragram's technologies augment, strengthen, and extend SAS's ability to combine structured and unstructured data – not only in our text mining solution but embedded across the entire SAS Enterprise Intelligence Platform – to drive better answers faster." SAS chief text mining strategist Manya Mayes added in e-mail to me, "Teragram has a broad solution that includes enterprise search and mobile BI" that "complements SAS Text Miner."

I learned of the March 5 death of computer scientist Joseph Weizenbaum from a posting on the CPSR e-mail list. CPSR, Computer Professionals for Social Responsibility, recognized Weizenbaum with their Norbert Wiener Award in 1988. But he was of course best known for creating Eliza, a mid-'60s computer program that conducted natural-language conversations, notably mimicking a psychotherapist's interview with a patient. Eliza was named for the character in George Bernard Shaw's Pygmalion (familiar as the source of the musical My Fair Lady), someone who was similarly taught to impersonate something she was not. Weizenbaum wrote in his 1976 book Computer Power and Human Reason, "Eliza created the most remarkable illusion of having understood in the minds of the many people who conversed with it."

Greenplum recently released a new version of its BI optimized DBMS, Greenplum 3 (G3). The software is based on the PostgreSQL open-source database system; proprietary extensions add support for parallel loading and query and other scalability and reliabilty features.

But with G3, Greenplum appears to be moving ever farther from the company's open-source roots. Specifically, Greenplum sponsors Bizgres, an open source, BI-optimized but non-MPP DBMS, downloadable as an April 2006 0.9 version at bizgres.org or via Greenplum's site. Later versions are available in the code repository but no later version has been packaged as a General Availability release. Note that Greenplum does, separately, contribute to core PostgreSQL development. Frankly, I wonder if/why that contribution isn't the focus of the company's open-source involvement.

Just as every presidential candidate this cycle is the candidate of Change, it seems that all the DBMS vendors offer the preferred data-warehouse appliance solution. That's the message I heard from appliance panelists at today's TDWI Washington DC chapter meeting. For a couple of them it was a real stretch, which in one case wasn't a bad thing. The net take-away is that we are seeing Change in the DBMS world, even if for the politicians that word is still only a promise.

TDWI-DC's panel consisted of Doug Cardin from IBM, Victoria Eastwood from Infobright, Phil Francisco of Netezza, Foster Hinshaw of Dataupia, and Rita Sallam of Oracle. Now my definition of DW appliance is a packaging of processor, storage, operating system, and DBMS that is optimized for data warehousing. A scalability model is essential. And only one of the represented companies hits the mark: Netezza, with an asterisk for IBM.

The saying "There are three kinds of lies: lies, damned lies, and statistics" is attributed to Benjamin Disraeli, and it's nicely illustrated in a couple of Intelligent Enterprise reality-check photos from this week's TDWI conference.

Check out the TDWI Image Gallery photos posted by Intelligent Enterprise Editor-in-Chief Doug Henschen. Executive Summit attendees used the "dots method" to identify Important BI Technologies and Biggest BI Challenges. You get simple histograms showing what's hot and what's not. And you get clear illustrations of "selection bias": not TDWI's fault, but an effect to keep in mind when you assess formal and informal research findings.

My last blog article relayed key points about e-discovery and potential knowledge-discovery (KDD) applications in the legal sector that were reinforced by my participation in the recent LegalTech conference. A LegalTech exhibitor I spoke to mentioned his company's discussions with IBM, so I dropped IBM text-technologies researcher Aaron Brown a note to learn his company's side of the story.

Aaron is program director, Content Discovery and Search, IBM Information Management Software. His thoughts on legal-sector KDD were very much in line with mine. He graciously gave me permission to share his response, which I'll post verbatim —

"Discovery" is a legal process whereby parties to a lawsuit request and provide documents and information that may be pertinent in litigation. "Discovery" also describes an analytics goal that has nothing to do with the court system: extraction of useful information — data, facts, and rules, which together constitute knowledge — from databases and textual sources.

I had expected the December 2006 federal rules amendments on discovery of electronically stored information — "discovery" here in the legal sense — to open new vistas for application of knowledge-discovery technologies: data mining, machine learning, visualization, and the like. The reasoning is simple. Corporations must now retain vast volumes of electronic records including e-mail and information from enterprise operational systems. To comply with e-discovery mandates, they must be able to "produce" records in response to discovery processes, and that means metadata-management, classification, search, and similar systems.

Curt Monash shares my disdain for PowerPoint: not the software per se but rather the rigid communication dysstyle (="dysfunctional style") it encourages. Seeming solutions such as pecha-kucha — "a simple set of rules to presentations: exactly 20 slides displayed for 20 seconds each" — are seductive. You pick up the pace and limit the text that can appear on a slide, which seem like pluses. Per Daniel H. Pink's description in Wired, you "say what you need to say... and then sit the hell down." On the other hand, you’re still locked in that rigid PowerPoint sequence. Just think of that stereotypical image of the American tourist abroad: If foreigners don't understand English, speak slower and louder and maybe they'll get it. It doesn't work of course. Similarly, faster, simpler presentations aren't necessarily better presentations.

The same principle applies to communicating analytical results.

I’m a fan of network visualizations, by which I mean display of interconnectedness mined from disparate sources. The subject matter could be just about anything: witness the collection of projects at Manuel Lima’s VisualComplexity site. Social networks inferred from on-line media prove particularly interesting, the sort of stuff you’ll find in static form at Jeffrey Heer’s and Danah Boyd’s vizster site and dynamically in Linkinfluence’s Map of the Political Blogosphere, which I wrote about last month. Silobreaker (as an on-line application) takes these efforts a big step further.

Silobreaker visualizations add huge value to the company’s underlying news-aggregation service. They classify nodes by type. If you hover your mouse cursor over a node, you can explore its connectedness, and if you hover over the node’s text, you can learn more about that node, whether it represents a person, company, key phrase, or other type of entity. Hover over a link (edge) and you’ll see “documents indicating a relationship.” Naturally, you can double-click on a node to remake the visualization. Please, visit the site yourself and explore… and do some searches on terms that interest you. Then try the 360° Search, which aggregates content related to the search topic and displays retrieved and analyzed information via a variety of widgets.

Prof. David Farber has issued an interesting challenge:

Endlessly people talk about the Next Generation Internet. In fact the term has been used so badly that it is meaningless. I need a name for the Internet-like network we will need when we are faced with end to end optical communications at hundreds of gigabits; multi-core computers (large number) and other now-research technologies.

The best analysis of the motivations and implications of Microsoft's bid for Fast Search & Transfer (FAST) is Stephen Arnold's, yet his in-depth look, along with the rest of the reporting I've read, has nothing to say about some of the most interesting technology that Microsoft will acquire. Sure, given about FAST that "Our Business is Enterprise Search," and given that enterprise-search mojo and customers are what Microsoft seeks to acquire. There's much more to FAST, however: a fresh approach to data warehousing, a search-integrated BI dashboard, and an ad-delivery platform, that last being where the real search money is to be made.

ParAccel has won well-deserved attention in recent months, including Intelligent Enterprise recognition as a Company to Watch. They're a start-up that boasts an all-star cast of executives, positioned in a hot category, namely column-store DBMSes that are optimized for analytics. There's irony, however, in their market positioning. It's not that column stores, most notably SybaseIQ, have been around for decades. It's that ParAccel chose to explain their product with an application, analysis of U.S. Census data, that is essentially owned by a competing column-store system, SuperSTAR from Space-Time Research.

I have personal history here: I designed the U.S. Census Bureau's Census 2000 tabulation system, working on subcontract to IBM. Back in 1998, I wrapped up the selection of SuperSTAR over competing options. We chose SuperSTAR for superior performance and ease of use. I then led the development team that created a system that supported both ad-hoc queries and the production of hundreds of billions of statistical tables for subsequent publication via the Census Bureau's American FactFinder Web site.

The $1,000,000 Netflix Prize competition has produced interesting results, even if no winner, 15 months in. Some of those results are a bit surprising; others we should have expected but didn't anticipate. So while participants haven't yet bettered the accuracy of Netflix' Cinematch recommendation algorithm by 10%, the threshold to win the $1 million prize, we can still take away lessons about predictive-analytics fundamentals.

I recently checked on competition status after receiving a note from Alex Lupu, VP Marketing USA for Scio Systems; Alex has been keeping me apprised of his company's progress toward launch of property-lease abstracting and analysis tools. Like Alex I'm into text analytics, and I liked his take that "intelligent communication between customer and the [Netflix suggestion] system" could provide an alternative route to better recommendations. Alex sees analysis of "'open questions' that allow customer to write a sentence or two" about movies as potentially beneficial in complementing traditional, pure-numbers predictive modeling. Alex says "assuming the customer is a static entity seems wrong to me, thus looking at databases only is not of much help."

It has been a year since Intelligent Enterprise magazine went on-line only. The last print issue, dated January 2007, came out last December. I thought I would miss the paper edition but now I see that, from a writer's point of view, the overhead of a print run, particularly for an IT publication, is a greater liability than may be justified by the extra value delivered.

I was afraid that writing for Web distribution would diminish my authority as an analytics-industry observer. After all, the expense of producing and distributing printed magazines says that someone, even if only the publisher, thinks that the content justifies the cost. Sure, items on the Web are more findable and, simultaneously, timely and long-lived. But for an established author, the threat of losing the distinction conferred by paper counterbalances greater reach and timeliness. That threat hasn't been realized.

I wrote last week about a set of New York Times campaign visualizations that caught my eye. They met my "good" criteria: data-appropriate, designed to communicate rather than (merely) show off. The good is often contrasted with the bad and the ugly. Let's check out examples and then look at a TIBCO-Spotfire demonstration site.

The Bad: A Map of the Political Blogosphere from Linkinfluence, a company that "engineers mapping, monitoring, and analytics solutions for the social web." I read about the site in Matthew Hurst's Data Mining blog. Matthew writes, "I believe that they have put plenty of effort in to the design of the data visualization and the overall look and feel to really make the site stand apart from others in this space."

I do admire a nice visualization, one whose composition suits the nature of the underlying data, one designed to communicate rather than as a means of showing off technology. Given these criteria, the New York Times delivered twice last Sunday with a pair of visualizations that nicely distill presidential-campaign themes and dynamics from what was otherwise a mighty big pile of words: debate transcripts. The Times's visualizations are useful in another way. They exemplify good design, especially when contrasted with other technology-first visualizations on similar topics.

Facebook is good for something (beyond wasting time)! It brought me to a BI 3.0 discussion thread started by Darren Cunningham, prompted by his LucidEra colleague Ken Rudin's blog entry, What's in Store for Business Intelligence in 2008.

Ken is perceptive. His five predictions are:

1. SaaS BI will continue to gain market traction.
2. BI innovation will be led by newer vendors.
3. There will be a shift away from tools towards prebuilt analytic applications.
4. Applications that integrate data and improve processes across transactional systems will drive the next wave of SaaS.
5. A new breed of BI channel partners will emerge.

Do follow the link and read the full blog article, and then consider my BI 3.0 comment addressed to Darren, that LucidEra does interesting enough work, but Ken Rudin's hot-in-2008 list is mighty solipsistic. Is there really nothing (significant) in store for BI in 2008 that isn't touched on by LucidEra offerings?

Given the strengths of open source database management systems (DBMSes), open source seems like a natural platform for data warehousing. We've seen a number of success stories over the last few years, Travelocity, O'Reilly, FTD, and Frontier Airlines among them, but the roster of case studies is mighty thin. But I've only recently (re-)started looking — in the last couple of years, on the open-source front I've covered mostly BI (e.g., May, March) — and I hope to find many more for a report I am planning on open source (based) data warehousing (OSDW).

I'm getting pretty tired of the agonizing whether recent market events and trends mean the end of business intelligence as we know it.

Some of my fellow pundits are scrutinizing vendor consolidations and they're studying the impact of the emergence of new analytical approaches and application-delivery methods. Consolidation will mean a refocusing of product development for acquired vendors, that's all, not "the end of BI as a separate application" as Ephraim Schwartz, for instance, sees it. Schwartz acknowledges authoritative views that are contrary to his — "As Howard Dresner, principal at Dresner Advisory Services, says, 'For every vendor that is acquired, there are 20 emerging companies offering new approaches, technologies, and business models.'" — but dismisses them and slights the impact that Software as a Service, SaaS, has already had on BI.

IBM buys Cognos. SAP buys Business Objects. I agree with Neil Raden: "One big yawn."

Neil's take:

[Hyperion, Business Objects, and Cognos] have made substantial progress in refurbishing their products for a completely new world, but refurbishing only goes so far. The architectures can't really cut it. They need to scale, they need to be intelligent, they need to react in real-time when necessary, unattended when appropriate. They need to live on the Web.

Products like that are coming from start-ups.

Neil's analysis is spot on but calls for elaboration. Neil sees outmoded product (and process and business?) architectures as impeding innovation, for the established BI vendors and implicitly for the organizations that rely on their tools. But there's more to the picture than scalability, "intelligence" (whatever that is), real-time reaction, autonomicity, and webification. The more is imagination: openness to, and the ability to deliver, new ways of analyzing data and using analytical findings.

In a year when the Net is abuzz about Web 2.0, Tableau Server, out this week, qualifies as Web 0.2. But don’t get me wrong. Web 2.0 is about social media and collaboration, user-driven integration, on-demand access, and a first level of semantic search and discovery. On the back-end, Web 2.0 is about network-accessible services that enable all that stuff. Tableau Software’s first foray onto the Web is a modest step when considered in light of Web 2.0 agendas, and also in light of the very high expectations created by the company’s stand-alone Tableau Desktop application. It is not, however, a failure. Rather it shows caution, implicit care to get it (collaborative Web computing) right and not overextend and underdeliver.

It's disconcerting to live in a world where security can be seen as delivering competitive advantage, yet that's the idea behind Unisys's Enterprise Security initiative. But after all, the company's Trusted Enterprise Model only extends the security selling point that is a marketing mainstay for financial institutions and that has been adopted or embraced by IT vendors, sometimes far too slowly, with the rise of network computing.

I promised to follow an earlier article that looked at database management systems as a commodity technology with a similar assessment of business intelligence. In drafting the promised article, however, I realized that I couldn't limit my evaluation to the software side of BI.

BI is complex. It is simultaneously software, transformational work practices, and business information. Admittedly, I am going far beyond IE Editor in Chief Doug Henschen's take on BI, but consider: What value is reporting or OLAP or data mining — software — that doesn't tap all data that contributes to relevant business insights — information — that can help you restructure, realign, or optimize business operations — practices? To understand if BI is a commodity technology, we need to examine all three, complementary aspects of business intelligence: software, information, and practices.

Let's start with information, with BI sources and BI results.

A quick notice to let everyone know that the nomination period for the 2008 Jolt Awards is now open.

I'm a Jolt Awards judge, my second year, while this is the 18th go-around for the awards. The main sponsor is Dr. Dobb's, like Intelligent Enterprise a CMP computing magazine (or is IE a business magazine focusing on computing or is IE a portal rather than a magazine?)

The Jolt Awards target various aspects of software development practice: development environments, management and coding tools and utilities, libraries, books, developer networks, and lots more.

Prof. Jorge Cardoso of the University of Madeira, Portugal, has written a very interesting paper titled "The Semantic Web Vision: Where are We?" Cardoso surveyed over 600 academic and industry researchers in December 2006. He published his findings in the September-October 2007 issue of IEEE Intelligent Systems. They include that "mainstream adoption is still five to ten years away."

Cardoso defines the Semantic Web as "a machine-readable World Wide Web" and he notes "a significant evolution of standards as improvements and innovations allow the delivery of more complex, more sophisticated, and more far-reaching semantic applications." (Bill Inmon, please note.)

Is there anything to add to an item that was the rage of the political media a month back, the misuse of one of our favorite miscommunication tools, PowerPoint, by U.S. military leadership? Check out Gen. David Petraeus' September 10, 2007 slideshow explaining and justifying the drawdown of U.S. troops inserted into Iraq in the recent "surge." The bloggers (e.g., Yglesias, Benen, Drum) found particularly notable an egregiously amateurish slide that related simply that in nine months, troop levels would be back to the levels of nine months previous.

Is database management a mature, commoditized software technology? It depends whom you ask. Start with IE Editor-in-Chief Doug Henschen, who wanted my take last July after he and I each wrote on Oracle 11g. My answer: definitely!, which no doubt confirmed Doug in his intention to later write that Oracle President Charles "Phillips's higher calling was to dispel the idea that database management systems have been commoditized in a mature market."

OK, I admit that the president of Oracle speaks with more authority than I do although he perhaps speaks with more bias as well. I don't have a 47.1% share of the RDBMS market to protect.

Nor is Microsoft's 17.4% RDBMS market share my responsibility. Contrast Phillips with Microsoft Technical Fellow David Campbell, quoted in an interview published in the September issue of Database Trends and Applications magazine. DBTA asked Campbell about factors distinguishing Microsoft SQL Server and its primary competitors. Campbell's response: "For 99 percent of your information-management needs, you can get the job done with SQL Server, DB2, or Oracle." Campbell's conclusion was that "you want to choose the one where your ongoing cost of operations is lowest."

OpenOffice.org has reached a significant anniversary. Earlier this month, OO passed the five-year mark as the only office software on my laptop computers, first installed when I bought a Windows 2000 machine in 2002, reinstalled a couple of months ago on a replacement laptop running Windows Vista and Ubuntu Linux. With open-source Apache Tomcat, Cygwin, Firefox, MySQL, Python, R, and Thunderbird to keep OO company, there's been no looking back. Instead, given diverse project-health indicators such as the release of IBM's new Lotus Symphony, the reported assignment of 35 programmers to the project by IBM, and the continued evolution of the NeoOffice native version for MacOS X, I'm looking forward to my next five years of OpenOffice.org.

Complex Event Processing (CEP) seemed like a no-brainer for broad-market acceptance when I first wrote about a key constituent technology a couple of years back. Relational data warehouses and conventional analytics have not kept up with the explosive growth in real-time data volumes and the perceived demand for real-time analytics. CEP promised to fill the gap: technology developed for the extreme high-volume, low-latency processing demands of capital-market algorithmic trading and communications networks, compatible with emerging service-oriented architectures, applicable to a broad spectrum of security, logistics, and click-stream challenges. Further, CEP is supported by a vibrant, diverse community of academic and industrial researchers. IBM and Oracle and other established companies are doing very significant work, and the field has spawned half-a-dozen start-ups. Yet two years on, CEP is still struggling for market definition outside of capital markets.

I'm grateful to Actuate for giving me an preview look at BIRT Exchange, a new community site set to launch next Monday, September 24. Like the sponsoring company, the new site straddles the commercial open and closed source worlds. It will surely benefit BIRT Java programmers whether they use the open-source Eclipse version of BIRT or the closed source Actuate version. But make no mistake: Actuate's motives remain staunchly commercial and the company will retain tight control over BIRT development.

Cognos's planned takeover of performance-management (PM) and OLAP vendor Applix was eminently predictable. BI companies have been hungry for PM capabilities; think of the Actuate-Performancesoft, Business Objects-Cartesis, Oracle-Hyperion, and SAP-OutlookSoft deals. Cognos's step -- preceded by their 2003 Adaytum acquisition -- leaves Information Builders and Microstrategy as the only major BI players without deep, in-house PM solutions and Longview as one of the last remaining independent PM vendors, albeit noting their alliance with Information Builders and others.

I've been puzzling out a technique, used by sites that machine-aggregate content, that may boost pages' Google rankings. The aggregators stuff their pages with (Google) ads and contextually similar - albeit just similar enough - content. All that pseudo-content surely moves them up the Google rankings. How else to explain the success of the bottom-feeders who exploit others' content in order to sell ads?

Search has many limits but the price is right and the alternative, reliance on traditional interfaces and human-structured information, is increasingly perceived as unacceptable. Yet there's still much to be said for old-school information-retrieval methods. Witness a new, search-free IT-market research tool, TechNavio, just launched by Infiniti Research.

The appetite for Search continues to grow rapidly, and Fast Search and Transfer (FAST) has been one of the most aggressive players in the market. I've covered FAST's move to provide a contextual advertising alternative. Another company initiative boldly claims to revolutionize business intelligence. Yet FAST's recent difficulties, which appear to involve technical missteps and not just operational issues, should make us rethink the limits of search, particularly when it comes to extravagant claims about search-BI.

I received Gartner e-mail this week marketing their up-coming open-source summit. The message contains gems that illuminate Gartner's perspective on open source and the larger IT world.

I characterized Gartner as the oracle of IT establishment and looked at their summit plans in a blog entry last week. Analysts will explain the heretofore anti-establishment open-source movement, albeit without the help of representatives of the communities that lend open source its power and vibrancy. Gartner's theme -- a quite valid one -- seems to be that establishment IT needs to come to grips with open source, and of course that Gartner is the organization that can show the way. They claim to be good at it.

Yes, Oracle 11g is a blockbuster release, sure to maintain the company's dominant market position.

There couldn't be a greater contrast between the views on semantics of Web creator Tim Berners-Lee and of data warehousing figure Bill Inmon.

Gartner has announced an Open Source Summit for this coming fall. The summit will bring together, on the one hand, an analyst firm known for authoritative pronouncements on all things IT, and on the other, a disruptive model for software development that is, at its core, anti-authoritarian. The term that comes to mind is cognitive dissonance. How will the Gartner summit bridge two conflicting world views?

As Curt Monash reports in his Text Technologies blog, Voice of the Customer was a central theme at this year’s Text Analytics Summit. The aim is to stay on top of reputation, quality, and product-design issues by crunching blog- and message-board text, call-center notes and e-mail, and free-text survey responses. (Some vendors call these activities "Enterprise Feedback Management.") Yet VOC and the analytical approach it typifies are only half the overall text-analytics picture. Text analytics still delivers very high value in traditional, non-VOC application domains such as life sciences and intelligence, areas where vendors still derive the major part of their revenues.

The relationship between search and text analytics was a recurring topic at this week's Text Analytics Summit in Boston. The one supports information retrieval and the other just about anything else automated you can do with a document set, from knowledge extraction to automated classification and processing: complementary functions that rely on similar technical underpinnings. Ramana Rao, who has a wonderful ability to clarify, put it this way: "Google's white box makes everything seem so simple," but "we got to simplicity without handing the complexity of reality." It's text analytics, of course, that will equip search to handle complexity.

Retired Vice Admiral Lowell Jacoby, former director of the Defense Intelligence Agency, currently an executive at a Washington DC IT solutions provider, gives exactly the keynote presentation one would expect. He gives a keynote whose essence I've heard before: IT is a information sponge that can clean up some nasty, real-world spills. I heard this theme in 2003 when Richard Perle, former assistant secretary of defense, spoke at a Capital Hill program on data mining. It was the rationale for DARPA's ill-fated-but-resurgent Total Information Awareness program. It bespeaks an attitude that would apply IT on a massive scale in a rear-guard attempt to contain a political situation gone horribly wrong - we have to do something, right? - with not a moment's thought given to alternative paths.

Buying Inxight is a smart move for Business Objects. Business Objects has strong analytics and is well positioned in the BI marketplace. Folding the capability to extract information from text into their technology stack is a natural next step for the company.

This acquisition affirms the text-BI/integrated-analytics strategy being pursued by other vendors, notably Attensity, Clarabridge, Intelligent Results, SAS and SPSS. It also follows the precedent of data-integrator Informatica's late 2006 purchase of text-analytics vendor Itemfield. While the Itemfield deal closed less than six months ago, a demo I saw at last week's TDWI World Conference showed that Informatica has made quick work of seamless integration of text-extraction into their flagship PowerCenter ETL product. Given that Business Objects has been an original equipment manufacturing (OEM) licensee of Inxight technology for several years – which provides at least one reason they didn't go after fellow French company TEMIS – I expect similarly quick inclusion of Inxight text analytics into Business Objects' Data Integrator product.

An Accelovation briefing earlier this w