The New York Times has a very interesting story [sub required] about how the Department of Homeland Security is going to invest $2.4 million over the next three years to fund Cornell, The University of Pittsburgh and the University of Utah building a “sentiment analysis” system.
The goal of the program is to create a software system that can monitor and analyze what foreign newspapers and journalists are saying, and therefore thinking, about the United States. Researchers are going to test the system on “hundreds or articles published in 2001 and 2002 on topics like President Bush’s use of the term ‘axis of evil'”.
There are two concerns I have after reading through this particular story, however: first, I don’t necessarily expect Homeland Security to understand the blogosphere, but surely someone at one of these three universities could have said “uh, guys, what about tracking blogs?” My other concern is that they haven’t done their homework and identified companies that already have content analysis systems in place, like Colorado firm Umbria.
It’s this lack of industry savvy that constantly makes me sigh when I read about government-funded (read “our tax money funded”) research projects. Now maybe I’m wrong and these college researchers are already well informed about the state of the art here in the blogosphere, but it’s darn hard to imagine it.
As the story explains: “The new software would allow much more rapid and comprehensive monitoring of the global news media, as the Homeland Security Department and, perhaps, intelligence agencies look “to identify common patterns from numerous sources of information which might be indicative of potential threats to the nation,” a statement by the department said. It could take several years for such a monitoring system to be in place, said Joe Kielman, coordinator of the research effort.”
Now compare that with the description of one of the key services offered by Umbria: “Sentiment and Satisfaction Analysis: Umbria provides insights into the discussion and the context of discussion about brands and whether or not the discussion is positive, negative or neutral.”
It’s not quite the same, but it’s darn close. And a little birdie tells me that there are even more startups that’ll be introducing semantic and content analysis systems to track the millions of new articles added to the blogosphere every day.
Project supervisor Joe Kielman of the Department of Homeland Security sounds like the kind of guy who doesn’t spend much time on the blogosphere, frankly. He’s officially Science Advisor to the Under Secretary for Science and Technology within the Department of Homeland Security, which is quite impressive, but his prior experience is that he “worked for 20 years at the FBI, where he was successively chief of the Advanced Technology Group, chief of the Research and Engineering Unit in the Engineering Section, and Chief Scientist for the Information Resources Division.” [ref]
Joe, before you and the Department spends $2.4 million on this project, it’d make a lot of us taxpayers happy to know you’re aware that:
- Just about every newspaper now has an RSS feed (including foreign papers)
- RSS feeds are easily aggregated
- There are many tools to monitor these aggregated feeds
- Vendors already have tools that do semantic analysis of large bodies of text
Maybe I’m just being picky, but I’d rather see the $2.4 million spent on new research and new tools, not reinventions of existing commercial systems.
What do you think?
I’ll do it for 1/2 that! I can start tomorrow.
Jim
Well howabout doing something about china? They’re running us into the ground despite the Dow Jones reaching a new peak. China is freaking taking over Tibet for crying out loud!!! TIBET! NOTHING BUT BUDDHISTS!! I think China, Korea, and a few others need to be taken down a notch, along with the members of our government. So tell me why it’s necessary to spend a few billion dollars worrying about other countries instead of taking care of your own? Think Bush would setup coastal defences? Anti-air ANYTHING? I mean, why can’t the US just close itself up and take care of itself before it takes care of anyone else? Great Britan has money, China has money, Japan has money, Taiwan has money… Why don’t we see them air-dropping free stuff to undeveloped countries? Can’t get water in Africa? Here! Sell this Sony HDTV and get a lifetime supply of bottled water! Like seriously wtf? The US government’s expenditure on the dumbest stuff should be reflected in creating an increasingly larger hole in the senate, house of reps, white house, and pentagon. Need i even mention Area 51? There should be black holes where these centers of government are, because that’s how much heart (and grey matter) they have there.
No coastal defences!!! Come on! how freaking naked can we be here!?!?
Will you be my friend Dave? I don’t think I could have written about this better. Your little birdie is right regarding the startups. We did a beta demo with BuzzLogic last week, and have been talking to others in the space as well.
It would be nice to know if Dr. Science over at Homeland Security did any real research beyond the University selection committee on this.
It does seem to point out a gap in Homeland Security’s analysis and selection committees. I would also point out that you left out another local (Boulder, CO) company devoted to extracting sentiment from blogs, message boards, and news. That is Collective Intellect (http://www.collectiveintellect.com) with a fairly successful track record in the Financial Service information industry!
Also, you can check out my response to the Times article at: http://timwolters.blogspot.com/2006/10/software-being-developed-to-monitor-us.html
cheers,
~
Tim Wolters
CTO
Collective Intellect
Maybe you should have read the article more closely: the project is to monitor foreign newspapers, most of which are in other languages and many of which aren’t online.
OTOH, so they have to start from scratch instead of licensing the technology of a company that has a proven track record analysing large data sets and calculating “sentiment”, etc. from that data? Sorry, it’s my tax money at work too, and $2.4 million is a lot of money for what they’re seeking to accomplish…
There may be a possibility that in the long run – it would be better to OWN the technology and develop it internally – because it may have been seen as a permanent asset to the government, and a technology with tremendous potential to be developed and expanded into other security-related areas.
Amen Dave, good write up and GREAT comments.
They need to use “Umbria” to put their finger on the pulse of fed-up Americans who are sick of our swiss-cheese borders, and monitor the “sentiments” of the terrorist-recruiting jihadist websites. Chertoff is an obliviot.