Tag Archives: big data

Telefonica and Big Data

Posted on by

In addition, he notes there are a number of analytics experiments underway at different operating units, including Vivo in Brazil, which is pushing ahead with projects around location analysis, Web navigation analysis based on deep packet inspection (DPI) data, and call center message analysis.

via . Telefonica Battles Big Data Hype

I found the mention of DPI in their big data strategy rather interesting as well as location analysis.  The kind of location analysis a telecom operator can perform on vast populations is mind boggling.

Telco Analytics Firm Raises $30M

Posted on by

Scranage also notes that, unlike other data collection and analysis systems, Guavus’s software screens the vast volumes of data created constantly by networks, devices and subscribers for particular data types, stripping out vast quantities of data that provide no insight and feeding only “useful” data into analytics engines.

via Light Reading – Telco Analytics Firm Raises $30M.

Military turns to ESPN to help analyze drone footage

Posted on by

The amount of video streaming into this base, one of a number of sites that monitors and analyzes the images, is immense. Drone video transmissions rose to 327,384 hours last year, up from 4,806 in 2001.

Given the huge amount of feeds, the Air Force has launched an aggressive effort to seek out technology or techniques that will help them process video without adding more people to stare at monitors.

via Military turns to ESPN to help analyze drone footage.

For Riot Games, Big Data Is Serious Business

Posted on by

Once Riot Games opened up a European base of operations, it couldn’t fit all its data into one instance of mySQL. “So we created a separate instance. That was a bad precedent and we needed to change that,” Livingston added. “We moved quickly to Hadoop as a scalable low-cost storage system. We use Hive to overlay an SQL-type interface on top of the Hadoop File System.” That helped scale up, but “the downside is that it takes a long time to spin up to do your queries, some taking a minute or more to complete, so it is difficult to iterate and build complex queries using Hive.”

via For Riot Games, Big Data Is Serious Business.

Part of the challenge is to maintain a level playing field for all players, yet constantly tweaking game play and game mechanics to make it more interesting for returning players: “We need lots of insight so that competitive play will continue to happen. We don’t want different versions of the game for pros and noobs, for example.”

Obama Wins: How Chicago’s Data-Driven Campaign Triumphed

Posted on by

For all the praise Obama’s team won in 2008 for its high-tech wizardry, its success masked a huge weakness: too many databases. Back then, volunteers making phone calls through the Obama website were working off lists that differed from the lists used by callers in the campaign office. Get-out-the-vote lists were never reconciled with fundraising lists.

via Obama Wins: How Chicago’s Data-Driven Campaign Triumphed | TIME.com.

The new megafile didn’t just tell the campaign how to find voters and get their attention; it also allowed the number crunchers to run tests predicting which types of people would be persuaded by certain kinds of appeals.

Twitter, PayPal reveal database performance

Posted on by

Cole revealed that Twitter’s MySQL database handles some huge numbers — three million new rows per day, the storage of 400 million tweets per day replicated four times over — but it is managed by a team of only six full-time administrators and a sole MySQL developer.

via Twitter, PayPal reveal database performance – Software – Technology – News – iTnews.com.au.

Daniel Austin, a technology architect at Paypal, has built a globally-distributed database with 100 terabytes of user-related data, also based on a MySQL cluster.

Austin said he was charged with building a system with 99.999 percent availability, without any loss of data, an ability to support transactions (and roll them back), and an ability to write data to the database and read it anywhere else in the world in under one second.

Shopping or browsing on Main St? India’s Big Data firms know

Posted on by

The business of storing, decoding and analyzing unstructured data – think video, Facebook updates, Tweets, Internet searches and public cameras – along with mountains of facts and figures can help companies increase profits, cut costs and improve service, and is now one of the world’s hottest industries.

It’s called Big Data, and although much of the work is done in the United States, India is getting an increasing slice of the action, re-energizing an IT sector whose growth has begun to falter.

via Shopping or browsing on Main St? India’s Big Data firms know – chicagotribune.com.

New Apache project will Drill big data in near real time

Posted on by

Because Hadoop uses MapReduce to perform data queries, searches have to be done in batches. So, while you can perform highly detailed analysis of historical data, for instance, one area you would not want to use Hadoop for is transactional data. Transactional data, by its very nature, is highly complex and fluid, as a transaction on an ecommerce site can generate many steps that all have to be implemented quickly.

Nor would it be efficient for Hadoop to be used to process structured data sets that require very minimal latency, such as a Web site served up by a MySQL database in a typical LAMP stack. That’s a speed requirement that Hadoop would poorly serve.

via New Apache project will Drill big data in near real time | ITworld.

Expanding supported query languages will be one area of focus for the Drill project. Another will be adding support for additional formats, such as JSON, since right now Dremel only supports the Google Protocol Buffer Format.

NSA Mimics Google, Pisses Off Senate

Posted on by

But the NSA also saw the database as something that could improve security across the federal government — and beyond. Last September, the agency open sourced its Google mimic, releasing the code as the Accumulo project. It’s a common open source story — except that the Senate Armed Services Committee wants to put the brakes on the project.

via NSA Mimics Google, Pisses Off Senate | Wired Enterprise | Wired.com.