Behind the scenes at eBay are a group of analysts and researchers working with â€˜big data', Â some people join the trend of calling them data scientists. But whatever you want to call them they are developing algorithms and building platforms that are continuously propelling eBay forward by mining the company's big data, going back several years and querying it in real-time. Putting this in context eBay has over 30k categories and deals with over 300m queries per day, all generated from a simple premise on which eBay rests; "one persons trash is another person's treasure", as Dr Sundersan puts it in his Big Data World talk in London.
The data left behind in the millions of purchases happening on eBay is now big data and is being used to build and test economics models, one example from Dr Sundersan involves building better results from queries that novice users might use, based on the search terms and habits of more experienced users. In fact, the research lab at eBay found that sellers on the site were running their own experiments just like economists would, changing the initial sale price, the number of images and other factors to see for themselves how this might affect a sale. All this data is available to the researchers in eBay labs, who are developing the technology to provide better services, which is what it all boils down to. "You can't manage what you can't measure" says Dr Sundersan, and he's right. The ability to measure customers is driven in part by the potential to make money, it's like panning for gold (there's a new metaphor for big data that doesn't involve oil). Gathering vast amounts of information and refining it to create new insights that lead to better user experiences. If everyone benefits – eBay, the online seller and the online buyer – and no one has any bad experiences with privacy and security to tell (yet), then it may all be about keeping the computation of data to stay real-time with the volume as it grows increasingly bigger.
Blogged on behalf of guest blogger: Paul Booth #bdworld