There’s a temptation when it comes to big data to regard it as an infallible holy grail that provides us with the kind of insights that are impossible to glean in any other way.  Of course, that isn’t always the case, and I covered a recent study by academics at UC Berkeley saying just that.

“If you want to know what individuals feel or how they become sick, you have to conduct research on individuals, not on groups,” the researchers say. “Diseases, mental disorders, emotions, and behaviors are expressed within individual people, over time. A snapshot of many people at one moment in time can’t capture these phenomena.”

It’s a conclusion that is shared by a second study, undertaken by researchers from Columbia Business School.  The paper addresses some of the challenges, and indeed opportunities, when data gleaned from social media is used to construct psychological profiles of customers.

“Big data usage is quickly evolving,” the authors say. “With technological advances in the collection, storage and analysis of large amounts of data, businesses can now gain valid insights on millions of consumers as they go about their daily lives.”

An accurate window?

There has been a huge amount of publicity given to the ability for sites like Facebook to accurately profile its users, with advertisers then paying big bucks to be able to target advertising incredibly accurately.  The belief is that by combining what we do with who we are presents a very rich picture of us as individuals.

“One benefit of psychological profiling is that the pre-selection of ads based on psychological needs can alleviate the problem of choice overload,” the authors explain. “It can even help target highly neurotic individuals who display early signs of depression with ads that guide them to self-help pages or offer professional advice.”

The authors believe that alongside these opportunities however are a number of clear ethical challenges, not least as such algorithms increasingly draw inferences about our intimate psychological characteristics.  The authors suggest that more needs to be done to highlight the value of the data being collected about each of us.

Despite various ethical challenges however, they are nonetheless largely optimistic about the potential for social data to provide a comprehensive consumer profile that will lead to significantly more effective and tailored marketing campaigns that are even capable of anticipating needs and desire both in real time and potentially even ahead of time.