Published on [Permalink]

“Inside Palantir, Silicon Valley’s Most Secretive Unicorn”:

“Here’s the dirty secret of all of these data-analytics solutions,” a former Pentagon research manager told me. “They all claim to take these disparate data sources and put them together and then discover these amazing correlations between variables. But the problem is that all of these data sets are terrible. They’re dirty.” Many types of information, after all, are gathered and processed by humans. It may be entered inconsistently or provided in wildly different formats or riddled with inaccuracies. It’s messy, like the real world it reflects and records, and it doesn’t always fit into software with any sort of mathematical precision. When I saw a recent demonstration of Palantir software, it became clear that this dirty secret isn’t very secret. The interface struck me as user friendly, something anyone with basic computer literacy could figure out. Want to know how many aircraft are available for a specific mission and how long it will take them to get to their destination? With a simple query, Palantir can tell you. Then I was shown a data set on military personnel, which had to be “cleaned up” to make it usable on Palantir. It wasn’t only a magic code doing the cleanup; it was human beings — and even locating someone who could explain what needed to be done had proven time consuming. “It took many calls to find a subject-matter expert,” one person involved told me. It sounded a lot like Rooms Full of People.