The Human Face of Big Data

There's a lot of excitement this morning about the Human Face of Big Data project. It's an attempt to use a smartphone data capture tool to "help measure the world." From September 25 to October 2 participants in what the organizers are calling a "crowdsourced media project" will provide data on their lives, families, sleep, trust, dating and dreams. The idea is to illustrate some of what we can learn about ourselves by aggregating data. There are a lot of practical uses for this kind of data aggregating; you can see some of them by clicking through a series of screens here. There are some cool videos here; you can download the app here if you decide to participate.

I've written before about self-quantifiers, though I'm thinking about participating in this one. The idea of finding a data doppelganger, someone who matches your data closely, which is one of the promised payoffs, is appealing. (Your identity, and your doppelganger's will not be revealed.) The production team was responsible for the Day in the Life series. Judging by the photos and video that's already been produced, the pictures will be beautiful. The screenshot above? It's a photograph showing pizza deliveries in Manhattan.

I have a couple of analysis questions: even with the hoped-for 10 million downloads, is that enough to represent a world with a population of more than seven billion people? And what about people who download the app but contribute only a few data points? There's also some reason to be cautious. According to news reports, the project will not collect identifying information, but the privacy policies have not yet been posted. What do you think? Are you participating? Share your experience in the comments.

I'm coming late to this, I realize, but if you haven't already read it, take a look at Nate Silver's article about weather prediction in the NY Times Magazine of September 9. It's a good explanation of the uncertainty that goes into making predictions based on statistical models, and the assumptions and corrections that are required.

Silver demonstrates how dramatically weather predictions have improved over the last few decades. For an example, you can see my post from last year about the predicted path of Hurricane Irene here, and a follow up post here.

The article is an excerpt from Silver's forthcoming book "The Signal and the Noise: Why So Many Predictions Fail - but Some Don't."

Grammar and employment

If you've ever wondered why grammar matters to some people, this Harvard Business Review column by Kyle Wiens provides a good explanation. His reasons? Grammar, especially when your first contact is in writing (cover letter, resume, website) is how you present yourself to the world. As he puts it:

In blog posts, on Facebook statuses, in e-mails, and on company websites, your words are all you have. They are a projection of you in your physical absence. And, for better or worse, people judge you if you can't tell the difference between their, there, and they're.

And second, grammar requires attention to detail, something that's important in most jobs. Wiens says:

Grammar signifies more than just a person's ability to remember high school English. I've found that people who make fewer mistakes on a grammar test also make fewer mistakes when they are doing something completely unrelated to writing — like stocking shelves or labeling parts.

 If you're interested in exploring the grammar wars further, you can't do better than Fowler's "Modern English Usage". There's also "Eats Shoots and Leaves" by Lynne Truss. What are your favorites?

Conversion shortcut

Here's a math shortcut, new to me, from the very cool Futility Closet blog: you can use consecutive Fibonacci terms to convert miles to kilometers (and back, using the preceding Fibonacci numbers.) For example, 3 and 5. There are about 5 kilometers in 3 miles. To convert back, just reverse: there are about 3 miles in 5 kilometers. It's an estimate, good for figuring things on the fly, not exact.

And it works for numbers that are not Fibonacci numbers - express the original as a sum of Fibonacci numbers, do the conversion for each separately, and add them up. Here's an example from Peteris Krumins' coding blog:

For example, how many kilometers are there in 100 miles? Number 100 can be expressed as a sum of Fibonacci numbers 89 + 8 + 3. Now, the Fibonacci number following 89 is 144, the Fibonacci number following 8 is 13 and the Fibonacci number following 3 is 5. Therefore the answer is 144 + 13 + 5 = 162 kilometers in 100 miles. This is less than 1% off from the precise answer, which is 160.93 km.

And if you're interested, that blog also explains why it works.

Facebook roundup

I published a post a couple of months ago about the high expectations around Facebook's stock trading price right around the time of its IPO. This week there have been a couple of interesting articles analyzing what happened then and since:

Andrew Ross Sorkin, in a NY Times column, blames David Ebersman, Facebook's CFO for "Facebook's catastrophe of an initial public offering." Sorkin goes on to say, about Ebersman:

 He signed off on the ever-increasing offer price, which ended up at $38 after the company had originally planned a price range of $28 to $35.
He — almost alone — pushed to flood the market with 25 percent more shares than originally planned in the final days before the offering. And since then, as the point person for investors, he has done little to articulate how or why the company’s strategy will lift the stock price any time soon.

It's pretty powerful. And yet.

Here, from Business Insider, is a view why what happened might have been not so bad, at least from Facebook's perspective. Facebook took advantage of demand - and as a result the company has a lot of cash available. And here is a column about how the employees feel about the drop in stock price: not so bad.

And finally, here is Henry Blodget arguing that the market got excited, but if investors had read Facebook's IPO prospectus or Zukerberg's letter to shareholders, they would have learned some useful information. For example

• Facebook's CEO had a nearly unprecedented amount of control over the company.
• Facebook's CEO had set up this astounding level of control intentionally.
• Facebook's social mission is more important to Mark Zuckerberg than Facebook's business.
• Facebook's business exists to support Facebook's product development, not the other way around.

There's more--it's worth reading the entire column. What do you think?

Arctic Sea Ice Melt

The graph above is the best visual depiction of the extent of the Arctic Sea ice melt I've found. This year, more ice was lost, earlier, than in all the previous years. The illustration shows pretty clearly the difference compared with five years ago and with the 1979-2000 average. The shaded gray area is two standard deviations from that average - and the 2012 line is well below it.

Why does it matter? Well, it's a pretty good sign that the planet is warming, especially, as you can see from the graph, that the loss of ice is also happening earlier in the season. Sea ice is already floating, so it won't raise sea levels. Now forecasters predict that the Arctic Sea will be ice-free in the summer in the next 15 years. And there will likely be other consequences, including further melting of the Greenland ice sheet. And that will raise sea levels. See more details here.

For more information about the extent of the melt, and comparisons with older data, check out the National Snow and Ice Data Center's Arctic Sea Ice page.





Image via: http://nsidc.org/arcticseaicenews/

The ironclad Monitor and social impact bonds

There's a lot of interest in my earlier post about social impact bonds, and I'll be keeping an eye on the topic and providing updates. In the meantime, here's a link to an interesting column from the Harvard Business Review blog (free when you register) pointing out the special funding arrangements required of the designer of the ironclad ship the Monitor because of:

how challenging the Monitor was to the Navy establishment when its inventor John Ericsson and his partner Cornelius Bushnell asked them to approve its design: not only was it "ironclad" unlike the wooden ships of the era, it sailed almost wholly submerged, with only its strange gun turret cresting the waves.
Here's the key passage for our purposes:

Only after more deft politicking by Mr. Bushnell and more explanations from Mr. Ericsson did the board approve it, and only then with special conditions protecting the Navy. It doled out the $275,000 cost in installment payments, and made them all conditional on the Monitor proving itself in a "test"—meaning an actual battle with the enemy.

The authors speculate as to why a funding method that was used during the Civil War was not used much in the interim but has become a creative tool now.

Arguably, today's public sector has become so generally risk-averse that an arrangement designed for an extreme risk situation —dire military threat, drained war chest, dodgy inventor — now is needed for even the surest bets. Meanwhile, perhaps the ranks (and wallets) of socially-minded investors have swelled so dramatically that it's much easier to round up private capital to take flyers on potential game-changers.

Maybe the explanation is somewhat simpler, having to do with large salaries and low taxes. It's a provocative question in any case, and adds a little more color to an interesting picture.