Anil Dash (one of my heroes, and is also hilarious) is talking at a Library of Congress event on Digital Preservation, part of the National Digital Information Infrastructure and Preservation Program. Anil’s talk is called “Make a Copy.” (Anil is now at ThinkUp.)

Anil says he’s a geek interested in the social impacts of tech on culture, govt, and more. He started Expert Labs a few years ago to enable tech to talk with policy makers. Expert Labs built ThinkUp. He wants to talk about the issues that this group or archivists confronts every day that the tech community doesn’t know about. He warns us that this means he’s starting with depressing stuff. So…

…Picture the wholesale destruction of your wedding photos, or other deeply personal mementos. They are being destroyed by an exclusive, private, ivy league club: Facebook. FB treats memories as disposable. “Maybe if I were a 25 year old billionaire, I’d think of these as disposable, too.” “The terms of service of digital social networks trumps the Constitution in terms of what people can share and consume.” Our ordinary conversations are treated as disposable, at Facebook, Twitter, Microsoft, etc. They explicitly say that they can delete all of your content at any time for any reason. “100s of millions of Americans have accepted that. That should be troubling to those of us who care about preservation.”

You can opt out, but not without compromising your career and having severe social cost. And you can’t rely upon the rest of the Web, because “there’s a war ranging against the open Web.” “The majority of time spent on the Web in the US is spent in an application,” not on pages. Yet we’re still archiving Web pages but not those applications. “They are gaslighting the Web,” Anil says, referring to the old movie. E.g., you can leave FB comments on Anil’s blog, but when you click from FB to his blog, FB gives you a warning that the site you’re going to is untrustworthy. “I don’t do that to them,” he says, even though they’ve consistently “moved the goal posts” on privacy, and he has registered his site with FB.

After blogging this, Anil got a message from a tech at FB saying that it was a bug that’s being fixed. But suppose he hadn’t blogged it, or FB had missed it? “The best case scenario is that we’re left fixing their bugs.” He adds, “That’s pretty awful, because they’re not fixing our bugs. And we’re helping them to extend their prisons over the Web.” And is the only way to get our words preserved is to agree to Twitter’s ToS so that we’ll get archived by the Library of Congress, which has been archiving tweets. Anil says that he’s conscientiously tried to archive his own works for his new baby, but it shouldn’t rely on that much effort by an individual.

And, he says, that’s just the Web, not the apps. You can’t crawl his phone and preserve his photos. And when FB buys Instagram which has a billion photos, and only 5% of the content FB has bought has been preserved…? And yet the Instagram acquisition is considered a success by the Valley. If you’re a Pharaoh, your words are preserved. Anil is worried about the rest of the conversations.

“If I were to ask you what is the most watched form of video, what would you say?” Anil guesses that it’s animated gifs. And we don’t archive them. “We’re talking about the wrong things.” We’re arguing that we should be using Ogg Vorbis, but the proprietary forms are the ones that are most used. The standards ecology is getting more complicated. “We need to reflect back to the tech community that they have an obligation to think about preservation.” They’ve got money and resources. Shouldn’t they be contributing?

We’re losing metadata, he says. You can’t find Instagram photos because they have no Web presence and are short on metadata. Flickr, on the contrary, has lots of metadata. The Instagram owners are now multi-millionaires and are undermotivated to fix this problem. Maybe we’ll get something in 5 years, but then we will have lost a full decade of people’s photos. There’s no way to assign Instagrams open licenses at this point.

Indeed, “they are bending the law to make archiving illegal.” You can’t hack your own phone. You can’t copy your own photos from one device to another.

“Content tied to devices dies when those devices become obsolete.” The obsolesence cycle is becoming faster every year.

So, what should we do?

The technologists building these devices don’t know about the work of archivists. They don’t know that what this group is doing is meaningful. Many are young and don’t yet have experiences they want to preserve. They may not have confronted their own mortality yet.

But, the Web at its base level is about making copies. So, if we get things on the Web as opposed to in apps, we win. Apps should be powered by, or connected to, a Web experience. How can we take advantage of the fact that every time you go to a Web page, you’re copying it? How can we take advantage of the CDN’s, which are already doing a lot of the work needed for preservation?

“There is also a growing class of apps that want to do the right thing.” E.g., TimeHop, that sends you an email reminding you of what you tweeted, etc., a year ago. This puts a user experience around the work of preservation. They’re marketing the value of the preservation community, but they don’t know it yet. Or Brewster, an iPhone address book that hooks up to all the address books you have on social services, reminding you to connect with people you haven’t touched in a while. This is a preservation app, although Brewster doesn’t know.

Then, how do we mine our personal archives? (He notes that his company’s tool, ThinkUp, is in this space.) His Nike fuel band captures data about his physical activity. The Quantified Self movement is looking at all sorts of data. “They too are preservationists, and they don’t know it.”

Then there are institutions. People revere the Library of Congress. Senior people at Twitter speak in a hushed voice when they say, “The tweets go to the LoC.” Take advantage of the institution’s authority. Don’t be shy. Meet them halfway. And say, “By the way, look at my cool email address.”

“PR trumps ToS.” ThinkUp archived the FB activity of the White House. At the time, FB’s ToS forbid archiving it for more than 24 hours. But the WH policy requires it. I said, “Please, FB, please cut off the White House’.” It turns out that FB was already planning on revising the policy. “What a great conversation we would have gotten to have.” You are our advocates, says Anil. You have an obligation to speak on our behalves.

The public is already violating “Intellectual Property” rules. “We don’t look at YouTube as the Million Mixers March, but that’s what it is.” It’s civil disobedience: People violating the law in public under their own names. These are people who recognize the value of preserving cultural works that otherwise would disappear. Sony won’t sell you a copy of Michael Jackson’s Thriller, but there are copies on YouTube. The heart and soul of those posting those videos is preservation. “All they want to do is what you do: make a copy of what matters to them.”

2 Comments »

I’m at the “Symposium on Digital Curation in the Era of Big Data” held by the Board on Research Data and Information of the National Research Council. These liveblog notes cover (in some sense — I missed some folks, and have done my usual spotty job on the rest) the morning session. (I’m keynoting in the middle of it.)

Alan Blatecky [pdf] from the National Science Foundation says science is being transformed by Big Data. [I can’t see his slides from the panel at front.] He points to the increase in the volume of data, but we haven’t paid enough attention to the longevity of the data. And, he says, some data is centralized (LHC) and some is distributed (genomics). And, our networks are unable to transport large amounts of data [see my post], making where the data is located quite significant. NSF is looking at creating data infrastructures. “Not one big cloud in the sky,” he says. Access, storage, services — how do we make that happen and keep it leading edge? We also need a “suite of policies” suitable for this new environment.

He closes by talking about the Data Web Forum, a new initiative to look at a “top-down governance approach.” He points positively to the IETF’s “rough consensus and running code.” “How do we start doing that in the data world?” How do we get a balanced representation of the community? This is not a regulatory group; everything will be open source, and progress will be through rough consensus. They’ve got some funding from gov’t groups around the world. (Check CNI.org for more info.)

Now Josh Greenberg from the Sloan Foundation. He points to the opportunities presented by aggregated Big Data: the effects on social science, on libraries, etc. But the tools aren’t keeping up with the computational power, so researchers are spending too much time mastering tools, plus it can make reproducibility and provenance trails difficult. Sloan is funding some technical approaches to increasing the trustworthiness of data, including in publishing. But Sloan knows that this is not purely a technical problem. Everyone is talking about data science. Data scientist defined: Someone who knows more about stats than most computer scientists, and can write better code than typical statisticians :) But data science needs to better understand stewardship and curation. What should the workforce look like so that the data-based research holds up over time? The same concerns apply to business decisions based on data analytics. The norms that have served librarians and archivists of physical collections now apply to the world of data. We should be looking at these issues across the boundaries of academics, science, and business. E.g., economics works now rests on data from Web businesses, US Census, etc.

[I couldn’t liveblog the next two — Michael and Myron — because I had to leave my computer on the podium. The following are poor summaries.]

Michael Stebbins, Assistant Director for Biotechnology in the Office of Science and Technology Policy in the White House, talked about the Administration’s enthusiasm for Big Data and open access. It’s great to see this degree of enthusiasm coming directly from the White House, especially since Michael is a scientist and has worked for mainstream science publishers.

Myron Gutmann, Ass’t Dir of of the National Science Foundation likewise expressed commitment to open access, and said that there would be an announcement in Spring 2013 that in some ways will respond to the recent UK and EC policies requiring the open publishing of publicly funded research.

After the break, there’s a panel.

Anne Kenney, Dir. of Cornell U. Library, talks about the new emphasis on digital curation and preservation. She traces this back at Cornell to 2006 when an E-Science task force was established. She thinks we now need to focus on e-research, not just e-science. She points to Walters and Skinners “New Roles for New Times: Digital Curation for Preservation.” When it comes to e-research, Anne points to the need for metadata stabilization, harmonizing applications, and collaboration in virtual communities. Within the humanities, she sees more focus on curation, the effect of the teaching environment, and more of a focus on scholarly products (as opposed to the focus on scholarly process, as in the scientific environment).

She points to Youngseek Kim et al. “Education for eScience Professionals“: digital curators need not just subject domain expertise but also project management and data expertise. [There’s lots of info on her slides, which I cannot begin to capture.] The report suggests an increasing focus on people-focused skills: project management, bringing communities together.

She very briefly talks about Mary Auckland’s “Re-Skilling for Research” and Williford and Henry, “One Culture: Computationally Intensive Research in the Humanities and Sciences.”

So, what are research libraries doing with this information? The Association of Research Libraries has a jobs announcements database. And Tito Sierra did a study last year analyzing 2011 job postings. He looked at 444 jobs descriptions. 7.4% of the jobs were “newly created or new to the organization.” New mgt level positions were significantly higher, while subject specialist jobs were under-represented.

Anne went through Tito’s data and found 13.5% have “digital” in the title. There were more digital humanities positions than e-science. She posts a lists of the new titles jobs are being given, and they’re digilicious. 55% of those positions call for a library science degree.

Anne concludes: It’s a growth area, with responsibilities more clearly defined in the sciences. There’s growing interest in serving the digital humanists. “Digital curation” is not common in the qualifications nomenclature. MLS or MLIS is not the only path. There’s a lot of interest in post-doctoral positions.

Margarita Gregg of the National Oceanic and Atmospheric Administration, begins by talking about challenges in the era of Big Data. They produce about 15 petabytes of data per year. It’s not just about Big Data, though. They are very concerned with data quality. They can’t preserve all versions of their datasets, and it’s important to keep track of the provenance of that data.

Margarita directs one of NOAA’s data centers that acquires, preserves, assembles, and provides access to marine data. They cannot preserve everything. They need multi-disciplinary people, and they need to figure out how to translate this data into products that people need. In terms of personnel, they need: Data miners, system architects, developers who can translate proprietary formats into open standards, and IP and Digital Rights Management experts so that credit can be given to the people generating the data. Over the next ten years, she sees computer science and information technology becoming the foundations of curation. There is no currently defined job called “digital curator” and that needs to be addressed.

Vicki Ferrini at the Lamont -Doherty Earth Observatory at Columbia University works on data management, metadata, discovery tools, educational materials, best practice guidelines for optimizing acquisition, and more. She points to the increased communication between data consumers and producers.

As data producers, the goal is scientific discovery: data acquisition, reduction, assembly, visualization, integration, and interpretation. And then you have to document the data (= metadata).

Data consumers: They want data discoverability and access. Inceasingly they are concerned with the metadata.

The goal of data providers is to provide acccess, preservation and reuse. They care about data formats, metadata standards, interoperability, the diverse needs of users. [I’ve abbreviated all these lists because I can’t type fast enough.].

At the intersection of these three domains is the data scientist. She refers to this as the “data stewardship continuum” since it spans all three. A data scientist needs to understand the entire life cycle, have domain experience, and have technical knowledge about data systems. “Metadata is key to all of this.” Skills: communication and organization, understanding the cultural aspects of the user communities, people and project management, and a balance between micro- and macro perspectives.

Challenges: Hard to find the right balance between technical skills and content knowledge. Also, data producers are slow to join the digital era. Also, it’s hard to keep up with the tech.

Andy Maltz, Dir. of Science and Technology Council of Academy of Motion Picture Arts and Sciences. AMPA is about arts and sciences, he says, not about The Business.

The Science and Technology Council was formed in 2005. They have lots of data they preserve. They’re trying to build the pipeline for next-generation movie technologists, but they’re falling behind, so they have an internship program and a curriculum initiative. He recommends we read their study The Digital Dilemma. It says that there’s no digital solution that meets film’s requirement to be archived for 100 years at a low cost. It costs $400/yr to archive a film master vs $11,000 to archive a digital master (as of 2006) because of labor costs. [Did I get that right?] He says collaboration is key.

In January they released The Digital Dilemma 2. It found that independent filmmakers, documentarians, and nonprofit audiovisual archives are loosely coupled, widely dispersed communities. This makes collaboration more difficult. The efforts are also poorly funded, and people often lack technical skills. The report recommends the next gen of digital archivists be digital natives. But the real issue is technology obsolescence. “Technology providers must take archival lifetimes into account.” Also system engineers should be taught to consider this.

He highly recommends the Library of Congress’ “The State of Recorded Sound Preservation in the United States,” which rings an alarm bell. He hopes there will be more doctoral work on these issues.

Among his controversial proposals: Require higher math scores for MLS/MLIS students since they tend to score lower than average on that. Also, he says that the new generation of content creators have no curatorial awareness. Executivies and managers need to know that this is a core business function.

Demand side data points: 400 movies/year at 2PB/movie. CNN has 1.5M archived assets, and generates 2,500 new archive objects/wk. YouTube: 72 hours of video uploaded every minute.

Takeways:

• Show business is a business.

• Need does not necessarily create demand.

• The nonprofit AV archive community is poorly organized.

• Next gen needs to be digital natvies with strong math and sci skills.

• The next gen of executive leaders needs to understand the importance of this.

• Digital curation and long-term archiving need a business case.

Q&A

Q: How about linking the monetary value of the metadata to the metadata? That would encourage the generation of metadata.

Q: Weinberger paints a picture of flexible world of flowing data, and now we’re back in the academic, scientific world where you want good data that lasts. I’m torn.

A: Margarita: We need to look how that data are being used. Maybe in some circumstances the quality of the data doesn’t matter. But there are other instances where you’re looking for the highest quality data.

A: [audience] In my industry, one person’s outtakes are another person’s director cuts.

A: Anne: In the library world, we say if a little metadata would be great, a lot of it would be great. We need to step away from trying to capture the most to capturing the most useful (since can’t capture the most). And how do you produce data in a way that’s opened up to future users, as well as being useful for its primary consumers? It’s a very interesting balance that needs to be played. Maybe short-term need is a higher thing and long-term is lower.

A: Vicki: The scientists I work with use discrete data sets, spreadsheets, etc. As we get along we’ll have new ways to check the quality of datasets so we can use the messy data as well.

Q: Citizen curation? E.g., a lot of antiques are curated by being put into people’s attics…Not sure what that might imply as model. Two parallel models?

A: Margarita: We’re going to need to engage anyone who’s interested. We need to incorporate citizen corporation.

Anne: That’s already underway where people have particular interests. E.g., Cornell’s Lab of Ornithology where birders contribute heavily.

Q: What one term will bring people info about this topic?

A: Vicki: There isn’t one term, which speaks to the linked data concept.

Q: How will you recruit people from all walks of life to have the skills you want?

A: Andy: We need to convince people way earlier in the educational process that STEM is cool.

A: Anne: We’ll have to rely to some degree on post-hire education.

Q: My shop produces and integrates lots of data. We need people with domain and computer science skills. They’re more likely to come out of the domains.

A: Vicki: As long as you’re willing to take the step across the boundary, it doesn’t mater which side you start from.

Q: 7 yrs ago in library school, I was told that you need to learn a little programming so that you understand it. I didn’t feel like I had to add a whole other profession on to the one I was studying.

1 Comment »

Eric Schmidt is being interviewed by Jeff Goldberg about the Net and Democracy. I’ll do some intermittent, incomplete liveblogging…

NOTE: Posted without having even been re-read. Note note (a few hours later): I’ve done some basic cleanup.

After some amusing banter, Jeff asks Eric about how responsible he felt Google was for Arab Spring. Jeff in passing uses the phrase “Internet revolution.”

ES: Arab Spring was enabled by a failure to censure the Internet. Google enabled people to organize themselves. Especially in Libya, five different militias were able to organize their armed revolt by using the Net. It’s unfair to the people who died to call it an “Internet revolution.” But there were fewer people who died, in part because of the incessant media coverage. And we’ve seen that it’s very easy to start what some call an Internet revolution, but very hard to finish it.

JG: These were leaderless revolutions, crowdsourced revolution. But in Egypt the crowd’s leaders were easily pushed aside after Mubarek fell.

ES: True leaders are very hard to find. In Libya, there are 80 militias, armed to the teeth. In most of the countries there were repressed Muslim groups that have emerged as leaders because they organized while repressed. Whoever takes over inherits financial and social problems, and will be thrown out if they fail.

JG: Talk about Google’s tumultuous relationship with China…

ES: There are lots of reasons to think that China works because its citizens like its hierarchical structure. But I think you can’t build a knowledge society without freedom. China wants to be a knowledge society. It’s unclear if China’s current model gets them past a middle income GDP. Google thought that if we gave them free access to info, the Chinese people would revolt. We were wrong, and we moved Google to Hong Kong, on the open side of the Great Firewall. (We had to because that’s the Chinese law.) Now when you enter a forbidden query, we tell the user that it’s likely to be blocked. We are forbidden from announcing what the forbidden terms are because we don’t want employees put in jail.

JG: Could Arab Spring happen in China? Could students organize Tianamen Square now?

ES: They could use the Chinese equivalent of Twitter. But if someone organizes a protest, two people show up, plus 30 media, and 50 police.

JG: Google’s always argued that democratization of info erodes authoritarian control. Do you still believe that?

ES: The biggest thing I’ve learned is how hard it is to learn about the differences among people in and within countries. I continue to believe that this device [mobile phone] will change the world. The way to solve most of the world’s problems is by educating people. Because these devices will become ubiquitous, it’ll be possible to see how far we humans can get. With access to the Net, you can sue for justice. In the worst case you can actually shame people.

JG: And these devices can be used to track people.

ES: Get people to understand they have choices, and they will eventually organize. Mobiles tend to record info just by their nature. The phone company knows where you are right now. You’re not worried about that because a law says the phone company can’t come harass you where you’re sitting. In a culture where there isn’t agreement about basic rights…

JG: Is there evidence that our democracy is better off for having the Internet?

ES: When we built the Net, that wasn’t the problem we were solving. But more speech is better. There’s a lack of deliberative time in our political process. Our leaders will learn that they’ll make better decisions if they take a week to think about things. Things will get bad enough that eventually reason will prevail. We complain about our democracy, but we’re doing quite well. The US is the beacon of innovation, not just in tech, but in energy. “In God we trust … all others have to bring data.” Politicians should just start with some facts.

JG: It’s easier to be crazy and wrong on the Net.

ES: 0.5% of Americans are literally crazy. Two years ago, their moms got them broadband connections. And they have a lot of free time. Google is going to learn how to rank them. Google should enable us to hear all these voices, including the crazy people, and if we’re not doing that, we’re not doing our job.

JG: I googled “Syria massacre” this morning, and the first story was from Russia Today that spun it…

ES: It’s good that you have a choice. We have to educate ourselves and our children. Not everything written is true, and very powerful forces want to convince you of lies. The Net allows that, and we rank against it, but you have to do your own investigation.

JG: Google is hitting PR problems. Talk about privacy…

ES: There’s no delete button on the Net. When you’re a baby, no one knows anything about you. As you move through life, inevitably more people know more about you. We’re going to have to learn about that. The wifi info gathering by StreetView was an error, a mistake, and we’ve apologized for it.

JG: The future of journalism?

ES: A number of institutions are figuring out workable models. The Atlantic [our host]. Politico. HuffingtonPost. Clever entrepreneurs are figuring out how to make money. The traditional incumbents have been reduced in scale, but there are plenty of new voices. BTW, we just announced a tablet with interactive, dynamic magazines. To really worry about: We grew up with the bargain that newspapers had enough cash flow to fund long term investigative research. That’s a loss to democracy. The problem hasn’t been fully solved. Google has debated how to solve it, but we don’t want to cross the content line because then we’d be accused of bias in our rankings.

JG: Will search engines search for accuracy rather than popularity?

ES: Google’s algorithms are not about popularity. They’re about link structures, and we start from well-known sources. So we’re already there. We just have to get better.

JG: In 5 yrs what will the tech landscape look like?

ES: Moore’s Law says that in 5 yrs there will be more power for less money. We forget how much better our hw is now than even 5 years. And it’s faster than Moore’s Law for disks and fiber optic connections. Google is doing a testbed optical installation. At that bandwidth all media are just bits. We anticipate a lot of specialty devices.

JG: How do you expect an ordinary, competent politician to manage the info flow? Are we inventing tech that is past our ability to process info?

ES: The evidence is that the tech is bringing more human contact. The tech lets us express our humanity. We need a way of sorting politicians better. I’d suggest looking for leaders who work from facts.

JG: Why are you supporting Obama?

ES: I like having a smart president.

JG: Is Romney not smart?

ES: I know him. He’s a good man. I like Obama’s policies better.

Q&A

Q: Our connectivity is 3rd world. Why haven’t we been able to upgrade?

A: The wireless networks are running out of bandwidth. The prediction is they’ll be saturated in 2016. Maybe 2017. That’s understandable: Before, we were just typing online and now we’re watching movies. The White House in a few weeks is releasing a report that says that we can share bandwidth to get almost infinite bandwidth. Rather than allocating a whole chunk that leaves most of it unused, using interference databases we think we can fix this problem. [I think but please correct me: A database of frequency usages so that unused frequencies in particular geographic areas can be used for new signals.]

A: The digital can enhance our physical connections. E.g., a grandmother skyping with a grandchild.

JG: You said you can use the Net to shame govts. But there are plenty of videos of Syria doing horrible things, but it’s done no good.

ES: There are always particularly evil people. Syria is the exception. Most countries, even autocratic ones, are susceptible to public embarrassment.

Q: Saying “phones by their nature collect data” evades responsibility.

ES: I meant that in order to their work, they collect info. What we allow to be done with that info is a legal, cultural issue.

Q: Are we inherently critical thinkers? If not, putting info out there may not lead to good decisions.

ES: There’s evidence that we’re born to react quickly. Our brains can be taught reasoning. But it requires strong family and education.

Q: Should there be a bill of rights to simplify the legalese that express your privacy rules?

ES: It’s a fight between your reasonable point of view, and the lawyers and govt that regulate us. Let me reassure you: If you follow the goal of Google to have you as a customer, the quickest way to lose you is to misuse your information. We are one click away from competitors who are well run and smart. [unless there was money in it, or unless they could get away with it, or…]

Q: Could we get rid of representative democracy?

ES: It’ll become even more important to have democratic processes because it’s all getting more complicated. For direct democracy we’d have to spend all day learning about the issues and couldn’t do our jobs.

JG: David Brooks, could you comment? Eric is an enormous optimist…

ES: …The evidence is on my side!

JG: David, are you as sanguine that our politicians will learn to slow their thinking down, and that Americans have the skills to discern the crap from the true.

David Brooks: It’s not Google’s job to discern what’s true. There are aggregators to do this, including the NYT and TheBrowser. I think there’s been a flight to quality. I’m less sanguine about attention span. I’m less sanguine about confirmation bias, which the Web makes easier.

ES: I generally agree with that. There’s evidence that we tend to believe the first thing we hear, and we judge plus and minus against that. The answer is always for me culture, education.

Q: Will there be a breakthrough in education?

ES: Education changes much more slowly than the world does. Sometimes it seems to me that education is run for the benefit of the teachers. They should do measurable outcomes, A-B testing. There’s evidence that physics can be taught better by setting a problem and then do a collaborative effort, then another problem…

1 Comment »

Robert Putnam is giving a talk at the Aspen Ideas Festival called “Requiem for the American Dream? Unequal Opportunity in America.” It’s a project in its middle stages, he says. If a book comes out of the research, it’s a year or two out.

There’s a difference between the inequality of income and wealth, and inequality of opportunity. Historically Americans have not cared much about income inequality…less than is typical of the rest of the world. But we do care about unequal opportunity and unequal social mobility. Concern about income inequality has sometimes divided along party lines, but not opportunity inequality. Historically we’ve been better than most other countries in the distribution of opportunity.

Income distribution has become more skewed since the 1970s in America, and in many other countries. We’ve also become a more class-segregated society, even as we’ve become less segregated by race and religion. Class segregation is increasing by residence, education, organization, and marriage; i.e., it’s less likely you’ll marry outside of your class. There’s also been a fraying of family and social bonds within the working class. Robert asks if this has an effect on the growing inequality of opportunity.

He talks about his methodology. The standard way of measuring social mobility compares 30-somethings’ economic/educational/social standing with their parents’ standing when the parents were in their 30s. But this means all the action is at least twenty years old; the latest studies look at people raised in the 1980s. Robert is instead looking at today’s kids, to avoid the 20-30 year old blind spot in the rearview mirror. “If we look out the windshield, we’re about to go over a cliff when it comes to social mobility…Social mobility and opportunity are going to plummet.”

If something is important enough to write about, it ought to show up in multiple measures, he says. He will show us robust patterns in multiple data sources, focusing only on class differences. He’s only looking at white youth for now, because while racial gaps remain important, they are increasingly based on class, not race. It’s important to look at race issues, but that’s not what Robert is considering. He says if you look at race as well, the social mobility trends look even worse. So, he’s going to show us growing class gaps over the past 30 years among white kids with 2-parent families.

He shows charts. The rate of births to unmarried mothers who are college grads hasn’t changed. But the percentage of those births to women with some college and women with no college has significantly increased. About a third of the births to unmarried women are to women with some college; it’s about half for women with no college. Meanwhile, the racial gap (i.e., race controlling for class) fell dramatically while the class gap (class controlling for race) grew at about the same rate. I.e., high school educated white folks are behaving more and more like high school educated non-white folks. “I’m not saying race doesn’t matter. I’m saying class matters a whole lot more. And race matters a whole lot less.”

Another chart. ” Over the last two decades or so, white kids coming from less educated, less well-off backgrounds are more and more going through life with only one parent at home.”

A chart of the “growing class gap in enrichment expenditures [day care, tutors, games, etc., but not private school] on children, 1972-2006.” At the bottom of the hierarchy, the expenditure has increased about $400 per child over the past 40 years, but at the middle income, it’s gone up $5K.

The time people invest in their kids — reading to the kids, etc., but not including diaper changing time, etc. — again shows a growth gap between those with a higher ed and those without. In the 1970s, moms with only HS were investing slightly more time with their kids. Now the number of minutes for both is going up, the growth has been “much much faster” among college educated moms. When you add in the dads, the gap grows even larger — it’s up to an hour a day more quality time with their parents.

When in the lifecycle of the child is the class gap biggest? It’s concentrated among infants. “It’s terrible. Just terrible.”

How are kids connecting at schools? Looking at participation in extracurricular activities, and activities outside school like music lessons, dance lessons, art lessons, etc., excluding sports. (This is, he reminds us, only data about white kids, but the class gap gets worse if you add in non-white kids.) Kids in the lower income quartile have declining participation rates. Those in the highest quartile have a growing rate. (The decline began sharply in 1982.)

Same chart for participation in sports. For kids becoming team captains, it’s stayed steady for the lower quartile kids. Middle class kids were always more likely to become team captains, but now 26% of them say that they’ve been team captains. “Think about what kids are learning” from these activities: how to get along with kids, how to make connections with people who are not like them… the skill set we need in this new world. (Robert tells us HS football was invented by progressives about 100 years ago as a way to get kids from all classes playing together.)

Outside school in music, dance and art lessons: same growing gap.

There is a declining gap in participation in student government. But that’s happening in part because upper quartile kids are choosing not to participate. The other area in which the gap is declining is in “vocational clubs,” e.g., shop, motorcycle club. Again, the upper class kids are declining to participate.

Church-going: All are decreasing, but the upper third is decreasing much less rapidly. “There’s been a catastrophic drop in church attendance among children of working class parents.”

The chart of comunity volunteering is more complex, but overall the upper tercile has been rapidly increasing, while the bottom tercile has been dopping in the 2000’s. One possible explanation: Robert points out that colleges like to see community volunteering on applications.

Chart of social support: Do you have someone you can count on? Sharply increasing gap. Working class kids: it’s been pretty flat. Ask “Would you say most people can be trusted?” and you’ll see a long-term decline among the kids of parents in the bottom tercile, while the upper tercile kids have become more trusting. “And why not?” The upper class kids have plenty of social support, while the support systems are being withdrawn from the bottom tercile kids.

All this shows up in reading and math test scores. Increasing gap mapped against class. Declining gap mapped against race.

Bottom line: “There’s a growing class gap among American youth among all the predictors of success in life.” “A social mobility crash is coming” as these cohorts move into adulthood. Everyone who’s looked at the data agrees, he says.

But what has this happened? We don’t know for sure. About ten years ago he was in the White House talking with Pres. Bush, Karl Rove, and others, talking about this. (He charmingly apologizes for namedropping.) The first question W asked was “How much of this has do to family structure?” A: A little less than half. Even if you look only at 2-parent families, the gap is there but only about half of the size. None of it is due to immigration. But, suggested Robert, it might be due to the income gap. Then Laura Bush said “If you don’t know how long you’re going to keep your house or your job, you have less energy to invest in your kids.” Robert thinks this makes sense.

Possible explanations: (1) Upper class families have increased their investment in cognitive and non-cognitive development. (2) Collapse of white working class. (3) Laura Bush’s hypothesis. (4) The social safety nets are gone: churches, sports leagues, parks and rec, etc. “If the chick falls out of the nest, all that’s down there are gangs.” It is, he says, a perfect storm.

This is a problem that the two parties should be able to cooperate on, if they could cooperate about anything. We need to boost caring families, boost jobs and wages for the bottom half of the workforce, invest in public education, invest in in high quality Head Start, and have more reliable volunteer mentors. “I don’t know what else we can do to fix the problem…but if we don’t fix it we’re writing off a third of our workforce. And, it’s just not fair.” Until we think of all of these kids as our kids, “we’re in a pickle.”

Q&A

Q: Any data on kids of parents in the military?

A: I don’t have data. I wish I did because enlisted men and women are mostly drawn from the lower class, and my hunch is that their kids are doing better than non-military kids. I think that the discipline instilled into the military maybe carries over into the structure of the families.

Q: I teach HS in a rural area of OR. We got multicultural sensitivity training. I asked why aren’t we talking about class because I see it every day. I hope your work translates into teacher training.

A: Surprisingly to me, when I talk to groups, almost always when elementary school teachers speak up, they say they see this problem in their own class.

Q: Harrington, NSF, others have said the same thing over the years. This is the fourth time I’ve seen the same red flag. How does this translate into policy?

A: This particular growing gap wasn’t true in Harrington’s day. There’s always been a class gap in American society but it’s way worse than in the ’60s. I’m working on a book aimed at a mass market. We’re gathering the stories of these kids. I’m hoping that if you talk about real kids, it will get people’s attention. I desperately fear we’re going to have a partisan argument about who’s to blame, and I don’t care about that.

Q: Some will argue about the cost.

A: It’ll be much more expensive not to fix this. These lower third kids will be on unemployment and in prison….

Q: College admissions could be fixed…

A: Most of the damage is done way before then.

Q: The social safety net is under constant resource. We need govt policy and money. Our churches and philanthropies are not enough.

A: I’m a progressive Democrat so I think govt has a role to play. But this is a fundamental American issue. It does have to involve churches. I happen to think that hugs and time are more important than money, but money is important too.

8 Comments »

Tim Brown of Ideo is opening his Aspen Ideas Festival talk with a slide presentation called “From Newton to Design”. He says he’s early in thinking it through.

He points to a problem in how we’ve thought about design, trained designers, and have practiced design. The great thing about designing simple products is that you can know almost everything about them: who made them, who they’re for, how they were produced, etc. But as products get more complicated, it gets harder even for a team of designers to really understand what’s going on. They get so complicated that there are lots of places design can fail.

When we go out to urban planning , that becomes even more obvious, he says. He shows Union Sq. when it was designed and how wildly NYC has grown around it. Or, at the Courtyard Marriott chain, every element of the user’s experience has been thought through. He shows a script that specifies every interaction. But you can’t anticipate everything. E.g., JetBlue is one of the best designed customer experiences and even they got it wrong a couple of winters ago.

What’s going on? It’s all about complexity. Henri Poincaré in the 19th century tried to solve the three body problem that had been set by the French govt as an open source competition. HP couldn’t solve it. It sounds like a simple problem, but it’s very hard. [BTW, there’s a fascinating history of three French aristocrats hand-computing the movement of Halley’s Comet, which depended on calculating the gravitational influences of multiple bodies. Can’t find the ref at the moment]

Our basic ideas about design have been based on Newton, says Tim. Design assumes the ability to predict the future based on the present. We need to think more like Darwin: design as an evolutionary process. Design is more about emergence, never finished.

He presents a few principles of Darwinian design that he’s been exploring.

1. Design behaviors, not objects — the behaviors that come from our interactions with objects. If you’ve traveled on the high speed trains in Europe, there are signs urging men to be more accurate when peeing. But at Schiphol Airport, they print a fly at the right spot in the urinal; men became 80% more accurate. That’s designing behavior; the actual object doesn’t matter.

2. Design for information flow. Nicholas Christakis has looked at how networks affect behavior. Tesco uses its loyalty card — which cost them 20% of their margins — to increase sales.

3. Faster iteration = faster evolution. Viruses evolve faster than we do because they iterate faster than we do. E.g., State Farm tried out a new idea how to build relationships with the new generation. They built one storefront for this, and learned from it. “Launch to learn.”

4. Use selective emergence. This intrigues him, alathough he doesn’t know how useful it will be in design. Rather than random mutations, you choose what might be interesting and design things that get us there through many iterations. I.e., genetic algorithms. E.g., the Strandbeest walks along beaches with a hip joint unlike any in nature because the artist used genetic algorithms.

5. Take an experimental approach. I.e., testing hypotheses. Cf. Eric Ries, the Lean Startup (build, measure, learn). E.g., Ideo.org has been working on sanitation in Ghana. Where you can’t dig septic pits, Ideo has been experimenting with low cost receptacle toilets (with bio-digesters). But people didn’t want to pay for the service. So, they gave some to families and went away for three days. All the families changed their minds and said they are willing to pay for the service (which is provided by a local franchise).

6. Focus on simple rules. This comes from emergence theory. E.g., complex bird flocking patterns are based on simple rules. [Canonical example: Termite mounds.] E.g., Bi-Rite stores in SF uses simple rules: If an employee is within 10′ of a customer, you look the customer in the eye. If within 4′, you talk with them. This creates a wonderful service experience.

7. Design is never done. E.g., World of Warcraft is constantly being designed by its players.

8. The power of purpose. This creates the self-governance these complex environments succeed. Arab Spring and Occupy Wall Street are examples. Companies are experimenting with new ways of thinking about their business and products. E.g., Patagonia tells you not to buy its products because it also wants to preserve the environment.

The prototypical design artefact is a blue print. Once you created the blue print, the design was done. It was the instruction set for someone to make it. That’s how we think about design: finish and done. What replaces it: Code. It might be DNA (and Tim has people researching this), but more often it’s programming code. It’s an instruction set that can continue to evolve.

Now James Fallows [swoon] interviews him.

JF: You embody your principles. The rules are differen from a prior version. [ACK! Crash. Missed about 2 minutes]

TB: We’ve just finished designing the prototype experience for the new health care exchanges. It will affect how people choose which health care insurance to choose. Today it’s done with paper. Under the new health care laws, lots of people will get to make these choices. We worked with the CA Healthcare Foundation to prototype the user experience. What are the key pieces are parts? How can we keep the choices reasonably simple? Then each state will use this a platform to develop their own.

JF: And the govt had the wit to come to you to do this?

TB: The CA Health Care Foundation…

JF: What are the barriers? Does it cost more to do it your way?

TB: It’s often less costly. Most often they don’t have a good understanding of what their customers go through. When a health care org comes to us, relatively frequently we find out that a senior exec had to go through the health care experience. It’s true of all organizations. We don’t ask the right questions. The urgency to change is not there, and the resistance to change is always huge.

JF: Has the TSA come to you?

TB: Yes, but … well, we learned a lot. In the previous admin, we worked with them to find areas of change. Although going through the scanners has to improve, a lot of it has to do with the behavior of the people. They looked at a training program that was intended to take away some of the rule-based system they used. The more rules you apply, the less sensitive the system is. You need to give the people in that system much more independence to make judgments.

JF: Who do you hire?

TB: We look for a wide range of people. Many disciplines. We look for deep skills, and for empathy. It’s hard to solve problems for others without that. Also, most of what we do is too complex for individuals, so we work in teams, and thus people need an enthusiasm for empathy.

JF: Any unusual interview techniques?

TB: We put people into a situation in which they’re practicing design. E.g., intern program. Also, competitions. And we use Open Ideo as a way of seeing how people work.

JF: Beyond the toilet, what else are you doing for “design for poverty.”

TB: I got excited when I saw the opportunities for design in some social design work. At Open Ideo we’re working on clean water, early ed programs, etc. Ideo.org is a non-profit org. We want it to be sustainable and scalable so we look for external funding for it.

JF: How do you approach environmental sustainability?

TB: We try to build that into every project. Every project affects the environment. We try to bring sustainable thinking around systems, materials, energy flows, etc.

JF: What projects are you proudest of?

TB: The work we do in health care, including with Kaiser Permanente. Also, consumer-facing, post-crash financial services. PNC digital wallet. “Keep the change.” Etc. This is not an area where design has had much to do.

JF:

TB: For physical objects, it peaked maybe 20-30 years ago (with Apple as an exception). But we’re in ascendance for behavior-based designed. We get 25,000 apps a year for 100 openings. We’re a 600-person company. Etsy, Kickstarter, sw designed better than ever before…great things are happening. Soon if not already the number of digital designers will be greater than all other designers combined.

Q & A

Q: Your principles are so close to Buckminister Fuller’s [says the guy from the Fuller institute]. But the boundary between social and evolutionary systems is illusory.

TB: Yes, Fuller figured this out a long time ago. We’re perhaps resurrecting ideas, as every generation does. Design has operated as a priesthood for too long. When I started, I was only interested in how beautiful something is. That’s so much simpler. Opening design up to many more will convince us all that we’re all part of this big design ecosystem and have a responsibility to be thoughtful about the contributions we’re making to the world around us. I hope professional designers learn to enable that, more than controlling it. The B School at Stanford is introducing non-designers to design, which is great.

Q: What can we do to simplify the rules?

TB: The unstated bit of my thesis is that you still have to stop and design something. We develop an idea, perhaps more through iteration. That process doesn’t change. For rebuilding a complex system, maybe big data will help us to see patterns that allow us to understand what we’re designing’s complex effects…but I don’t think we’re there yet. We should be thinking about the hooks we’re building in. I’m big into APIs that allow other people to build with what you’ve built.

Q: Is it training or DNA that determines a good employee for you?

TB: Both. We hire people straight out of grad school because they’re moldable. We hire older people, but it’s harder for them to adapt. I don’t have much control as CEO. The future of all businesses is to have cultures that are a s self-governing as possible. That’s much more resilient and agile than cultures built on inflexible rule sets.

Q: I chair a land conservancy. We create parks in urban areas. Does Ideo have much experience in designing to create behaviors that will get people to use parks? What’s your view of the state of park design?

TB: We don’t have a lot of expertise in designing anything because we like designing everything. The High Line and the West Side park in NYC are remarkable examples. Projects like that show that parks can be remarkable assets to the city. We’re working with High Line on the third phase of that project. NYC’s life expectancy has gone up 3 yrs. Two explanations: People are closer to health facilities, and people walk more.

Q: What are the logistics of running a decentralized org? Mentoring? Sharing a vision?

TB: Purpose creates a sense of direction, so we talk about why the heck we’re doing what we’re doing. We think we should measure everything we do based on the impact it has on the word. We’ve done an occasionally decent job of mentoring; that can be a problem with a decentralized org. It’s a tension. Most of our employees probably want more mentoring, but we also want autonomy. We are not big believers in warehousing knowledge. Designers hate reusing other people’s ideas. It’s much better to have knowledge systems that inspire people to think in new ways. So we’re a storytelling culture. It’s a bit of an obsession of ours. If you do a piece of work, your job is to have some stories to tell about it. That’s more effective than big reports that live in a database somewhere.

(JF calls for all remaining questions)

Q: My group works with at-risk youth. Education is increasingly standards based, but your work is collaborative.

Q: How do you look at chaos? People in open markets are open and affectionate. In corporate controlled spaces, people shut down.

Q: Does form drive function or vice versa?

Q: Apple is a closed system. Google wants more control. Open vs. controlled systems?

TB: 1. University ed is not always the best way to teach entrepreneurship. Apprenticeships are interesting. 2. Great markets are vibrant, but not chaotic. I take clients to the Ferry building to point out all the interrelated pieces that make that such a great experience. It’s not top down, but you can see the patterns and use them as inspiration. 3. Form follow function? Hard to kick that notion because I believe in beautiful engineering, but most things we’re designing today have hundreds of functions, so you can’t get a single form for it. 4. I love closed systems but I think they’re inevitably part of an open system. IOS is part of an open system of everything else that I do with it. We need both. [At last! Something I disagree with! Sort of! :)]

[Fantastic. I’ve been a huge fan of Ideo’s work, and Ideo’s organizational ethos, and Tim Brown, for a long time. So I felt particularly narcissistic as I heard this talk through Cluetrain and Too Big to Know lenses. Substitute “knowledge” for “design” and you get a lot of the ideas in 2b2k. To hear them coming from Tim Brown, who is a personal idol of mine, was a self-centered thrill.]

7 Comments »

Amanda Michel, who I know from her time at the Berkman Center, is being interviewed by Matt Thompson. She’s pretty amazing: Howard Dean campaign, Huffpo’s Off the Bus, Pro Publica, and now social media at The Guardian. She’s talking with Matt Thompson from NPR.

She says that the Off the Bus effort now strikes her as surprisingly structured and profesionalized. For a year and a half, they recruited citizen journalists. Of the 12,000 of the people who participated, only 14% wanted to write articles on their own. The formalized approach Off the Bus took has been adopted by sites that invite readers to contribute their photos, their thoughts, etc.

The biggest shift, she says, is how much the campaigns rely upon data. E.g., how did Romney think he could win Iowa with just a few offices? The people who worked for him had identified die-hard supporters, who were asked to call other supporters, who were also then asked to call. In 2004, we the people were making media constantly. Now the engines driving the campaign are largely under the hood. So, if you’re reporting on campaigns today, you’re doing email analysis to understand the candidates’ strategies

Matt: It’s amazing how much media people now have woven into their days. A study shows that people are now spending 700 mins a day on media. Media is now a layer on top of people’s everyday experience. We looked at how a persistent story — a storm damaging a town — has been told throughout history. The single thing that stood out: We’ve gone from medium as an appointment you keep to media as a constant texture that both succors and buffets you.

Amanda: That’s why in 2008 we used a formalized approach — asking reporters to sign up and giving them assignments — and now people know if they go to a campaign event, they’ll be asked to post photos and twist.

Amanda: How has the shift between media and people changed?

Matt: We used to broadcast. We used to send out msgs. Now people use their mobile devices to talk with one another. We sit in this space, right alongside them. For us at NPR, that position is sweet. Radio is intimate. People can now carry us with them. That intimacy has created a drastically new dynamic for us.

Amanda: At Pro Publica, we worked on “explainers,” explaining questions people have. Readers told us they were particularly useful. I’m interested in how we can hold those in power accountable. We did the “stimulus spotcheck” to see how the economic stimulus money was being used. We asked our readers if we could tell what was going on. I asked readers to help us identify sites. Readers checked 550 sites around the country — 4.5% of construction sites aroiund the country — and we found that that gusher of work was further down the pipeline.

After making multiple phone calls, readers would sometimes say, “Journalism is hard,” which helps them understand the value of journalism.

The big challenge for media institutions is to keep their eye on the ball. The ubiquity of media can give you the false confidence that you’re seeing all there is. You’re checking Twitter, but many stories are much more difficult to find, and there are many people who don’t have a voice.

Amanda: Matt, what do you see coming?

Matt: I try to work through with the journalists the idea that we’re moving from stories toward streams. Humans have told one another stories forever, and will do so. But stories with beginnings, middles, and ends, are being augmented by the constant stream of info. Andy Carvin is constantly tracking events in the Middle East over Twitter. It’s a very different experience — no beginning, middle, end. Twitter gives you a sense of the texture of the lives of the people you follow. “We’re encountering the end of endings,” said Paul Ford. At NPR we’re trying to pull back to tell a longer story, a quest.

Amanda: There is this real need to see the context. Other trends: We’re going to be making sense of the world through the visual. We’re moving from the written word toward the image. At The Guardian, we think about how to bring people along in an ongoing process. How do you tether together items in the stream?

[Great session. My fave so far. But I’m a pretty big fan of both of these people.]

2 Comments »

Walter Isaacson is interviewing Ricardo Salinas (Mexicon media billionaire) and Richard Haass (president of the Council on Foreign Relations) about our relationship with Mexico.

Richard H. says that the media only covers drugs and guns when it comes to Mexico, but that’s a very small part of the problem. In fact, Mexico is a major economic success story. Indeed, he says, the Americas are an amazing success story: growing economies, democratic, at peace, overall. Ricardo says that his Mexican company employs 75,000, but they have 15,000 unfilled positions; Americans may emigrate to Mexico to get jobs.

Ricardo says that demand is driving the Mexican drug trade. It has become violent in Mexico, he says, because the president has said, as a policy, “Get them all.” The victims are cartel members — over 50,000 dead. Richard H. says the drug trade is more peaceful in the US because the police aren’t as corrupt. Ricardo says Mexico should adopt the US policy: raise the professionalism of the police, and focus on the violence; let the rest slip for now.

Richard H. asks if MExico will open itself up to foreign oil investors, because Mexico is an under-producer. Ricardo says the national govt gets about 30% of its revenues from oil. There will be foreign investment in extracting the oil, he says, but not in the oil territories themselves. There’s also solar, wind, and geothermal energy in Mexico.

The student movement, inspired by left wing leaders, says the media are closed to them, but that’s untrue, says Ricardo (media mogul). The media are balanced. Calderon’s electoral reform of 2007 gave political parties 3 mins of air time every hour of radio and TV for free. That’s “theft,” Ricardo says.

After the upcoming election, it’s 5 months of waiting. Also, presidents are limited to one 6-year term. Might either of these change? Ricardo says that the five month waiting period should be changed. But there’s no political possibility of changing the one-term rule.

Nicholas Burns: Presidents Bush and Obama have put a lot of their eggs into Brazil as the primary political partner in the Americas. Does that underestimate Mexico? Mexico is a regional power.

Ricardo: Mexico is such a friend that they don’t even take us into consideration. The relationship with Mexico is really good. Huge amounts of investment. The traffic in goods and services is huge. Everything is working fine. Richard H. says US-Mexican trade is 5x US-Brazilian trade. Brazil will probably double its oil output over the next few years, but they have problems with corruption. Ricardo says that Mexico has not been active in external affairs because it’s been distracted by its drug mafia. Richard H.: Our lack of a regional trade policy has been a mistake. We haven’t gone beyond NAFTA the ways that we should. NAFTA is one of the reasons Mexico’s economy survived the meltdown. And the next strategic area of partnership will be energy. Put together Canada, US, and Mexico and you get 18M barrels of oil a day.

Cuba? Ricardo: the poor people there are in a terrible situation. And the American blockade — an absurd thing — is the main reason. Castro would have been out a long time ago.

Q: What was your prior election — so close — like for osmeone in the private sector?

Ricardo: The election was so close because the PRI put forward a terrible candidate. Andres Manuel Lopez Obrador, the Democratic REvolution Party candidate, did a good job as mayor of Mexico City, and would do fine as president.

Q: Landscape for emerging entrepreneurs?

Ricardo: A year ago we started a project for women entrepreneurs. In one year, we’re up to 900,000 women. Entrepreneurs tend to go underground because it’s too hard to be legal and pay taxes and social security. It’s complicated to set up a new business. Forms, taxes. We should simplify business for small corporations. Richard H. adds that it’s become simpler over the past few years.

Ricardo: People say we charge too much. But it’s for small amounts. 80% of $100 is only $80.

WI: That’s a lot.

Ricardo: For small loans, the transaction costs are high.

WI: ou need to fix that to get more entrepreneurship.

Ricardo: You fix it by making more credit available.

Ricardo: The govt has not made a level playing field for telecom companies. And how are we going to bridge the digital divide except via mobile?

Q: How’s the tourism business been affected?

Ricardo: You’re missing something good and you should go. Nothing bad will happen to you. They’re not going after tourists. Best golf in the world.

Q: Why is it still under travel advisory from the State Dept?

Haas: Some areas are more dangerous, e.g. Monterey. The State Dept. is conservative.

Ricardo: Mexico is a very large country. It’s totally different in different places.

Be the first to comment »

I’m at an event put on by Sogeti, in Bussum, about 30 km outside of Amsterdam. Sogeti is a technology consulting company of about 20,000 people. Last night on the way to a dinner event, Michiel Boreel the CTO, explained that the company markets itself in part by holding events designed to provoke thought and controversy. At today’s event, they have a guy from IBM talking about Big Data, Andrew Keen, Luciano Floridi, me, and others. At tomorrow’s event, they are having a debate about whether Big Data is good or bad for you. (Disclosure: They’re paying me for speaking.)

Andrew Keen is giving the final speech of the morning. He’s going to talk about the themes of his book, Digital Vertigo, especially as they apply to Big Data.

“Real time is yesterday’s news,” he says. We’re into Web 3.0, he says. What does that mean? Paraphrasing Robert Scoble: the bartender knows what you want before you order. “The future arrives before we know it.” (He refers to his recent op-ed at CNN.com.)

He says he calls his book Digital Vertigo because the future is being scripted by Alfred Hitchcock. The premise is that Hitchcock’s Vertigo gives us a preview of what life is like in the age of Big Data. “It’s a movie about watching and being watched.” “Jimmy Stewart is us in the age of Big Data.” “Surveillance and voyeurism…a little preview from Hitchcock of the age of exhibitionism” In the Age of Big Data weve fallen in love with the idea that more we make public, the happier we will become.” People like, um, me (i.e., DW) and the Berkman Center are responsible for fooling us into thinking that the more together we are, the happier we are.

He plays a bit of The Social Network, when Sean Parker says, “We lived on farms, then we lived in cities, and now we’re going to live on the Internet.” Up through Web 2.0 the distinction between the real and virtual was clear. Now some authors (James Gleick) say that we are made of data. Many companies are in the business of collecting our data and enabling us to distribute ourselves and to define ourselves as data. People (he cites Loic Le Meur) are recording everything about themsevlves — his weight, his exercise runs, etc. “All these apps are designed to record, callibrate, intepret ourselves.” The location apps could have been invented by Orwell. The app Highlight keeps tabs on where we are. It aggregates our data.

He plays a bit of The Truman Show. “We’re all starring in the age of big data as ourselves…There’s no difference between private and public life.” “We have the collapsing of the public and private.” “Privacy is being destroyed. Many people in Silicon Valley say this is a good thing.”

“What’s behind this? Part of it is what I would call Digital Narcissism.” Andrew went to the Parthenon and found that no one was looking at the ruins because they were too busy photographing each other. The Age of Big data is an ideal complement to the Age of Narcissism, just as Jimmy Stewart fell in love with a fake blonde. “All love stories end badly. I’m British, not American.”

“Visibility is a trap,” said Foucault, says Andrew. “I’m not saying we should turn off all our devices, ” but visibility is a trap in three ways: 1. We, the innocent, are in fact the victim. The apps are collecting our data and selling it to advertisers, although they deny that. Eric Schmidt has said that he wants Google in 5 years to know what we want better than we do. 2. Even if we’re living in a post-1984 world, there still are governments whose eyes get big when they see they can know everything about us, telling us they’re fighting “absurd things such as terrorism.” Did social media bring down Mubarek? Yes, but there’s a darker side: 3. We’re watching ourselves. We’ve become little brothers.

History is repeating itself. He cites Bentham’s panopticon. Bentham thought if we all watched one another, it would aid progressive causes.

We need to do what Jimmy Stuart did: He sees the truth. We need to draw a line in the sand. “I’m not against some elements of the transparent network.” We’ve fallen in love with the idea that we become more human the more we distribute ourselves. “The problem with social media is that it’s not making us human. It’s doing away with the complexity of who we are.” Human essence is premised on secrecy, mystery. Individualism requires us to be alone. It does not require us to be in this perpetual social environment. Wozniak invented the personal computer by shutting himself in a room. If you want to bring the most out of your people, you need to put walls up in your office. You need to give people the space to develop their own ideas. You need to take them off the network.

We’ll finally be able to predict our own deaths. We need an alternative ending. We need to rethink the age of big data. We need government action. “I’m not a 20th century Stalinist. I’m not say the govt has to shut these companies down. But we need regulation.” We need apps that are premised on privacy and there are some. We need to rely on tech, e.g., some that’s being developed that allows data to degenerate. We need most of all to teach the Net how to forget. The Net is immature. It needs to learn how to forget. If data could fade away like writing, then the Net would be habitable. But now it is inhabitable. It is not a place fit for humans.

Andrew shows the end of the Truman Show where Truman realizes he’s on a TV set and he escapes. We need to discover that here’s a world beyond the network. Truman disappears into the darkness. That’s what we need to do in the age of big data. We need individually to discover that black space, where we can retire, where we can really work on ourselves as unique individuals. We’re born in that darkness and we die in it. The Net is a deception. We can civilize and humanize it. But we need collectively to work on it. [Collectively? Like on the Net?]

Q&A

Q: Do we have a right to be forgotten? Is it a right?

A: Brandeis wrote we have this as a core right because privacy allows us to build our individuality. I’m not a legal scholar, so I don’t know.But I do think the govt can’t legislate it. We have to be careful that this doesn’t turn into censorship.

Q: What’s worse than no regulation is bad regulation.

A: Clearly someone from Silicon Valley. The Net should be legislated like any other medium. I’m ambivalent about enforcing the right to forget. I’ve failed many times, but the business of America is reinvention. With a medium that doesn’t forget, then you can’t reinvent himself. Even Mark Zuckerberg reinvented himself. Facebook’s Timeline writes a narrative of our lives. I wrote an aggressively negative article about this and got 20,000 FB Likes.

Q: Who in the room sees mainly the positive side of Big Data? The negative side? [Very few hands go up for either side.]

A: The purpose of my work is not to trash the Internet; it’s to have us think more carefully.

Q: What is the positive side of big data?

A: The positive is that it enables people who have mastered themselves to improve that mastery. If you use medical apps to chart your weight and fitness, these platforms to reinvent yourself as a brand , enable us if we’re mature and responsible to improve the quality of our lives. The problem is that most people aren’t using social media that way. The biggest problem with big data is that it turns us into ones and zeroes. Bentham thought we can quantify everything about ourselves. The real way to happiness is not through data. [True. The positive side: Bentham quantified as a way to equalize interests across classes.]

During the break, Andrew and I had a lively conversation. In brief, we agree that we don’t trust social networks like (and especially) Facebook to handle our data in ways that reflect our interests. And where we fundamentally disagree is in our assessment of how humans flourish. Andrew emphasizes the individual. I can only see individuals as social creatures. That of course over-simplifies the discussion and the idea, but, well, I’m over-simplifying.

5 Comments »

Dries Buytaert [twitter:Dries] , the founder of Drupal and co-founder of Acquia, is giving a Berkman lunch talk about building and sustaining online collaborations.

Drupal is an open source content manager, Dries says. In the past twelve years, Drupal has “grown significantly”: 71 of the top 100 universities use it, 120 nations use it, the White House uses it, 2 of of the 3 top music companies use it, the King of Belgium uses it. [Dries is Belgian :) ] The NY Stock Exchange is converting from a proprietary Java solution to Drupal. Five of the 6 top media companies use it. One out of 50 wesbites run on Drupal. Drupal has 10,000+ modules, 300,000 downloads a month, 1.5M unique visitors a month at drupal. org. And it’s free as in beer.

Today he’s going to talk about: history, open source, community, the evolution of software, and how to grow and sustain it.

History

Dries began writing Drupal in his dorm room, more or less by accident. He wrote a message board for the Linux project, in part to learn PHP and MySQL. About a year later he released Drupal 1.0 as open source, as “a full-featured content management/discussion engine…suitable to setup a news-driven comunity or portal site similar to kuro5hin.org and slashdot.org” (as it said in the original annoucement). “It took me about 30 seconds to come up with the name Drupal, a terrible name.”

Three years later (v.4.1) he says it still looked “pretty crappy.” Two years laer,in 2005, 30 develoeprs showed up for the first DrupalCon, in Antwerp. There are now several year. By 2011, it was looking quite good, and 3,200+ developers showed up at DrupalCon. There are now weekly meetings around the world.

There were growing pains, he says. He tells us about The Big Server Meltdown. In 2004, the servers failed. Dries put up a blank page with a PayPal button to raise $3,000 for a server. Within 24 hours, they’d raised $10,000. One of the CTOs of Sun shipped him a $8,000 machine. Then Open Source Labs in Portland OR offered to house the servers. “That’s just one anecdote. In the history of Drupal, it feels like we’ve had hundreds of these.” (There are currently 8 staff members. They organize conferences and keep the servers up. )

But, Dries says, this shows a weakness in open source: you suddenly have to raise $3,000 and may not be able to do so. That’s a reason he started Acquia, which provides support for Drupal.

Open Source

Drupal is open source: It’s gratis, anyone can look at the source code, they can modify the code, and they can share it. The fact that it’s free sometimes let’s them win bids, but open source “is not just a software license. It’s a collaboration model.” “Open source leads to community.” And “ultimately, that leads to innovation.”

Dries shows photos of the community’s embrace of Drupal (and its logo). “Drupal is successful today because of the community.”

Q: How do we know there will be enthusiastic support a few years down the road? How do we know it won’t have a Y2K problem?

A: There isn’t an easy answer. Things can go wrong. We try to keep it relevant. We have a good track record of innovation and keeping the right trends. And a lot of it comes down to keeping the community engaged. We have a large ecosystem. They volunteer their time, but the are all making money; they have an economic interest in keeping Drupal relevant.

Community

“Drupal doesn’t win just because it’s cheaper. It wins because it’s better.” It is technically superior because it has thousands of developers.

Evolution of software

Dries points to a common pattern: From innovation to bespoke systems to products to commoditization. In each step, the reach becomes wider. Proprietary software tends to stop at the products stage; it’s hard to become a commodity because proprietary software is too expensive. This is an important opportunity for open source.

Growing large projects

Is Drupal’s growth sustainable? That’s a reason Dries founded the Drupal Association, a non-profit, in 2006. It helps maintain drupal.org, organizes events, etc. But Drupal also needs companies like Acquia to get it into new areas. It needs support. It needs people who can talk to CIOs in large companies.

Open source Joomla recently hired some developers to work on their core software, which has led some of the contributors to back off. Why should they contribute their time if Joomla is paying some folks? [Joomla’s experience illustrates the truth of the Wealth of Networks: Putting money into collab can harm the collab.] Drupal is not going to do that. (Acquia develops some non-open source Drupal tools.)

IBM and RedHat are the top contributors to Linux. What companies might make that sort of strategic investment in Drupal? Instead of one or two, how about hundreds? So Dries created “Large Scale Drupal,” a membership org to jointly fund developments. It’s new. They contribute money and get a say in where it’s spent. The members are users of Drupal. E.g., Warner Music. Module developers can get funded from LSD. Two people run it, paid by Acquia. There has not been any pushback from the dev community because there’s no special backdoor by which these projects get added to the Drupal core. In fact, the money is then spent to fund developers. Dries sets the technical roadmap by listening to the community; neither the Drupal Association or LSD influences that.

Of these collaborative projects often start as small, volunteer-driven projects. But then they become institutionalized when they grow. Trade routes are like that: they were originally worn into the ground, but then become driven by commercial organizations, and finally are governed by the government. Many others exhibit the same pattern. Can open source avoid it?

Q&A

If you’re thinking of starting an open source commercial company, you could do dual licensing, but Drupal has not made that choice.

Q: How much does Drupal contribute to the PHP community?
A: A little. There are tribes: some are active in the PHP tribe, others in the Drupal tribe. It’s unfortunate that there isn’t more interaction. Dries says he’d love to grow Acquia enough so that it can put a couple of people on PHP, because if PHP isn’t successful, neither is Drupal.

Q: Governance?
A: We don’t have a lot of decision-making structure. I’ve always been opposed to formal voting. We work through discussion. We debate what should be in the core. Whoever wants to participates in the debate. Ultimately we’re structured like Linux: there are two people who are committing changes to a core version of Drupal. For every major version I pick someone to work alongside me. When we release the version, he or she becomes the maintainer of it. I move on to the next version and select someone to be my co-maintainer. The 15,000 modules are maintained by the community.

Q: Do your biggest contributors agree to programming standards?
A: We are strict about our coding and documentation standards. I make the final decisions about whether to accept a patch. Patches go through a workflow before they reaches me.

Q: What advice would you give to someone trying to attract people to a project?
A: If people can make money through your project, it will grow faster. We built a community on trust and respect; we make decisions on technical merit, not dollars. We have a darwinian model for ideas; bad ideas just die. See what rises to the top. Include it in the next version. Then put it into the core, if it’s worth it. The down side is that it’s very wasteful. I could tell people “If you do x, it will get in,” but I try to get out of the way. People have taken Drupal in sorts of directions, e.g., political campaigns, elearning platforms, etc.

Q: [me] How important are you to Drupal these days?
A: I think I’m more important as the face of Drupal than I used to be. In the governance sense I’m less important. I was the lead developer, the admin for the servers, etc., at the beginning. The “hit by a bus factor” was very risky. Nowadays, I don’t write code; I just review code. I still have a lot of work, but it’s much more focused on reviewing other people’s work and enabling them to make progress. If I were to die, most things would continue to operate. The biggest pain would be in the marketing . There are a lot of leaders in Drupal. One or two people would emerge or be elected to replace what I do.

Q: What’s hard for Drupal?
A: One of our biggest risks is to keep nimble and lean. It takes longer to make decisions. We need to continue to evolve the governance model to encourage us to accelerate decision making. Also, we have some real technical issues we need to address, and they’re huge projects. Volunteers can only accomplish so much. LSD is perfectly positioned to tackle the hardest problems. If we did it at the pace of the volunteers, it would take years.

Be the first to comment »

Filipe L. Heusser [pdf] is giving a Berkman lunchtime talk called “Open Data for Open Accountability.”

How is the open Web been changing accountability and transparency? Filipe is going to share two ideas: 1. The Web is making the Freedom of Information Act (FOIOA) obsolete. 2. An open data policy is necessary to keep freedom of information up to date, and to move toward open accountability.

Lots of people praise transparency, he says. There are multiple systems that benefit from it. Felipe shows a map of the world that shows that most parts of the world have open government policies, although that doesn’t always correlate with actual openness. We continue to push for transparency. One of the cornerstones of transparency policy is freedom of information regulation. In fact, FOIA is part of a long story, going back at least back to 1667 when a Finnish priest introduced a bill into the Swedish parliament. [Entirely possible I heard this wrong.]

Modern FOI laws require governments to react to requests and to proactively provide information. (In response to a question, Filipe says that countries have different reasons for putting FOI laws in place: as a credential, to create a centralized info system (as in China), etc.), etc. Felipe’s study of 67 laws found five clusters, although overall they’re alike. One feature they share: They heavily rely on reactive transparency. This happens in part because FOI laws come out of an era when we thought about access to documents, not about access to data. That’s one big reason FOI laws are increasingly obsolete. In 2012, most of the info is not in docs, but is in data sets.

Another reason: It’s one-way information. There’s no two-way communication, and no sharing. Also, gatekeepers decide what you can know. If you disagree, you can go to court, which is expensive and slow.

In May 2009, data.gov launched. The US was the first country to support an open data policy. Sept. 2009 the UK site launched. Now many have, e.g., Kenya and the World Bank. These data are released in machine-readable formats. The open data community thinks this data should be available raw, online, complete, timely, accessible, machine processable, reusable, non-discriminatory and with open licenses.

So, why are these open data initiatives good news? For one thing, it keeps our right to FOI up to date: we can get at the data sets of neutral facts. For another, it enables multiway communication. There are fewer gatekeepers you have to ask permission of. It encourages cheap apps. Startups and NGOs are using it to provide public service delivery.

Finally, Felipe runs an NGO that uses information to promote transparency and accountability. He says that access to open data changes the rules of accountability, and improves them. Traditional gov’t accountability moves from instituational and informal to crowd-source and informal; from a scarcity of watchdogs to an abundance of watchdogs; and from an election every four years to a continuous benchmark. We are moving from accountability to open accountability.

Global Voices started a project called technology for transparency, mapping open govt apps. Also, MySociety, Ushahidi, Sunlight Foundation, andCuidadano Inteligente (Felipe’s NGO). One of CI’s recent apps is Inspector of Interests, which tries to identify potential conflicts of interest in the Chilean Congress. It relies on open data. The officials are required to release info about themselves, which CI built an alternative data set to contrast with the official one, using open data from the Tax and Rev service and the public register. This exposed the fact that nearly half of the officials were not publishing all their assets.

It is an example of open accountability: uses open data, machine readable, neutral data, the crowd helps, and provides ongoing accountability.

Now Felipe points to evidence about what’s going on with open data initiatives. There is a weird coalition pushing for open data policies. Gov’ts have been reacting. In three years, there are 118 open data catalogs from different countries, with over 700,000 data sets. But, although there’s a lot of hype, there’s lots to be done. Most of the catalogs are not driven at a national level. Most are local. Most of the data in the data catalogs isn’t very interesting or useful. Most are images. Very little info about medical, and the lowest category is banking and finance.

Q: [doc] Are you familiar with miData in the UK that makes personal data available? Might this be a model for gov’t.

Q: [jennifer] 1. There are no neutral facts. Data sets are designed and structured. 2. There are still gatekeepers. They act proactively, not reactively. E.g., data.gov has no guidelines for what should be supplied. FOIA meets demands. Open data is supplied according to what the gatekeepers want to share. 3. FOIA can be shared. 4. What’s the incentive to get useful open data out?
Q: [yochai] Is open data doing the job we want? Traffic and weather data is great, but the data we care about — are banks violating privacy, are we being spied on? — don’t come from open data but from FOIA requests.
A: (1) Yes, but FOI laws regulate the ability to access documents which are themselves a manipulation to create a report. By “neutral facts” I meant the data, although the creation of columns and files is not neutral. Current FOI laws don’t let you access that data in most countries. (2) Yes, there will still be gatekeepers, but they have less power. For one thing, they can’t foresee what might be derived from cross-referencing data sets.
Q: [jennifer] Open data doesn’t respond to a demand. FOI does.
A: FOI remains demand driven. And it may be that open data is creating new demand.

Q: [sascha] You’re getting pushback because you’re framing open data as the new FOI. But the state is not going to push into the open data sets the stuff that matters. Maybe you want to say that WikiLeaks is the new FOI, and open data is something new.
A: Yes, I don’t think open data replaces FOI. Open data is a complement. In most countries, you can’t get at data sets by filing a FOI request.

Q: [yochai] The political and emotional energy is being poured into open data. If an administration puts millions of bits of irrelevant data onto data.gov but brings more whistleblower suits than ever before,…to hold up that administration as the model of transparency is a real problem. It’d be more useful to make the FOI process more transparent and shareable. If you think the core is to make the govt reveal things it doesn’t want to do, then those are the interesting interventions, and open data is a really interesting complement. If you think that you can’t hide once the data is out there, then open data is the big thing. We need to focus our political energy on strengthening FOI. Your presentation represents the zeitgeist around open data, and that deserves thinking.
Q: [micah] Felipe is actually quite critical of data.gov. I don’t know of anyone in the transparency movement who’s holding up the Obama gov’t as a positive model.
A: Our NGO built Access Inteligente which is like WhatDoTheyKnow. It publishes all the questions and responses to FOI requests, crowdsourcing knowledge about these requests. Data.gov was the first one and was the model for others. But you’re right that there are core issues on the table. But there might be other, smaller, non-provocative actions, like the release of inoffensive data that lets us see that members of Congress have conflicts of interest. It is a new door of opportunity to help us move forward.

A: [juan carlos] Where are corporations in this mix? Are they not subject to social scrutiny?

Q: [micah] Can average citizens work with this data? Where are the intermediaries coming from?
A: Often the data are complex. The press often act as intermediaries.

Q: Instead of asking for an overflow of undifferentiated data, could we push for FOI to allow citizens’ demands for data, e.g., for info about banks?
A: We should push for more reactive transparency

Q: [me] But this suggests a reframing: FOI should be changed to enable citizens to demand access to open data sets.

Q: We want different types of data. We want open data in part to see how the govt as a machine operates. We need both. There are different motivations.

Q: I work at the community level. We assume that the intermediaries are going to be neutral bodies. But NGOs are not neutral. Also, anyone have examples of citizens being consulted about what types of data should be released to open data portals?
A: The Kenya open data platform is there but many Kenyans don’t know what to do with it. And local governments may not release info because they don’t trust what the intermediaries will do it.

1 Comment »