This is one of the most amazing examples I’ve seen of the complexity of even simple organizational schemes. “Unicode Collation Algorithm (Unicode Technical Standard #10)” spells out in precise detail how to sort strings in what we might colloquially call “alphabetical order.” But it’s way, way, way more complex than that.

Unicode is an international standard for how strings of characters get represented within computing systems. For example, in the familiar ASCII encoding, the letter “A” is represented in computers by the number 65. But ASCII is too limited to encode the world’s alphabets. Unicode does the job.

As the paper says, “Collation is the general term for the process and function of determining the sorting order of strings of characters” so that, for example, users can look them up on a list. Alphabetical order is a simple form of collation.

Sorting inconsistent alphabets is, well, a problem. But let Technical Standard #10 explain the problem:

It is important to ensure that collation meets user expectations as fully as possible. For example, in the majority of Latin languages, ø sorts as an accented variant of o, meaning that most users would expect ø alongside o. However, a few languages, such as Norwegian and Danish, sort ø as a unique element after z. Sorting “Søren” after “Sylt” in a long list, as would be expected in Norwegian or Danish, will cause problems if the user expects ø as a variant of o. A user will look for “Søren” between “Sorem” and “Soret”, not see it in the selection, and assume the string is missing, confused because it was sorted in a completely different location.

Heck, some French dictionaries even sort their accents in reverse order. (See Section 1.3.)

But that’s nothing. Here’s a fairly random paragraph from further into this magnificent document (section 7.2):

In the DUCET, characters are given tertiary weights according to Table 17. The Decomposition Type is from the Unicode Character Database [UAX44]. The Case or Kana Subtype entry refers either to a case distinction or to a specific list of characters. The weights are from MIN = 2 to MAX = 1F16, excluding 7, which is not used for historical reasons.

Or from section 8.2:

Users often find asymmetric searching to be a useful option. When doing an asymmetric search, a character (or grapheme cluster) in the query that is unmarked at the secondary and/or tertiary levels will match a character in the target that is either marked or unmarked at the same levels, but a character in the query that is marked at the secondary and/or tertiary levels will only match a character in the target that is marked in the same way.

You may think I’m being snarky. I’m not at all. This document dives resolutely into the brambles and does not give up. It incidentally exposes just how complicated even the simplest of sorting tasks is when looked at in their full context, where that context is history, language, culture, and the ambiguity in which they thrive.

8 Comments »

The debate over whether municipalities should be allowed to provide Internet access has been heating up. Twenty states ban it. Tom Wheeler, the chair of the FCC, has said he wants to “preempt” those laws. Congress is maneuvering to extend the ban nationwide.

Jim Baller, who has been writing about the laws, policies, and economics of network deployment for decades, has found an eerie resonance of this contemporary debate. Here’s a scan of the table of contents of a 1906 (yes, 1906) issue of Moody’s that features a symposium on “Municipal Ownership and Operation.”

Click image to enlarge

The Moody’s articles are obviously not talking about the Internet. They’re talking about the electric grid.

In a 1994 (yes, 1994) article published just as the Clinton administration (yes, Clinton) was developing principles for the deployment of the “information superhighway,” Jim wrote that if we want the far-reaching benefits foreseen by the National Telecommunications and Information Administration (and they were amazingly prescient (but why can’t I find the report online??)), then we ought to learn four things from the deployment of the electric grid in the 1880s and 1890s:

First, the history of the electric power industry teaches that one cannot expect private profit-maximizing firms to provide “universal service” or anything like it in the early years (or decades) of their operations, when the allure of the most profitable markets is most compelling.

…

Second, the history of the electric power industry teaches that opening the doors to anyone willing to provide critical public services can be counterproductive and that it is essential to watch carefully the growth of private firms that enter the field. If such growth is left unchecked, the firms may become so large and complex that government institutions can no longer control or even understand them. Until government eventually catches up, the public may suffer incalculable injury.

…

Third, the history of the electric power industry teaches that monopolists will use all means available to influence the opinions of lawmakers and the public in their favor and will sometimes have frightening success

…

Fourth, and most important, the history of the electric power industry teaches that the presence or threat of competition from the public sector is one of the best and surest ways to secure quality service and reasonable prices from private enterprises involved in the delivery of critical public services.

Learn from history? Repeat it? Or intervene as citizens to get the history we want? I’ll take door number 3, please.

8 Comments »

My blogging has gone way down in frequency and probably in quality. I think there are two reasons.

First, I’ve been wrapped up in trying to plot a new book. I’ve known for about three years the set of things I want to write about, but I’ve had my usual difficult time figuring out what the book is actually about. For example, when I was planning Everything is Miscellaneous, I knew that I wanted to write about the importance of metadata, but it took a couple of years to figure out that it wasn’t a book about metadata, or a book about the virtue of messiness, or two dozen other attempts at a top line.

I’m going through the same process now. The process itself consists of me writing a summary of each chapter. Except they’re not summaries. They’re like the article version of each chapter and usually work about to about 2,000 words. That’s because a chapter is more like a path than a list, and I can’t tell what’s on the path until I walk it. Given that I work for a living, each complete iteration can take me 2-3 months. And then I realize that I have it all wrong.

I don’t feel comfortable going through this process in public. My investment of time into these book summaries is evidence of how seriously I take them, but my experience shows that nineteen times out of twenty, what I thought was a good idea is a very bad idea. It’s embarrassing. So, I don’t show these drafts even to the brilliant, warm and forgiving Berkman Book Club — a group of Berkfolk writing books — not only because it’s embarrassing but because I don’t want to inflict 10,000 words on them when I know the odds are that I’m going to do a thorough re-write starting tomorrow. The only people who see these drafts are my literary agents and friends David Miller and Lisa Adams, who are crucial critics in helping me to see what’s wrong and right in what I’ve done, and working out the next approach.

Anyway, I’ve been very focused for the past couple of months on figuring out this next book. I think I’m getting closer. But I always think that.

The second reason I haven’t been blogging much: I’ve been mildly depressed. No cause for alarm. It’s situational and it’s getting better. I’ve been looking for a new job because the Harvard Library Innovation Lab that I’ve co-directed, with the fabulous Kim Dulin, for almost five years has been given a new mission. I’m very proud of what we — mainly the amazing developers who are actually more like innovation fellows — have done, and I’m very sorry to leave. Facing unemployment hasn’t helped my mood. There have been some other stresses as well. So: somewhat depressed. And that makes it harder for me to post to my blog for some reason.

I thought you might want to know, not that anyone cares [Sniffles, idly kicks at a stone in the ground, waits for a hug].

8 Comments »

In case anyone has forgotten what honesty sounds like:

Be the first to comment »

Well, here’s what I would do if I were Shakespeare & Co., a theatre company in Lenox, Massachusetts of which I am inordinately fond, as consistent readers of this blog know (hi, Mom!).

Yesterday my wife and I went to an open rehearsal of a scene from Henry IV, Part 2, Scene 2. For about an hour we watched Malcolm Ingram (Falstaff), Kevin Coleman (Shallow), Ariel Bock (Silence) and Michael F. Toomey (Bardolph) being directed by Jonathan Epstein, who has abridged and combined the two Henry IV’s. The rehearsal started out fascinating and got even better from there.

The actors in Shakespeare & Co. rehearse before they’ve learned their lines by being shadowed by someone who whispers their lines to them. That way (as Kevin Coleman explained) they can rehearse while looking at the person they’re talking to instead of looking down at a piece of paper. The result is an early rehearsal in which the actors can act together and experiment.

Jonny Epstein is an actor and a highly collaborative director. He interceded occasionally to punch up a reading, and always kept an eye on the audience’s interests: We need a gesture to understand what “bona-robas” are (high quality courtesans — literally “the good stuff”); Falstaff should turn to the left while pointing to the right so that both sides of the audience are involved, etc.

But as the scene came to a close, it took a turn towards the awesome.

It’s a short and humorous scene in which Justice Shallow is greeting his old friend Falstaff. There’s funny business about rounding up men for Falstaff, which in this abridged, small-cast version had the actors pointing into the audience. Very amusing.

The scene ends with Shallow inviting Falstaff to dinner. They’re about to wander off, in a convenient scene-closing way, when a memory from fifty-five years ago pops into Shallow’s mind. “O, Sir John, do you remember since we lay all night in the windmill in Saint George’s field?” This becomes a chat about old acquaintances who now are old or dead.

The first time through, the actors played it lightly: a bunch of old folks remembering their lusty youths. But Epstein then suggested that they stop their funny business. Just stand there and talk. Without further direction, the actors changed everything: posture, cadence, expression, diction, interaction. And it became a scene about age and youth that touched me deeply.

It was, in short, a moment of transcendence. I got yer magic of the theatre right here.

Shakespeare & Co. is a great company, but it rarely plays to full houses. If I were them, here’s what I would do:

1. Video every lecture they give and put it on the Web for free. In fact, do more lectures, at least one for every play they produce. These lectures have been consistently fascinating. I want people to get used to looking up the Shakespeare & Co. lecture before going to see a Shakespeare play performed by any group.

2. Video a performance of each play presented, and post it for free on the Web. Have some of the summer interns do it. No one who comes would have stayed at home if they could have watched a video of it, especially since the company doesn’t have the resources to do studio-quality video production.

3. Post a second version of these videoed performances with a director’s track. Have the director and some of the actors explaining both the play and their decisions about it. We want teachers to play these scenes when introducing students to Shakespeare, and we want people who just saw a performance to then see the thinking behind it.

Now, there may be Actors Equity rules that prevent this, which would be a shame because videos like these would help expose the actors’ talents more broadly. And I suspect that Shakespeare & Co. may have reservations about posting content that’s not of the highest professional quality. If so: get over it! It’s the Web! Trust comes from imperfection.

In any case, when you’re in the Berkshires, do come. And bring the kids.

2 Comments »

We saw Shakespeare & Co.‘s Julius Caesar last night. What a rich production! And certainly not because of its production values: the performance was in the tiny Bernstein Theatre with a cast of just seven and an almost bare stage.

The acting was up to the company’s high standard. New to me was James Udom as Marc Antony. He gave the famous address — which stands out for its devious plainness amidst the torrent of language in which it is embedded — brilliantly. Eric Tucker made the noblest Roman, Brutus, human. I could listen to Jason Asprey all day long. (I embarrassed myself after the performance when he came out to the lobby.) Kristin Wold switched characters on stage instantaneously and before our eyes, nevertheless bringing us along.

It is a hard play. It never lets you settle. And it has perhaps the most despairing final words of any Shakespeare play. We may not be 100% sure that Caesar was so ambitious that he needed to be killed for Rome’s sake, but he at least had the good sense to mask his ambitions. When Octavius stands amidst the carnage and celebrates the “glories of this happy day,” we see what naked ambition truly looks like. It was a devastating moment last night.

The small audience consisted almost entirely of people over 60. Such a shame.

2 Comments »

I finally got Bob Dylan’s triple pun.

Sure, everybody must get stoned because that’s the price of being a non-conformist. Or maybe we all fail to conform to someone’s idea of normal.

And, obviously we all should get stoned, high, baked, whatever. (Stay in school, kids. Don’t do drugs. — the Management)

But, as the last stanza says:

Well, they’ll stone you when you are all alone
They’ll stone you when you are walking home
They’ll stone you and then say they are brave
They’ll stone you when you’re sent down in your grave
But I would not feel so all alone
Everybody must get stoned

Bob Dylan: ‘stoned

D’oh?

2 Comments »

Thank you, Gravity, for keeping the water — most of it — in the lake, and for making sure it reaches all the way to the bottom.

1 Comment »

During this seemingly-endless interregnum when we have e-books that suck at letting us take notes, I buy paper books when I’m doing research. I have a complex little application I’ve endlessly developed over the years that lets me type notes into a plain text editor or OPML-based outliner using a minimal markup. The app turns the notes into a database that I can then slice ‘n’ dice. Someday I’ll get it stable and done enough to publish. And that day is never.

A couple of years ago I wrote a Chrome extension (“Kindle Highlights Exporter”) that scrapes all of the passages you’ve highlighted with your Kindle, exporting them as a csv, xml, or json file. The only problem is that I seem to be the only person it works for. More precisely, it crashed for the only person I ever showed it to, my supersmart developer nephew. It still works for me, though. If you want (yet another) chance to laugh at me, feel free to download it and install it. Suckers.

So, how about if someone were to write some software that lets me import photographs of the pages of a book that I’ve highlighted in, say, yellow. The app finds the highlighted portions of each page, looks for the page number, does the requisite OCR, and returns a well-marked-up set of those annotations. (These days, outputting in the Open Annotation standard, as well as the usual suspects, would be extra cool.) That way, when I’m done with a book, I could snap images of all the pages with highlights and get a list at the end, instead of doing what I do now: type them in as I read.

I’d give it a try, but processing images is waaay beyond my hobbyist-programmer capabilities. As for the possible copyright violation: OH FOR HEAVENS SAKE WHAT THE HELL IS WRONG WITH US? (Note: The previous sentence should not be construed as legal advice.)

In any case, as the digital/networked world continues to develop its superpowers, the mud wall that confines the physical becomes more and more aggravating.

4 Comments »

I was writing a long-ish tutorial about how to use LibraryCloud that refers to bunches of Javascript. I wanted a way to refer to that code in the HTML document. So I wrote a converter that turns Javascript code (and maybe others) into HTML tables you can just plunk into your document.

Here’s an example of some code:

function init(){
// get textarea to accept tab key
// thank you http://stackoverflow.com/questions/6140632/how-to-handle-tab-in-textarea
	$("#rawjs").keydown(function(e) {
		if(e.keyCode === 9) { // tab was pressed
			// get caret position/selection
			var start = this.selectionStart;
			var end = this.selectionEnd;

			var $this = $(this);
			var value = $this.val();

			// set textarea value to: text before caret + tab + text after caret
			$this.val(value.substring(0, start)
						+ "\t"
						+ value.substring(end));

			// put caret at right position again (add one for the tab)
			this.selectionStart = this.selectionEnd = start + 1;

			// prevent the focus lose
			e.preventDefault();
		}
	});
}

And here’s what it looks like once converted into an HTML table:

There’s definitely some clunkiness to this. For example, if you click on the “toggle line numbers” button it hides or shows the line numbers for every such table in your document. (I’ll fix this eventually.) By the way, the “Toggle” button is there so your readers can select the code and not get the lline numbers.

Also, in order to get the formatting totally encapsulated within your document, I clunkily add a “<style>” section before each table. I couldn’t figure out any other way to do it without making your HTML document dependent on my server being up.

The utility is here: http://hyperorg.com/programs/convertJStoTable/convertJStoTable.html. The code itself is up at Github (but not quite up to date at the moment).

Please let me know all the ways it’s broken and sucks, although I don’t promise to do anything about it. It worked for me.

1 Comment »