Skip to main content

So, hypothetically speaking, let's say you want to design a system where you had absolute control over which applications your customers were allowed to install on your device. Certainly you would want to ensure that you were the only source for applications. But for extraordinary cases, you might also need to create a blacklist of applications.

Each entry in the blacklist would also need a human-readable title -- presumably the name of the app -- and perhaps even a human-readable description to explain why the app was blacklisted. But each entry would also need a unique identifier, of course, so you don't accidentally get confused between six apps named "TODO." Finally, you would probably want to include the date that the entry was added to the list.

Furthermore, since you anticipate continually adding new applications to this blacklist to protect your and your partners' business model, you would need your proprietary non-browser-based client to periodically poll the list for changes.

All of which raises a very serious question: what data format should you use for the list?

If you answered "JSON" then congratulations, you win the Trendy Tech of the Month Award lose! To collect your prize, please proceed through the door marked "This way to the egress." Some restrictions apply.

Update: OK, OK, it's a "Core Location" blacklist. Big deal. I'll see your tree and raise you a forest:

... an independent engineer discovered code inside the iPhone that suggested iPhones routinely check an Apple Web site that could, in theory trigger the removal of the undesirable software from the devices.

Mr. Jobs confirmed such a capability exists, but argued that Apple needs it in case it inadvertently allows a malicious program -- one that stole users' personal data, for example -- to be distributed to iPhones through the App Store.

As I've said before, "protecting users from malicious programs" is code for "cryptographically enforcing restrictions on applications to protect our and our partners’ business model." The bullshit about "stealing personal data" is just a rhetorical sleight of hand, like the RIAA claiming that piracy hurts "artists and other rights holders" when 99% of artists don't own the rights to their own songs. How many apps has Apple de-listed over privacy concerns? Only one that I know of, and it was quickly reinstated after a quick update. How many apps has Apple de-listed (or prevented being written in the first place) to protect their business? Lots and lots.

§

This is a letter I sent to my father to explain what it means that Microsoft is pulling support for MSN Music. Tech issues like this often bubble up into the media that he reads, but they are rarely explained well. My father assumes I have an opinion on such stories, and he is rarely wrong.

Actually, it is still technically in the future tense. The day the music dies will be August 31, 2008.

But first, some backstory.

It was the Dark Ages, around 2004 or so. The iTunes Store was new and booming. Microsoft, in its bid to be the center of everything without having to deal with pesky "end users", decided that the way to fight Apple was to create a developer platform. This developer platform would handle all the technical details of ensuring that people could "purchase" music files from a variety of online vendors, and play these music files on their (Windows) PC or on a variety of handheld music players. This developer platform would also ensure that such "purchased" music files could not be copied. This involves a lot of fancy math (encryption) which Microsoft was happy to license to companies running online music stores and companies making handheld music players, as well as including by default in all modern versions of Windows.

Bruce Schneier, a famous cryptologist -- or at least as famous a cryptologist as cryptologists are likely to get in this century -- once described attempts to make digital bits uncopyable as "trying to make water not wet."

Microsoft named this developer platform "PlaysForSure", and they (and their partners) ran many, many ads decrying the fact that music purchased from Apple's iTunes Music Store would "only" play in iTunes and on iPods. This was, technically speaking, true -- and indeed it is still true, and it is why I have cautioned Dora and you and anyone else who would listen that you should never "purchase" anything from the iTunes Music Store that you might want to "own" longer than Apple was willing to allow. Nor should you "purchase" anything from a "PlaysForSure"-compatible music store, and for the same reasons, only with the word "Apple" crossed out and "Microsoft" written in in crayon.

To their credit, if that's the right word, you can now purchase some music from the iTunes store that is unencrypted and plays anywhere. Apple calls these songs "iTunes Plus", because it sounds so much better than calling everything else "iTunes Minus." Apple has also promoted podcasts and other non-traditional sources of "things you might want to download onto our handheld devices where we make all of our money." Steve is many things, but he is not an idiot.

To demonstrate the awesomeness of their developer platform, Microsoft opened their own online store, MSN Music, so they could compete directly with their business partners who also offered "PlaysForSure"-compatible music downloads. Because there's nothing end users love more than fake choices.

The Electronic Frontier Foundation (EFF) -- to whom I donate money every year because they are the digital embodiment of Tom Lehrer's description of folk singers as "the people who get up on stage and come out in favor of all the things that everyone else in the audience is against, like peace and justice and brotherhood and so on" -- has also been warning anyone who would listen that they should not "purchase" encrypted music from these services, since if these services go under then all that "purchased" music will no longer... what's the word... "play". But mostly people ignored them (and me), because, you know, Microsoft was at the center of it all, and nobody ever got fired for "buying" from Microsoft. Or something.

So what happens on August 31, 2008? On that day, Microsoft will turn off the servers that they maintain for the sole purpose of validating that the songs that people have already "purchased" through MSN Music are still theirs to play. Those people (hereafter "the victims") will not notice the change right away. The victims will only notice it when they purchase a new computer, or when they upgrade the operating system on their current computer, or when the hard drive in their computer dies and needs to be rebuilt/reinstalled. At that point -- transferring the music files they have "purchased" to another drive or a new computer -- the Microsoft music player running on the victim's PC (like iTunes, but all Microsoft-y instead of Apple-y) will make a call to Microsoft's validation servers to verify that the music files were legitimately purchased. This call will fail, since the servers are not responding, since Microsoft has intentionally turned them off. The Microsoft music player will then conclude, incorrectly but steadfastly, that the music files were downloaded illegally and that the victim is a filthy pirate, and it will refuse to play them. In this case, the left hand knows exactly what the right hand is doing: they're both giving you the finger.

It is at this point that I am reminded of one classic call that I fielded when I worked at the AT+T Relay Service. One Friday night, a deaf person called Pizza Hut to, well, I don't know, but probably to order a pizza of some kind, and the guy answered the phone with "Pizza Hut, we're out of dough... can I help you?" Can you make me a pizza? No, we're out of dough. Do you make anything else? No. Then you can't help me! Does your music player play this music I "purchased"? No. Does your music player do anything other than play music? No. Then you can't help me either.

Outside the EFF, a few of the smarter industry analysts (not this guy) have been predicting this doomsday scenario for a while. In 2006, Microsoft tacitly admitted that its PlaysForSure strategy wasn't working when they announced that they were going to sell their own handheld music player (the "Zune", which competes with the iPod... and with all the other handheld music players from Microsoft's "PlaysForSure" business partners) and start a second music service (which would directly compete with the iTunes Store... and Microsoft's "PlaysForSure" business partners... and Microsoft's own MSN Music store). End users, it turns out, aren't so bad after all; they just can't be trusted to make the right choices.

Also, to ensure that no one could screw this one up except Microsoft, this new music service and new handheld music player would use an entirely new encryption system that was incompatible with "PlaysForSure", and the encryption system would not be available for licensing. Any victim who had "purchased" music through Microsoft's old MSN Music store had no upgrade/migration path to transfer those music files to their new Microsoft Zune; the victim would have to re-purchase the same music all over again. But the victims were assured that their existing MSN Music "purchases" would continue to work as long as they owned "PlaysForSure"-compatible devices. Except now they won't, because Microsoft is turning off the servers that verify that the music they "purchased" a long time ago is still theirs to play.

As you might expect, the EFF is just bursting with joy at the prospect of rubbing salt in the wound and saying "I told you so." This is their "I told you so" letter. I would join in their jubilation, but frankly I'm tired of being right all the time. It was fun for a while, but now it's just depressing.

§

Act I: The act of buying

When someone buys a book, they are also buying the right to resell that book, to loan it out, or to even give it away if they want. Everyone understands this.

Jeff Bezos, Open letter to Author's Guild, 2002

You may not sell, rent, lease, distribute, broadcast, sublicense or otherwise assign any rights to the Digital Content or any portion of it to any third party, and you may not remove any proprietary notices or labels on the Digital Content. In addition, you may not, and you will not encourage, assist or authorize any other person to, bypass, modify, defeat or circumvent security features that protect the Digital Content.

Amazon, Kindle Terms of Service, 2007

Act II: The act of giving

[I]f he lent her his computer, she might read his books. Aside from the fact that you could go to prison for many years for letting someone else read your books, the very idea shocked him at first. Like everyone, he had been taught since elementary school that sharing books was nasty and wrong...

Richard Stallman, The Right to Read

[Y]ou can't give them as gifts, and due to restrictive antipiracy software, you can't lend them out or resell them.

Newsweek, The Future of Reading

Act III: The act of lending

As you may have read in the newspapers over the past few days, we've been criticized by the leadership of a small, but vocal organization because we sell used books on our website. This group (which, by the way, is the same organization that from time to time has advocated charging public libraries royalties on books they loan out) claims that we're damaging the book industry and authors by offering used books to our customers.

Jeff Bezos, Open letter to Author's Guild

Libraries, though, have developed lending procedures for previous versions of e-books -- like the tape in "Mission: Impossible," they evaporate after the loan period -- and Bezos says that he's open to the idea of eventually doing that with the Kindle.

Newsweek, The Future of Reading

Act IV: The act of reading

It was terribly dangerous to let your thoughts wander when you were in any public place or within range of a telescreen. The smallest thing could give you away. A nervous tic, an unconscious look of anxiety, a habit of muttering to yourself -- anything that carried with it the suggestion of abnormality, of having something to hide. In any case, to wear an improper expression on your face… was itself a punishable offense. There was even a word for it in Newspeak: facecrime...

George Orwell, "1984", Book One, Chapter 5

The Device Software will provide Amazon with data about your Device and its interaction with the Service (such as available memory, up-time, log files and signal strength) and information related to the content on your Device and your use of it (such as automatic bookmarking of the last page read and content deletions from the Device). Annotations, bookmarks, notes, highlights, or similar markings you make in your Device are backed up through the Service.

Amazon, Kindle Terms of Service

Act V: The act of remembering

Another possible change: with connected books, the tether between the author and the book is still active after purchase. Errata can be corrected instantly. Updates, no problem.

Newsweek, The Future of Reading

Day by day and almost minute by minute the past was brought up to date. In this way every prediction made by the Party could be shown by documentary evidence to have been correct; nor was any item of news, or any expression of opinion, which conflicted with the needs of the moment, ever allowed to remain on record. All history was a palimpsest, scraped clean and reinscribed exactly as often as was necessary.

George Orwell, "1984", Book One, Chapter 3

Act VI: The act of learning

If they can somehow strike a deal with textbook publishers, I could see a lot of college students switching to this. Get rid of all your text books and have this single electronic device.

Ankit Gupta

School policy was that any interference with their means of monitoring students' computer use was grounds for disciplinary action. It didn't matter whether you did anything harmful -- the offense was making it hard for the administrators to check on you. They assumed this meant you were doing something else forbidden, and they did not need to know what it was.

Students were not usually expelled for this -- not directly. Instead they were banned from the school computer systems, and would inevitably fail all their classes.

Richard Stallman, The Right to Read

Your rights under this Agreement will automatically terminate without notice from Amazon if you fail to comply with any term of this Agreement. In case of such termination, you must cease all use of the Software and Amazon may immediately revoke your access to the Service or to Digital Content without notice to you and without refund of any fees.

Amazon, Kindle Terms of Service

§

This is not the usual "DRM sucks, piracy rocks" screed. Everybody in my regular audience knows about DRM (I've been talking about it since my very first post), and everybody knows you can illegally download pirated material for free (until you get caught). There's interesting stuff beyond that, if you're willing to learn.

Disclaimer: I have neither bought nor pirated this movie. Seriously. I saw it in the theater, and that was more than enough. Everything here is based on my personal analysis of public information.

Here's the NFO for a pirate release of "Pirates of the Caribbean: Dead Man's Chest." The NFO gives us:

  • Title
  • Genres
  • Plot summary
  • Exact theatrical release date
  • Running time
  • User-submitted rating
  • Link to movie page on IMDB
  • Source (retail Blu-ray)
  • Video codec
  • Video bitrate
  • Video resolution
  • Video framerate
  • Audio codec
  • Audio language
  • Audio bitrate
  • Subtitle languages (note: this release features subtitles in 11 languages)
  • Recommended playback software

Granted, most of the information about the original movie is stolen from IMDB. (What did you expect? They're pirates.) But the rest of the metadata -- video, audio, subtitles -- is unique to the release itself (more on this in a minute).

Some thoughts about the IMDB link. I have seen NFOs without a lot of this metadata, but I have never seen an NFO without an IMDB link. First, it serves as a unique indentifier in the case where the movie title is not unique (not a problem here, but think about "Hamlet" or "Pride and Prejudice" which have been remade a dozen times). Second, it effectively offloads the "research-y" part of the decision-making process. Commercial distributors can't afford to link to IMDB because it's a competitor (it's owned by Amazon and offers links to buy things on Amazon). Which is a shame, because it's packed with information -- everything from cast biographies to famous quotes to movie trailers. I often get lost in IMDB in the same way I get lost on Wikipedia, except without ending up reading about famous fictional dogs of the 1930s.

Some thoughts about video. Video is outrageously complex and technical, even when you're doing it correctly and you have no commercial interest in confusing your customers. On my desk is a plain old DVD that I rented from the local video store over the weekend; it proudly proclaims to be "MASTERED IN HIGH DEFINITION!" Whatever that means, it does not mean that it's anything but a plain old DVD. The marketing of "high definition" content that actually is high definition (at least, higher than plain old DVDs) is even worse. Will you get high definition video out of your Blu-ray disc? That depends on your player, your TV, the cables in between, and the phase of the moon. Is your cable TV high definition? That depends on what the meaning of the word "is" is. And so on.

It is against this backdrop that I appreciate the specificity of the video and audio codec information in this NFO. It is doubly relevant when you consider that pirates can and do redistribute stolen content in a wide variety of formats. You can only legitimately buy "Pirates of the Caribbean" in a preselected number of vague categories, generally limited by disc format (Blu-ray, HD-DVD, or DVD -- but don't forget "full screen" vs. "wide screen" vs. "Ultimate Edition" with enhanced audio). But pirates can choose to redistribute content in an infinite number of formats, each with its own features and strengths. Plain old DVDs can be re-encoded into a file that fits on a single CD (700 MB), two CDs, one DVD-R (4.4 GB), or untouched. High-definition discs like Blu-ray and HD-DVD can be re-encoded to fit on one DVD-R (4.4 GB), two DVD-Rs, untouched, or somewhere in between. Or they can be encoded to play on specific devices like PSPs, video iPods, or standalone DivX players. And that's not even accounting for audio formats. (Some high definition discs only contain a new audio format that open-source video players don't understand, so pirates combine high definition video with the matching audio track from a plain old DVD. Ingenious!)

This NFO tells us that this particular release comes from a retail Blu-ray disc, has a very high video quality at the same dimensions as the original, and features English audio in the (old, compatible) 5.1 DTS format. You will only be able to play it on an extremely fast computer and a high resolution monitor (maybe one of those new MacBook Pros with a 1920x1200 screen), but if your hardware can manage it, this will basically be the ultimate movie experience on your desktop.

Lesson 1: don't bullshit me. 1080p is 1080p; 720p is 720p. You'd be surprised how many "average" customers know the difference. Ironically, it was the last generation of your marketing bullshit that forced us to learn.


So that's what the pirates offer, in a nutshell. Highly technical, information-rich, and, of course, completely illegal.

Here's what Best Buy's product page offers people wanting to buy the same movie:

  • Title
  • Disc format (Blu-ray)
  • Genre (breadcrumbs)
  • Plot summary

That's it. No other information about the movie -- not even a running time! No information about audio -- not even whether it's in English! No information about subtitles. No links to additional information, even on their own site.

Lesson 2: There is no shelf space on the Internet. I need to know more than just the title before I plonk down $35 for a movie, and you have infinite space to display it. If you don't have the information I want, find someone who does and link to them. (And if you do have it, why the hell are you hiding it from me?)


Here's what Walmart.com offers:

  • Title
  • Two incompatible listings of disc format

I'm not even going to bother listing the rest. (It's funny though. "Product in Inches (L x W x H): 5.5 x 0.5 x 7.44." Wow, thanks Walmart!)

Let's go back to the part about the disc format. Two incompatible listings? WTF is he talking about? See for yourself: in the movie title it mentions "Blu-ray," but in the details below it says "Format: DVD." In the plot summary it mentions "Blu-ray," but in the ultra-small print below it says (again) "Format: DVD." Apparently no one had permissions to add another row in that "format" database table, so they're making up for it every which way they can. Result: utter confusion.

Before you say I'm nitpicking, keep in mind that Walmart.com lists the disc format as "DVD" twice in their search results (along with the movie title, which still says "Blu-ray"). So they recognize, at some level, that the disc format is important; they just don't get it right.

Lesson 3: In this world of intentionally incompatible, mutually exclusive formats, make it clear what you're selling. By the time I figure out you sold me the wrong thing, I probably can't return it. I will only be burned by this once, and I will never forget who burned me.


Barnes and Noble fares somewhat better:

  • Title
  • Disc format
  • Genres
  • Plot summary & reviews
  • Major actors
  • Theatrical release date (year)
  • Running time
  • User-submitted comments and ratings
  • List of extra on-disc features
  • Director, cast, and same-site links for more information

They mention the disc format up-front and give you a link to "learn more about formats." They even let you search by format, which is simultaneously a nice touch and a sad necessity. The plot summaries are high quality, and their origins are clearly labeled ("Barnes & Noble," "All Movie Guide," "Customer Reviews"). The directors and cast listings are a nice touch, and they're links to (same-site) searches for more information. In fact, Barnes and Noble replicates the core of what IMDB gives you -- cast biographies, related links -- while keeping you on-site. All in all, a job well done.

But notice what's still missing: still no mention of audio, still no mention of subtitles. Am I the only one who likes to watch movies with subtitles? How many millions of people in the United States alone only speak English as a second language? I guess those people don't shop at Barnes & Noble.

Lesson 4: "secondary" features like audio and subtitles can be a dealbreaker. Again, just be clear about what you're offering. Not everyone speaks English. Not everyone who can speak English can hear perfectly. Not everyone who can hear perfectly can watch movies at full volume without waking up their kids upstairs. (When did you think we could find two hours to watch a movie, anyway?)


Finally, here's DVD Empire's page for the same movie. They've got it all: title, disc format, running time, complete and accurate audio and subtitle information, on-disc extras, reviews, ratings, actors and directors and producers and writers and -- God bless 'em -- links wrapped around every last one of them. They even list the video aspect ratio and the UPC code. I don't think there's a single bit of readily available information that they don't list or link to. They've got it all.

And after all that, the pirate release is still better. Really. I don't mean "it's better because it doesn't have DRM," or "it's better because it's free (until you get caught)." It's better because it has unique features that you simply can not find from any legitimate distributor. Look at those subtitles: 11 of them! English, Spanish, French, Swedish, Finnish, Norwegian, Dutch, Czech, Hungarian, Romanian, and Portuguese. Holy crap, where did all those come from? The original disc only has 3!

Well, they come from people, real people who take the time to translate subtitle files into other languages and share them on communities like OpenSubtitles.org. User-generated subtitles are a massive worldwide phenomenon that most English speakers don't even know about. Ever since DVDs were cracked wide open, open-source video players have offered the capability to play retail DVDs but display subtitles from a separate file, which you could download without feeling too guilty about stealing anything. Translation quality varies widely, of course, but something is better than nothing. In this release, the pirates have gone to the trouble of locating all those subtitle files for you, and they ultimately provide a "total package" that even the best legitimate distributor can't match.

Lesson 5: in the edge cases, you can't compete with pirates because you don't control what you're selling. (Lesson 5½: there are more edge cases than you think.) Pirates aren't just distributors; they can also be content creators. They can rip, mix, burn, and mashup content at will, and they have a twisted sense of pride in offering "the best" -- whether that's release speed, video quality, file format, or subtitles in 11 different languages. (What, you thought this was the only pirate release of this movie? There are dozens more, and they're all optimized for different definitions of "best.")

Every legitimate retailer could learn lessons 1 through 4. Some retailers have learned them already. But solving this last problem would require a complete overhaul of the content distribution network. More than that, it would require a rethinking of the fundamental roles of creator and distributor, to give retailers the sort of control that pirates take without asking. I don't think it's realistic that the copyright kings of the world would ever allow such an overhaul (after all, they designed the current system from top to bottom), but this is what we're losing by not even trying.

§

In an otherwise predictable Slashdot discussion of the only companies that manage to suck more than Microsoft in providing services that people want, I found this gem:

With the recent improvements to graphics cards, computers have now got enough power for the next level of PVR to become possible. I refer of course to Personal Video Rendering, ie locally generated real-time TV. Even modest AI can handle the retarded talk shows and formulaic sycophantic interviews.

Just imagine: you can watch computer generated random pointless drivel such as 'my boyfriend left me for a transexual limbo dancer and now i am marrying his mother' with 5.1 surround whooping and hollering from the audience for as long as you like (with artificial repetitive and annoying 'advertisement' breaks, of course), then decide to watch a blu-ray hd film. The software would automatically flip to rendering 20 minutes of a sports game, followed by 30 minutes of tedious analysis by virtual sports presenters before showing the film. Artificially intelligent filtering would then cut many of the scenes and redub profane dialog no matter what time it was being watched. Monitoring daemons would flag the kind of shows that you like to watch and then 'cancel' them.

In related news, I canceled our DirecTV subscription last week, as planned. Wake me up when I can watch quality content, on my terms, for a reasonable price. Or when I can buy one of those Personal Video Renderers. They sound cool.

§