September 2004 Archives

Cory Doctorow points out this report from CacheLogic (which makes appliances for monitoring network traffic) entitled "The True Picture of Peer-to-Peer Filesharing".

The bottom line?

- P2P is the single largest consumer of data on ISP networks.
- P2P traffic significantly outweighs web traffic.
- P2P traffic is continuing to grow.

Also notable is the finding that BitTorrent traffic has doubled in the last six months - from 26% of P2P traffic to 53%, while Kazaa use has decreased from 46% to 19%.

Thanks, Cory!

I spent last Thursday and Friday in Montreal for a Calendaring Roundtable sponsored by the new Calendaring and Scheduling Consortiumand hosted by Oracle.

The purpose of the roundtable was to get some of the major players in the calendaring software world
around the same table to see if we could make some concrete forward
progress towards real interoperability.

Bob Morgan, Michael Gettes (Duke), and Jeff McCullough (UC Berkeley), and I attended from
CSG-member higher ed institutions. Many of the commercial companies
with calendaring products were represented - The big-guns (IBM, Oracle,
Novell, Yahoo), the open-source community (Mozilla Foundation,
OSAF), and the smaller companies (Stata Labs, Cyrus), as well as one
other major player who did not want to be named in public communiques.
Noticeable by their absence at this gathering, though they
were invited, were Microsoft, Sun, and MeetingMaker. A person from the
Outlook group at Microsoft had intended to participate, but was told
two days before the meeting that he did not have corporate approval to
participate. MeetingMaker responded that they were not interested in attending at this time.

The meeting went extremely well - there was a great deal of enthusiasm
towards achieving short-term progress in interoperability, and people
generally seemed to be ready and willing to get on with working
together on the nuts and bolts. While it's hard to tell whether this
will be sustained with real ongoing effort, the folks at the table
appeared to be willing to commit real resources towards this end.

There was general agreement around this table that the CAP protocol has
proven to be unworkable and may now, for all practical intents and
purposes among this group, be considered at least moribund, if not
entirely dead.

Several of the companies represented (Mozilla, Stata, Novell) have
reverse-engineered the method Apple uses for publishing calendars from
its iCal product to the web. There was an agreement to work on
documenting that method so that others can more easily implement it.

There was a very general enthusiasm for the new CalDAV proposal , and
several of the companies represented are starting to code to the
current document, while acknowledging that their implementations may
have to change as the draft evolves. It seems possible that we may see
some early test implementations as early as next spring. Future
CalConnect events will include interop bake-offs to demonstrate the
state of success (or issues arising) of this approach.

There was agreement that the activities of the Consortium would be
complementary to IETF work in this area. There will likely be an IETF
Working Group on revising the base calendar formats, and there may be
one on CalDAV.

This event will likely be followed by an interoperability event in January.

I think we all came away feeling like a log-jam in the calendaring
standards world might at long last be breaking apart, and we came away
quite hopeful.

As I reported separately, OSAF has slipped their development schedule for Chandler and have made significant changes in their architecture and plans.

During the Westwood Advisory Council meeting today, Mitch Kapor presented a list of lessons learned that I think generalizes for all sorts of development efforts, so I share it here:

Things learned

• Underestimated cost of ambition


• Hard decisions about product strategy and focus could have been made earlier


• Proved harder to build engineering organization


• cross-platform and rich clients are hard


• build and integration work is non-trivial

The Chandler Westwood Advisory Council met today with Mitch Kapor, Chao Lam, and Pieter Hartsook from OSAF. The Chandler folks have, unsurprisingly, found that their development timelines are longer than originally predicted. The plans now call for a 0.4 release in late October that will support a basic user interface for email, calendar, and sharing items via WebDav.

After 0.4 there will be a 0.5 release in the first quarter of 2005 that will be designed to be usable for basic individual and collaborative tasks in a small workgroup - OSAF intends to adopt that release for their own uses, hence the nickname "dogfood" (as in "eat our own...") for this release.

The "Kibble" release, coming in the fourth quarter of 2005, will have enough functionality for use by groups of early adopters.

We had a lot of discussion about whether it would make sense for OSAF to elevate the priority of adding calendaring client functionality (using CalDAV) earlier rather than later (that of course begged the question of who was going to build a CalDAV server). The prevailing sentiment in the group was that sounded like a good idea, but we wanted some more detail on the level of effort required and which other features were likely to be deferred to make that happen.

Chao's slides for the meeting today are available from the agenda page on OSAF's Wiki.

Tracy Futhey, the CIO of Duke University, is talking about their experiment where they are giving iPods to all incoming freshmen this year.

One thing I hadn't realized is that they are handing out a recording device with each iPod, so it can be used to gather audio information as well as just playing the content. This plays out in courses like an introductory engineering course, where the students have to gather audio as data for signal processing experiments.

One thing the Duke folks are already noticing is the iPods have elicited interest in use from faculty who have not previously been involved with using technology in their courses.

I had a "d'oh!" dope-slap sort of moment (is that an epiphany?) listening to Tracy - while much of our talk in higher ed about
multimedia and pervasive computing devices has been about text devices (Palm, PocketPC, Blackberry) and video, the actual technology that is currently pervasive is the mp3 player. We could realize some tremendously effective educational enhancements by concentrating on providing audio content and making that widely available. Note to self: see what campuses are doing with audio content and whether there are any large scale sites for university audio content.

On the plane flight over I was reworking one of my web sites, basically rebuilding from the ground up with content, look & feel, and navigation. I was using the latest version of BBEdit, version 8. I have to say that the more I work with BBEdit, the more I find it to be my favorite editor, on any platform. It does all the things I want to do with text and stays out of my way.

Now keep in mind that my basic text authoring approach is that most of the time I can use plain old ascii text, because it's going to be in email. When I am writing something that needs formatting, I'm going to do it as a web page, not as a Word attachment. Why not Word, you ask? Well, for one thing, lots of people don't have Word - don't forget that Microsoft Office costs hundreds of dollars. But pretty much everybody has a web browser. For another thing, Word just gets in my way too much - it thinks it's smarter than I am, which I can't stand.

But BBEdit just puts it all in plain visible text on the screen, and it highlights markup and syntax from common languages, and it's fast and just plain works.

John Gruber has his finger on what makes it a great application, and why the new version is a worthy update, in a recent post on Daring Fireball.

The appeal of BBEdit is in its balance of powerful text-editing features and an elegant, intuitive, and unabashedly Macintosh-style interface — and where by “interface” I don’t mean in the sense of superficial cosmetic appeal, but in the deeper, interactive sense.

Molly Baird - Brown U - Exchange 2003

Migrating from an imap/pop service to Exchange. About 95% migrated.
Hosting about 12k users

Two 2-node clusterd Exchange mailbox servers
Four load-balanced Exchange front-end servers (OWA, IMAP, POP)
250 Gb total storage

Brown has a central Active Directory
- people container is totally separate from the departmental container that has machines and groups/lists.

Two kinds of databases in an exchange information store
mapi store (available to mapi clients)
direct access by imap, pop, OWA
on the fly conversion of content when accessed by the other kind of client

Exchange public folders

Not in heavy use at Brown, but some use - e.g. for storing vacation calendars, shared contacts, shared lists, collections of project files.

Converting shared IDs into shared mailboxes with ACLs.

sizing -

DB size based on wanting to recover single database based on backup needs.
Many small databases on each server (about a dozen)

Exchange costs:
1 FTE for migration and maaintenance
- One time costs:
-- migration 1.5 fte split between admin and tech duties
- license - Microsoft CAL license requred
- license: OS and Exchange sw - $11k
HW - four front end servers - $30k
HW: staff cluster - $60k
HW- Student cluster $60k
SAN storage - $70k
Provisioning changes - coincided with provisioning avehaul (automatic password creation and sync, print services, etc).

Total one-time cost: $231k

(by my math that's about $50/user/year just for hardware, based on a four-year life cycle)

Quotas - fac/staff 100Mb quotas, students 25 Mb

Students don't get MAPI access - pop, imap, or web client only

Have to recover an entire database to recover a single mailbox or folder.

An interesting article in Fast Company about how communities of commited, networked amateurs are changing professions, including computer games, pop music, and information technology.

Pro-Ams could fuel mass participation in formal politics and in social entrepreneurship. They will play important economic roles as coproducers of services and sources of ideas. Democracy will be livelier, innovation more vibrant, social capital stronger, and individual well-being more securely grounded. After a century in decline, amateurs will rise again. And they will change the world.

Thanks to Dan Krimm for pointing this out on the Pho list.

Wired News reports here that

Over 40 technology companies and consumer rights advocates sent a letter to Sens. Orrin Hatch and Patrick Leahy on Friday urging them to hold public hearings on the Induce Act, in hopes that Congress won't act hastily in passing a law that would have huge effects on the tech industries.

I hope they're successful - the Induce Act is a terrible piece of proposed legislation.

Thanks to virtualaw for pointing this out on the Pho list.

In this room of senior technology people from 25 or so major US research universities, I count 21 Mac Powerbooks and 19 intel-based machines. The question is, are these folks leading indicators, or just outliers?

We're at the Sheraton Commander in Cambridge, across the street from where a tree used to stand that supposedly was the place where George Washington accepted command of the Continental Army in the 1770s. That tree was mistakenly cut down by a Cambridge street worker in the 1920s. The subsequent fate of the city worker was not mentioned :)

Vace notes that 1/3 of his staff are involved with supporting email

From the informal survey, institutions identify over 30% of email is spam or virus email. That sounds low to me, at least from our UW experience.

Susan DeLellis - Harvard - Evaluating Email/Calendar Soutions

Harvard email environment - about 50k accounts, 60% students. less than 5k calendar accounts. Currently Central Admin provides for-fee service, individual schools (8) provide accounts, plus others, using lots of different solutions. No central authentication for email or calendaring (but there is a harvard ID/pin system used for applications and wireless authentication).

Central IT provides a central email aliasing service (first_lastname@harvard.edu). There is a central voice-mail to email service.

There is a University-wide email broadcast applications (presmailer) developed for emergency and cticial university correspondences.

They sent out a suvery to users - 13% of 6300 email customers responded.

97% rated email as most important communications services (over phone and mobile phone service!)
Provide better integration between calendar and email and IM (single client)
Provide more robust web client and remote access
Improve reliabilty of claendar system
develop central group calendar system

Focus groups with staff and managers
- reliability of messaging is critical
- effectiveness of antispam and antivirus solutions directly impacts day-day productivity
- remote access growing in importance
- sensitive to change and cost

Did an RFP and got back around a dozen reponses
- emergence of collaboration suites, including file sharing, portals, web conferencing, unified messaging, IM in addition to email and calendar.
- how does that integrate with existing solutions and tools - what's the right infrastructure?
- clear trend towards database architectures as the message store.

Initial TCO show total proices in the $8-12/user/mo range. Fully loaded cost includes hw, sw, maintenance, facilities, monitoring, backups, storage, labor, disaster recovery, implementation, training and help desk services.

Brian Baird, a Democratic congressman from southwestern Washington state, gave a remarkable statement on how the current US administration is tampering with federal scientific efforts. I don't see this anywhere on the Web yet, so here it is in its entirety (thanks to Barbara Perry for passing this along):

[update] - I found this statement on the web - it's at http://thomas.loc.gov/cgi-bin/query/D?r108:47:./temp/~r108qNo3x2::

Now there's a friendly url.

THE ESSENCE OF SCIENCE

these discussions, or distortions, are manifold; and collectively, they
constitute nothing less than a coordinated attack on virtually every
stage and every aspect of the science/policy interaction.
Evidence of this attack comes from many sources, including a GAO
study which I am holding up here, which I requested along with my
ranking member on the Committee on Science, the gentlewoman from Texas
(Ms. Eddie Bernice Johnson). Interestingly and perhaps tellingly, we
had asked that a full committee hearing be conducted to study this
matter; but we were denied that privilege, leaving us to hold a
somewhat symbolic hearing of our own.
Nevertheless, based on testimony from that hearing and numerous other
sources, it is apparent to me and others that the assaults on
scientific independence and integrity includes all of the following:
limitations of the questions that are allowed to be asked; constraints
on the methods that are used to seek answers to questions; limits or
elimination of funding and resources to pursue certain questions that
are not politically correct; biased selections of people who will be
allowed to ask questions or serve on scientific panels; active and
intentional suppression of findings that are not to official liking;
unjustified claims and inflation of studies or results that are
approved of by the administration; punishment or ridicule of scientists
who disagree with official administration dogma; retribution for
political involvement on the part of scientists; disregard of
discomfiting scientific evidence; placement of nongovernmental
ideologues in charge of international missions to supervise U.S.
positions, vis-a-vis, scientific discussion; and creation of a climate
in which scientists and policy-makers have begun actually to self-
censor or self-select and actually leave government service.
Thomas Jefferson wrote in a letter to his nephew: ``Question with
boldness even the existence of a God because, if there be one, He must
more approve the homage of reason, than that of blindfolded fear.''
Clearly, at least in his private letters, Jefferson was not one to
believe in limiting questions, and indeed, if one visits Monticello and
sees his love for science, one realizes how important that was to him.
When one considers that Benjamin Franklin was considered one of the
greatest scientists of his age and that Madison, Jefferson, and
Washington and many of the Founders had a profound interest in science,
we realize the importance of that principle to the founding principles
of this Nation.
But we must contrast that attitude of the Framers with an
administration that removes from a National Cancer Institute Web site
fact sheets showing there is no empirical evidence linking abortion to
breast cancer. Contrast that attitude of scientific inquiry with
suppressing analyses of clean air legislation that will save lives and
cut pollution at negligible cost. Contrast the Framers' attitude with
initiatives in Congress to cut funding for research relating to
sexually transmitted disease prevention. Contrast that attitude with
limits to stem cell research. Contrast that attitude of the Framers
with the selective appointment or withdrawal of experts on scientific
advisory panels. Contrast that attitude with the willful stacking of
advisory committees and removal of any voices deemed unfriendly to a
predetermined outcome.
Within the scientific community, the effect of the administration's
and congressional actions have been chilling and demoralizing.
Researchers are practicing self-censorship or leaving government
careers entirely.
Let me conclude, if I may, with one final comment of Richard Feynman.
He said, ``It is our responsibility as scientists, knowing the great
process which comes from a satisfactory philosophy of ignorance,
knowing of the great progress which is the fruit of freedom of thought,
to proclaim the value of this freedom; to teach how doubt is not to be
feared but welcomed and discussed; and to demand this freedom as our
duty to all coming generations.''
We must do that not only as scientists but as Representatives.

I'm in Cambridge, Mass, for the fall meeting of the Common Solutions Group. wjere the workshop topics include: Messaging and collabaration services; the impact of "pervasive computing" on teaching and learning.

We'll also be having a meeting of the Westwood Advisory Council, talking about developments to date on OSAF's Chandler project.

There will be a policy discussion on Security Incident Response, Remediation, and Policy which should be timely.

On Thursday I'll be off to Montreal for a couple of days at Oracle's calendar unit, discussing calendaring standards with some interested folks.

And then I'll be in New York for the weekend, celebrating my mother's 80th birthday.

I'll be blogging the events as they happen (except for the birthday).

For those who are interested, I believe the CSG workshops will be webcast live, starting at 9:00 am Eastern US time on Tuesday, Sept 21. Check the CSG web site under Next Meeting for the URL.

I spent some time recently catching up on reading various blogs. One interesting thing I've noticed as my time's gotten busier over the summer and into the fall is that my blog reading habits have changed - while there are lots of blogs I think have regularly interesting information and perspectives in them, I tend to have a handful that I make a point of reading faithfully, and they tend to be blogs of highly intelligent writers who don't post all that much - as opposed to those of highly intelligent writers who post a whole heck of a lot. I put Tim Bray, Liz Lawley, Jon Udell, and Clay Shirkey in the first category, while relegating folks like Joi Ito and Marc Cantor to the second. It's not that I don't ever read those folks - but I tend to skim through their postings looking for relevance, instead of making a point of reading each post.

At any rate, I was following a link from Marc's Voice to Shelley Powers' Burningbird blog, which I hadn't previously read, and I clicked on a link on the left that just said "Shelley Powers", thinking it would give my some background on who Shelley is and what she's up to. But instead, of course, it was a mailto: link, and just launched my mail client.

I wish people would clearly mark those kinds of links, like "Contact Shelley" or something.

Seen in a parking lot in University Village on Friday - A Toyota hybrid with three bumper stickers: Howard Dean, Patty Murray, and.... Dino Rossi! ? !

I wanted to stop and ask the driver what manner of beast she or he is.

Yesterday saw the end of the IETF Calendaring and Scheduling Working Group.

What this means, in effect, is an end to work on the Calendaring Access Protocol (CAP). CAP was supposed to be the standard for client-server interaction on shared calendar servers. My (admittedly non-developer) perspective is that over the years of discussion, CAP grew to be so large and inclusive that nobody was going to bother to implement it in any interoperable way.

It will be interesting to see if the problems with CAP were due to the inherently complex nature of calendaring and scheduling, or whether there are some lighter-weight approaches that will solve most, if not all, of the problems of interoperable Internet calendaring.

Activity on calendaring standards has now morphed into two separate discussions:

- A proposal for a new client-server protocol being dubbed CalDAV, based on the WebDAV protocol. Lisa Dusseault from OSAF has an initial draft for this protocol, and there is an IETF mailing list for it here. CalDAV standardizes the type of approach taken by Apple's iCal and Mozilla Calendar (now dubbed Sunbird), using a WebDAV server to store calendar events. OSAF has (unsurprisingly) stated that they are planning to support CalDAV in Chandler.

- A new effort to revise the base iCalendar data format (rfc 2445), based on real-world experiences gained from actual products trying to achieve some level of interoperability. There is an active IETF mailing list for this topic - lately the discussion there has centered around whether or not alarms on events should be supported in the data exchange and to what extent access control should be supported in the format (which is necessary to support features like preserving confidentiality in specific calendar items when exchanging data).

Keep watching this channel for further developments :)

Update - 09 September 2004

Doug Royer, the author of the CAP protocol writes:

No - It is not the end of CAP. It is the first revision of CAP.

CAP will be released as EXPERIMENTAL until the next revision. This has been under consideration for months and all of the authors and primary IETF chairs and area directors are in agreement.

In a word - his Purpleness was astounding!

An introductory video clip featured shots of Prince through the years, concluding with Alicia Keys quote during his induction into the Rock And Roll Hall of Fame" "There are many kings, but there is only one Prince ... only one man who has defied restriction, defied the obvious and all the rules of the game." Just a little bit of ego at work there :)

As the band started pumping out the groove to Musicology, Prince rose up from beneath the center of the x-shaped stage in a halo of spotlit fog and the crowd was on its feet, to remain so for most of the concert.

He plowed through two hours of non-stop hits (he says this is the last tour he's going to perform them) - the opening number was the only recent song he played - unfortunately none of the tunes from the Rainbow Children got played at this show. But he did an incredible funky, long version of D.M.S.R, he played Controversy, and Raspberry Beret, and Little Red Corvette (during his solo-acoustic portion, which also included a nice slow blues of a man pleading with his woman to not make him sleep on the couch).

The singing and guitar playing were phenomenal, the band was deep into the groove, and the showmanship was untiring. I was struck by how much Prince has absorbed from the whole of popular music - the obvious references to Hendrix and Little Richard, but also moves showing the influence of Mick Jagger, and flashes of Santana in his playing, and at one point during his all-too-brief bass-playing segment, a note perfect quote from one of those old Stanley Clarke tunes (maybe Lopsy Lu?). He even covered Sam and Dave's Soul Man during the show!

The horn section featured both long-time James Brown sidekick Maceo Parker and Candy Dulfer. Maceo even sang a tune, doing a spot-on Ray Charles tribute on Georgia on my Mind.

Unfortunately we didn't get to see the totally fabulous Rhonda Smith play her upright bass this show, but man, did she lay down the funky lines - and how does she manage to play all night wearing those stiletto heels?