Abstract

The goal of the Content-Addressable Web (CAW) is to enable the creation of advanced content location and distribution services over HTTP. The use of content addressing allows advanced caching techniques to be employed, and sets the foundation for creating ad hoc Content Distribution Networks (CDNs). This document specifies HTTP extensions that bridge the current location-based Web with the Content-Addressable Web.

Table of Contents

1.  Introduction
1.1  Optimal Replicas
1.2  Untrusted Caches
1.3  Transient Web
2.  Scope
3.  Content Addressing
4.  HTTP Extensions
4.1  X-Content-URN
4.2  X-URI-RES
4.3  N2R
4.4  N2L and N2Ls
4.5  Proxies and Redirectors
4.6  Example Application
4.7  Replica Advertisement
4.8  Acknowldgements
§  Author's Address

1. Introduction

Content Distribution Networks (CDNs), such as Akamai, have shown that significant improvements can be made in throughput, latency, and scalability when content is distributed throughout the network and delivered from the edge. Likewise, peer-to-peer systems such as Napster and Gnutella have shown that normal desktop PCs can serve up enormous amounts of content with zero administration. And more recently, systems like Swarmcast have been introduced that combine the CDN and peer-to-peer concepts to gain the benefits of both. The goal of the Content-Addressable Web is to enable these advanced content location and distribution services with standard web servers, caches, and browsers.

There are a number of short-comings of current web architecture that the Content-Addressable Web aims to overcome. These include discovering optimal replicas, downloading from untrusted caches, and distributing content across the Transient Web.

1.1 Optimal Replicas

There are currently no mechanisms within HTTP that allows a user-agent to discover an optimal replica for a piece of content. This problem is due to the fact that HTTP caching practice assumes a hierarchical caching structure where each user has a single parent cache. Thus while one can discover an object's source URI from a cached copy, there is no mechanism to discover a list of replica locations from the source. This problem is evidenced by the fact that users must manually select the closest mirrors when downloading from Tucows, FilePlanet, or the various Linux distributions. The CAW solves this problem by providing distributed URI resolvers that user-agents can query to find an optimal replica.

1.2 Untrusted Caches

It is currently unsafe to download web objects from an untrusted cache or mirror because they can modify/corrupt the content at will. This becomes particularly problematic when trying to create public cooperative caching systems. This isn't a problem for private CDNs, like Akamai, where all of their servers are under Akamai's control and are assumed to be secure. But for a public CDN, the goal is to allow user-agents to retrieve content from completely untrusted hosts but be assured that they are receiving the content intact. The CAW solves this problem by using content addressing that includes integrity checking information.

1.3 Transient Web

The Transient Web is a relatively new phenomenon that is growing in size and importance. It is embodied by peer-to-peer systems such as Gnutella, and is characterized by unreliable hosts with rapidly changing locations and content. These characteristics make location-based addresses within the Transient Web quite brittle. Even if traditional HTTP caching was widely leveraged within the Transient Web, the situation wouldn't be helped much. This is because a single piece of content will often be available under many different URIs, which creates disjoint and inefficient caching hierarchies.

This multiplicity of URIs occurs for a couple of reasons:

• The original source for a piece of content will often cease to exist or the source's URI will change.
• Multiple independent sources often introduce the same content into the network.
• Most applications and file manipulation tools will tend to "forget" the source URI of a piece of content.

This URI multiplicity can also occur in the normal web, although it is RECOMMENDED that caching semantics be used when an authoritative source is known. The CAW solves the above problems by providing content-specific URIs that are location-independent and can be independently generated by any host. Additionally, various URI resolution services work in coordination to resolve issues associated with having multiple URIs for a web object.

2. Scope

The HTTP extensions for CAW are intended to be used for in the above scenarios where HTTP is currently lacking. This technology is focused on mostly static content that can benefit from advanced content distribution services. The extensions are intended to be hidden under the hood of web servers, caches, and browsers and should change nothing as far as end users are concerned. So even though a new URN scheme is introduced, there are very few situations where a human will ever interact with those URNs.

One of the more interesting applications of the Content-Addressable Web is the creation of ad hoc Content Distribution Networks. In such networks, receivers can crawl across the network, searching for optimal replicas, and then downloading content from multiple replicas in parallel. After a host has downloaded the content, it then advertises itself as a replica, automatically becoming a part of the CDN.

3. Content Addressing

This specification introduces a URI scheme with many interesting capabilities for solving the problems discussed earlier. A particularly useful class of URI schemes are "Self-Verifiable URIs". These are URIs with which the URI itself can be used to verify that the content has been received intact. We also want URIs that are content-specific and can be independently generated by any host with the content. Finally, to show the intent that these addresses are location-independent, a URN scheme will be used.

Implementations MUST support SHA-1 URNs at minimum.([footnote] A future version of this document will also specify a URN format for performing streaming and random-access verification using Merkle Hash Trees.)

Receivers MUST verify self-verifiable URIs if any part of the content is retrieved from a potentially untrusted source.

4. HTTP Extensions

In order to provide a bridge between the location-based Web and the Content-Addressable Web, a few HTTP extensions must be introduced. The nature of these extensions is that they need not be widely deployed in order to be useful. They are specifically designed to allow for proxying for hosts that are not CAW-aware.

• The following HTTP extensions are based off of the conventions defined in RFC 2169. It is RECOMMENDED that implementers of this specification also implement RFC 2169.
• The HTTP headers defined in this specification are all response headers. No additional request headers are specified by this document.
• It is RECOMMENDED that implementers of this specification use an HTTP/1.1 implementation compliant with RFC 2616.

4.1 X-Content-URN

4.2 X-URI-RES

• The service URI specifies the URI of the resolution service. It is not necessary for the service URI to conform to "/uri-res/ <service>?<uri>" convention specified in RFC 2169.
• The service type identifies what type of resolution is being performed and how to interpret the results from the service URI. The types are those defined in RFC 2169 and include "N2L", "N2Ls", "N2R", "N2Rs", "N2C", "N2Cs", "N2Ns", "L2Ns", "L2Ls", and "L2C".
• The target URI is the URI upon which the resolution service will be performed. The target URI can be any URI and is specifically not limited to the URI specified by the X-Content-URN header. If there is only a single X-Content-URN value, the target URI can be left off to imply that the X-Content-URN value is to be resolved.
• It is RECOMMENDED that receivers assume that the URI resolver services are potentially untrusted and should verify all content retrieved using a resolver's services.

It is believed that N2R, N2L, and N2Ls will be the most useful services for the Content-Addressable Web, so we will cover examples of those explicitly.

4.3 N2R

4.4 N2L and N2Ls

4.5 Proxies and Redirectors

It is useful to allow CAW-aware proxies that provide content-addressing information without modifying the original web server. This allows CAW-aware user-agents to take advantage of the headers, while simply redirecting user-agents that don't understand the Content-Addressable Web. It would be inappropriate to return an X-Content-URN header during a redirect, because HTTP 3xx responses often still include a message-body that explains that a redirect is taking place. Instead it is preferred to return a result of the text/uri-list media type that includes one or more URNs that would normally reside in the X-Content-URN header.

4.6 Example Application

The above HTTP extensions are deceptively simple and it may not be readily apparent how powerful they are. We will discuss an example application that will take advantage of a few of the features provided by the extensions.

With this response, a CAW aware browser can immediately begin downloading the content from www.linuxiso.org, linuxmirrors.com, and 123.24.24.21 all in parallel. At the same time the browser can be dereferencing the N2Ls service at 123.24.24.21 to discover more mirrors for the content.

The existence of the 123...21 host is meant to represent a member of an ad hoc CDN, perhaps the personal computer of a linux advocate that just downloaded the ISO and wants to share their bandwidth with others. By dereferencing the N2Ls, even more ad hoc nodes could be discovered.

4.7 Replica Advertisement

The URI-RES framework provides a significant amount of flexibility in how replica advertisement and discovery can be implemented. One example implementation will be provided in a future specification.

4.8 Acknowldgements

Gordon Mohr (gojomo@bitzi.com), Tony Kimball (alk@pobox.com), Mark Baker (distobj@acm.org)

Author's Address