Inside the struggle to preserve the world's data

Might be a better intro for 668 than @edsu's, which scares people off.

ArchiveReady is checking whether websites are ready to be archived.

Presentation for Stanford Drupal Camp on how and why to build archivable websites.

Excellent overall overview

Format Description for WARC -- Web ARChive file format.  ISO 28500:2009.  Used by archival institutions to store content harvested by web crawls, for example via use of the Heritrix harvesting tool.

... Read More

Some useful pushback on "significant properties" and all the other accounting.

D-Lib Magazine

The SCAPE project is developing solutions to enable the processing of very large data sets with a focus on long-term preservation. One of the application...

CiteSeerX - Document Details (Isaac Councill, Lee Giles, Pradeep Teregowda): Abstract. Dealing with documents that remain unchanged between harvesting rounds is an important topic for many organizations archiving the World Wide Web. This paper discusses some of the key problems in dealing with this and then outlines a simple, yet effective way of managing at least a part of it. This is done in form of an add-on module for the popular web crawler Heritrix. The paper contains the results of crawls using this new software. Finally, there is a discussion on the limitations and some of the future work needed to improve duplicate handling. 1

One of the logistical problems facing long-running publications is the archiving of old material. I say "problem," even though...

I recently spoke with a resource-limited organization that is investigating government corruption and wants to access various public datasets to monitor politicians and law firms. They don’t have developers in-house, but feel pretty comfortable analyzing datasets in CSV form. While many public datasources are available in structured form, some sources are hidden in what us data folks call the deep web. Amazon is a nice example of a deep website, where you have to enter text into a search box, click on a few buttons to narrow down your results, and finally access relatively structured data (prices, model numbers, etc.) embedded in HTML. Amazon has a structured database of their products somewhere, but all you get to see is a bunch of webpages trapped behind some forms.

Jacob Harris on the hows and whys of designing interactives to survive the future

I had three minutes to give a talk to a diverse audience, so this is my crack at it below. The audience could then follow up with us at our poster, which I’ll also put online. It was an incre…