Two of the tutors were part of the group that have drafted the TDR document and another of the tutors was involved in the peer review process. They gave some background to the document which grew out of a recommendation from a 1996 Task Force on Archiving of Digital Information. They explained the basic difference between auditing a repository and certifying a a repository, essentially saying that an audit is an internal or external process evaluating you against ideals. An audit is about continual improvement and does not necessarily have a pass/fail process. Certification, on the other hand, does result in a pass or fail, usually of an audit-like process. Certification may have elements of both processes such as ISO 9000 mandatory elements.

The authors of the TDR document were tasked to created a standard certification process or framework that can be implemented across domains or types of digital repositories.

The tutor who has been involved in the peer review process criticised the draft document on the following grounds:

• It sets the bar too high for many archives
• The drafting of the document was not led by archivists
• The authors claim that they are an international body, but in fact they are mostly N. American with some Europeans but no-one east of the Netherlands
• The OAIS standard is flexible but the TDR certification which is based on OAIS is very specific about wht the archive should achieve

However, before we throw the document out on the above criticisms, it is worth remembering that it’s only a draft and that it still offers a very useful tool for internal audit and self-improvement. As one of the tutors listed all the reasons why we should care about certification, I realised that none of them specifically apply to AI but are really aimed at public institutions who have obligations to to funding bodies, regulatory bodies, service purchasers, and external depositors. Of course, certification would still give a number of IRP staff a warm glow inside, smug with the satisfaction of international recognition of our work and the service we provide to our organisation, but I think the real value in the TDR document is that it provides practical guidance on how we can improve. Also, having audited our digital preservation archives using the TDR document, we may come to the conclusion that we’re just not up to or interested in developing an archive for long-term preservation and that we might want to outsource some or all of the work to another institution. Preferable one that is certified!

Outsourcing was touched upon at the very end of the hour and we were offered some step-by-step guidelines on approaching the possibility of outsourcing. Here they are, straight from the PPT slides:

Rationale
We can’t all be experts in everything
We need to carry out some tasks we are not well-equipped to do
We may have resource reasons (money but no space)
We may have policy reasons
It may be cheaper

Understand your needs
Define your problem before you look for a solution
Otherwise you will buy the answer to someone else’s question
Specify mechanisms to monitor and measure performance
Look at the DPC document on outsourcing

Outsourcing
Outsourcing digital preservation requirements may be an attractive option – particularly for smaller organisations
But these are also the most vulnerable in terms of what they should expect to ask for and receive
Having a system of certified repositories can help to provide assurance
The checklist of requirements can help organisations find a good match between what they think they asked for and what they receive.

That’s it really. I should add that the University of London Computing Centre which is a ten minute walk from AI will almost certainly be certified as a TDR because one of the authors of the report, runs the preservation programme there. And, yes, they take on consultancy work and are willing to discuss any outsourcing we might decide we want to do with them.

That’s the end of this blog. I hope you’ve enjoyed reading it and found at least some parts of it thought provoking. As I said early in the week, I wanted to do it because Chris sent me on the course on the condition that I gave a presentation to interested staff when I returned, which I’m happy to do, although a presentation is probably not the way to go about it. Hopefully this blog has provided some background reading (and light entertainment) for us to discuss in the near future.

See you on Monday.

The task then, was to examine whether the IA scored well with regards to the ‘three legs’ and ‘five stages’ I mentioned at the beginning of the week. The assumption is that a ‘trusted digital repository’ (basically, a ‘good archive’), should satisfy our requirements in the areas of resources, technology and organisational framework. We were asked to score the IA against the five stages: Acknowledge, Act, Consolidate, Institutionalise and Externalise. These basically represent five stages of maturity in a digital archive’s development.

If you haven’t spent much time on the IA’s website, it’s certainly worth doing so. It was originally the initiative of one man, Brewster Kahle, who made millions in software development in the 1990s. His foundation continues to fund around 50% of the IA’s expenses (we were given copies of the IA’s 990-PF Tax Returns for 2003 and excerpts from 1999, 2000, 2001 and 2002). In 2003, the IA had expenses of just over $3m, so Brewster’s millions are essential for the sustainability of the IA.

This was a concern for all of us since we were asked to consider how sustainable the IA appears to be. Now, admittedly, our figures are a couple of years out of date and we’ve no idea how Brewster’s foundation is set up and how sustainable that in itself is, but the donations from other companies over the years have been sporadic, project based and unpredicatable.

It’s an expensive operation to run when your mission is to collect every website in the world which is why in 2003, they spent $380,558 on disk and tape stock, $79,358 on misc. hardware and $27,002 on their phone/network charges.

The IA’s technological framework scored quite low with our class, too. It’s web crawling is done by Alexa every two months, although if you look at the history of AI on the IA’s Wayback Machine, you’ll see that the results are more adhoc than that. It’s worth pointing out that the IA provides researchers with another way of accessing the archived websites, although currently this is from a command line and they are developing tools to help make this easier.

We looked at how they approach long-term preservation, in particular the way the data is documented, stored, migrated, backed up and secured. The IA stores the websites in ARC and DAT containers which are open formats and open source tools are available to work with these formats. However, the original file formats harvested from the websites are not migrated to different formats for long-term preservation so proprietary file formats harvested, remain proprietary and therefore possibly inaccessible long-term. The IA do also collect software and emulators in the hope that some kind of access will be possible in the future, even for obsolete formats. It’s all a bit hit and miss, not least because their website has no formal policy or strategy for preservation. It’s just a few paragraphs here and there which you have to draw your own conclusions from. The preservation metadata which they add to the harvested pages is minimal since it’s done by machine. You can see what they add by using the Wayback Machine and then looking at the source of one of the harvested web pages.

The lack of formal, detailed policy was also a concern to the class. The IA have taken on such a huge task, but with the exception of some areas of their technological development, there’s very little to suggest that they have a strategy in place that will ensure the long-term preservation of their archive which is central to their mission. The other major organisational concern was that technically their activities are illegal or at least waiting to be challenged. Essentially, they are taking copyrighted material from owners without asking for their permission. They do say on their website that they respect robot.txt commands so that you can tell their crawler not to archive your website and they also say that if you contact them and ask them to remove your website from their archive they will do so, but this places all of the responsibility on the rights holder rather than the IA, which is contrary to the usual methods of agreeing terms and conditions before copying/taking/using owned materials.

While on the subject of their web crawler, as I said earlier in the week, it has quite obvious problems in that is only works reliably with flat html files. Javascript, database driven websites and in fact, anything interactive, including video, audio and animation, presents real problems to the IA. So even if you were happy about the IA copying your material, the likelihood is that they will only be able to do part of the job. This is increasingly apparent if you use their Wayback Machine as sites from a few years ago are much better archived than more modern sites which are increasingly using more interactive web technologies.

They are trying to address some of these issues with a new subscription based service which allows you to pay for tools to customise the way your website is archived, ensuring that all the right links are harvested on a schedule that suits your needs. You can also add Dublin Core metadata. This has the potential to make the IA’s technology directly available to you at a price. You can also request copies of their archive of your website (for a cost), and you can set up a private service so that your website archive is not available to anyone else, though this is clearly not what the IA would prefer you do. Unfortunately, this new service still suffers from the technological limitations I mention above and you are still advised that “as a general rule of thumb, simple html is the easiest to archive.”

So, as you can see, from our outsider view, we had major concerns about several key aspects (sustainabilty, technology, policy and strategy) of the IA. However, all the above doesn’t directly address the class project’s central concern which was to examine how the IA might preserve a University’s digital collections that are being made available on the web.

What the IA do is archive web content, not the institutional digital collections. Effectively, it’s like saying that they archive the ADAM web pages but not the actual ADAM digital archive which contains high resolution TIFF master images and a fuller set of metadata than is shown on the website. The IA can archive a site so that you can see what that site looked (and to a lesser degree functioned) like at a certain period in time but they make no claims to be preserving the digital collections which the website is drawing from when it presents a user with the results of their search for images on women in Sudan.

That was key to the class project and, in a way, to the week’s course over all. On the first day, we discussed how the OAIS model works, in that digital materials are ingested/acquired (the submission information package), then archived (the archive information package), and then made available in a different form (the dissemination package) according to the user or ‘designated community’. At each stage, the information collected/presented is different. The IA are only interested in the dissemination package, whereas as an institution, we’re interested in all three information packages, and as a preservation archive with resources invested in the creation and intellectual property rights of the digital objects, we’re especially interested in the archival information package.

We preserve because we expect access.
We must be able to derive one from the other.
We don’t have to let one dictate the other.
Access needs may drive decisions at ingest e.g. on metadata.
There are many ways to provide access.

We were advised to ‘preserve enough to tell the story’ and that good preservation refers to ‘preserving meaningful information through time’. Quotes like that can come in handy sometimes.

Of course, depending on the archive’s remit, there are various reasons why we might have to provide access (FOI) or restrict access (DPA). Fortunately only the latter applies to us right now. There was also a brief discussion about redacting certain information before providing access, something that we do in a way with MAV’s products and transcripts for security reasons before they are put on the public database. I don’t knnow if we do it to documents, too. Do we?

Finally, the OAIS standard clearly has a strong element dedicated to ensuring access. It’s based around knowing your ‘designated community’ which may be as broad as ‘people who can read english and use the internet’ (The National Archives) or as narrow as ‘my friends and family’ (imagine an online photo service like Flickr). As usual some of the best advice was about planning ahead, being proactive abouut offering ways to access material, seperating the preservation infrastructure from the access infrastructure and collaborating with other repositories.

Basically, the ADS received a collection of data, including images, from the excavation of Christ Church, Spitalfields. The images in this very interestining collection show the remains of bodies buried in the 18th century crypt. These images were found by a web site for necrophilia enthusiasts and some images were copied from the ADS site and republished on the sex-with-dead fan site. They also provided a link through to the ADS for enthusiasts to grab more images for themselves.

This was their mistake, because the ADS noticed an unexpected spike in the use of its website and traced it back to the link from the other website. It was the first time they’d had to deal with the mis-use of their digital collections and sought advice from the JISC legal team. They were advised a six-point plan spread over 70 days. The first was simply to contact the website, tell them they had broken the licence they agreed to on the ADS website and that they take the images down or else face further legal action. And they did take them down. End of story.

This was a satisfactory result for the ADS because despite the mis-use of the images, it would have been a long and difficult legal process had the web site not taken them down.

We’ve been thinking about such things for ADAM and intend to introduce a ‘handshake’ agreement prior to the download of ADAM images. Having seen how the ADS handle this ‘contract’ with its users, I’m now inclined to just have users agree to a licence when they first enter an ADAM session rather than each time they click to download. Legally it would appear to cover us. Our present system is based on authentication into the AI Intranet and then trusting that the AI staff member will respect the terms and conditions that are displayed with each image, but we think we can do better than this with little inconvenience to users. There will also be more changes to the way ADAM handles rights management and licence agreements.

The main piece of advice that the ADS gave from this example was that archives should not wait for the abuse of their content before forming a response but rather formulate a strategy for dealing with a potential incident so we can react quickly, methodically and legally. Wayne, Claire and Tim will know more about whether we’ve had to deal with this already. I’m not aware of such a strategy being in place though. In late May, an IPR expert from the Open University will be giving a one-day workshop on IPR issues for AI staff, something we intend to run each year. Having spent just an hour touching on such issues, I feel a day’s course would be well spent ensuring IS staff are informed of the risks and responsibilities involved in this area of our work. Not least because the European Copyright Directive, which applies to the UK, now makes breaking copyright protection a criminal offense rather than a civil offense, so theoretically someone could go to jail whereas it used to be that the individual/organisation would be fined based on the ‘loss’ (financial, of reputation, of relationships, etc) to the rights owner.

We did an interesting exercise comparing the costs of running an e-prints archive at Cornell University and The National Archive’s digital archive. Not surprisingly, the two archive’s costs are radically different because their remit and services provided are radically different. It costs TNA £18.76 to ingest/acquire a single file into their archive. A huge sum compared to Cornell’s £0.56-£2.84. This is not only because TNA’s remit is so much wider and therefore the ingest/acquisition process is much more complex, but because TNA operate in an environment where they catalogue the material themselves whereas Cornell have no catalogers but require the Professor submitting her document to provide and verify all the information/metadata. Also, TNA have huge preservation costs because they are dealing with a legacy digital material which are 20-30 years old, when no preparation was made for long-term preservation of these materials. Cornell on the other hand, are archiving simple, modern digital materials and their preservation activities are relatively easy and predicatable.

This raised a familiar and interesting question for me because we will be developing a facillity in ADAM for staff to upload images to a team catalogue and provide metadata for the image. In an ideal world, the member of staff would provide full and accurate metadata which would require no validation and could be entered directly into ADAM and immediately available on the Intranet. Of course, this is almost certainly impossible for AI. It works for Cornell because the Professor has a vested and very personal interest to ensure that her article is made widely available and correctly cited through the submission of complete and accurate metadata. Even then, an example was given where an academic catalogued their article with a single keyword representing their sole academic interest, disregarding the other subject areas which the article related to. I asked people if they had any advice on how we could have AI staff more involved in the cataloguing process but no miracle answers were forthcoming. Basically, while staff are essential providers of information about the digital object, supplying information only they might know, it’s an unacceptable organisational risk to then make those images directly available for other staff to reuse before AVR have checked and verified the metadata and, as is always the case, enriched it with further information. And of course, staff might justifiably argue that they could be making better use of their time than extensively cataloguing images and checking copyright and license agreements. There will be ways that we can ensure that the information provided to us is formed in a way that is easy to validate and enrich though and that’s the approach we’ll be taking with ADAM.

At one point while trying to breakdown the cost elements of a digital archive I realised that we were a room full of archivists trying to do the job that IT professionals have been doing for years. The element costs involved in digital archiving such as hardware, software, licenses, support, development, fixtures and fittings, etc. are costs that we share with ITP. Where IRP need to demonstrate costs is by detailing the work processes and therefore the staff time involved and the business reasons why archival preservation might require three or four times the storage requirements, a different approach to risk management, changes in data management, etc. But, with the exception of staff time, a digital archive uses readily available IT solutions in a specific way. I tried to make this point that we (archivists) are not the people best placed to cost IT systems but rather need to work with IT professionals and draw on their existing experience in planning, purchasing and maintaining systems. I think that to an IT department, a digital archive is just another application of IT hardware, software and processes. Do you agree?

This wasn’t the first time I’ve found that archivists tend to look at a digital archive infrastructure as something new and perculiar to them and completely alien to IT professionals. Sure, there might be different requirements that some IT staff might not be familiar with but it’s the archivist’s role to explain and justify these in business terms and in return, let the IT staff deliver the infrastructure requirements to meet the business case. It’s just data that needs to be treated a bit differently, that’s all.

Despite this frustration, this class had real practical value for me and was a morning well spent.

It’s a service that the ULCC perform for TNA by preserving UK government databases and records no longer in use. Sounds dull but of course these databases include the National Inventory of Woodland and Trees, a survey of British Bats, statistics on how many accidents there are in the home, crime statistics and the names and assets of victims of Nazi persecution who were compensated by the UK government. These databases are all transferred to NDAD who migrate them into sustainable formats, document them with good metadata and make them available to search online. It sounds like a great place to work if you’re interested in the history of computing as some of these databases were the largest of their kind at the time and represent significant historical moments in the history of computing. They also have to deal with all the legacy software and hardware issues, data analysis, system design and digital conversion as well as the development of emulators, data recovery and so on. Hacker heaven and it’s only a ten minute walk from AI.

We assessed NDAD against the OAIS standard and it does pretty well. Since TNA essentially do the selection of the databases, NDAD have no negotiation in the Ingest stage, but together TNA and NDAD are a functioning OAIS archive. And it’s only a ten minute walk from AI! It feels a bit like saying you live only ten minutes from Buckingham Palace.

This was a brief intro to RM discussing how it fits in with the digital presentation process. I’ve never been trained in RM so it was useful for me and these are the highlights of what I learned:

RM is the efficient control of the creation, reciept, maintenance, use, retention and disposition of records. It’s an archival skill but overlaps with business analysis and it’s assisted by international standards such as ISO 15489.

Why do we need records? Well, they might be evidence, required for accountability, for decision-making and to record institutional ‘memory’. Good records are authentic, accurate, accessible, complete and comprehensive. They are compliant, effective and secure. I was told that RM assists and supports an organisation’s business processes; it identifies and protects vital records, ensures legal and regulatory compliance, provides protection against litigation, and allows compliance with Freedom of Information legislation.

With the growth of digital records, there’s obviously been a massive quantative increase in information. Digital records share the same issues as paper records such as acquisition, preservation, storage and retrieval but also present additional challenges. Digital records are characterised by being easy to create, copy, share, modify and store in multiple locations. They can be complex, transient, vulnerable, software and hardware dependent.

Good digital records management is an underlying framework to good digital curation.

A sound migration plan is essential to good digital RM and inherent in the preservation planning recommended by OAIS. A migration plan is an essential part of ensuring that formats are retained and readable throughout their lifecycle.

Of course, digital records have metadata which also needs to be managed and assists with the authenication of a record. I was told that good electronic records management policy should cover the creation and capture of all corporate records within the RM system. It should cover the design and management of indexing and naming schemes. It should offer policy on the automated management of metadata, for retrieval and retention. It should ensure that records are ‘locked down’ to ensure their integrity and security. It should also provide guidance on the retention, preservation and destruction of digital records.

We discussed which type of records are selected for management: vital records needed to sustain the organisation’s business; records essential for legal compliance; and records with mid to long-term administrative value. These kind of things should form the basis of a selection policy.
It all sounds like archiving to me with the exception that there’s more destruction in RM. I do see how RM is more business focused though and the issues of preservation are not always so difficult when you might be retaining records for shorter periods of time. Still, there’s no reason why RM shouldn’t fit into an OAIS environment. The main characteristics of selection, validation, fixity, preservation planning, metadata standards and retrieval/access are clearly very simmilar. Perhaps Fiona or Lynda can explain more to me when I get back.

The title of this class was ‘File Formats: Matters to Consider’, and I found it fascinating.

First, we were shown where file formats fit in the hierarchy of the IT system:

Semantic Layer
Actions Layer
Format Layer (Alright!)
Filesystem Layer
Media Layer

Then an anecdote about how some file formats and their creating applications are better used for some tasks and not for others. The tutor knew someone who wrote a novel in Excel because he didn’t have any other software to hand and I guess curiosity didn’t get the better of him either.

We did a quick exercise in what features to look for in a file format for preservation purposes. Not too difficult:

Open, documented, widely used and therefore supported, interoperable over different Operating Systems, lossless/no compression, metadata support, etc. etc.

Another anecdote was that ten years ago, two men wrote a book detailing over 3000 graphic file formats. As the number of formats grew, it was revised and issued on a CD-ROM. Now it’s updated on the web. I’m sure Tim would love it.

I’ll state this here: ADAM handles two graphic file formats for a reason. They are both open, documented, widely used, well supported, interoperable and have metadata support. The list of supported graphic file formats may double or triple over time, but 3000+ demonstrates what an industry digital archives are having to deal with.

If you want guidance on file formats (and who doesn’t?), then look no further than these fine institutions:

AHDS
FCLA Digital Archive
Harvard University formats registry
PRESTOSPACE
ERPAnet file formats
Library of Congress (my favourite).

We finished up by looking at the conversion of file formats, something which presents problems when you want to preserve the original integrity of the file’s content but in a more suitable or non-obsolete file format.

I could go on about file formats but let’s face it, we’ve both had enough for one day. Let’s talk at length in the ‘breakout’ area when I get back, OK?

Basically, this was a discussion on DSpace and the OCLC. Fedora was mentioned but only briefly. That’s OK, because Fiona, Damon and I attended a conference on Fedora last year. Damon’s an expert so ask him all the questions about Fedora… The implementation of a ‘trusted repository’ is central to digital archiving and the two main course documents are the OAIS standard and the follow up document, Trusted Digital Repositories. The TDR document basically goes through all the attributes and responsibilities that an OAIS compliant have. The report defines a TDR as:

A trusted digital repository is one whose mission is to provide reliable, long-term access to managed digital resources to its designated community, now and in the future.

It’s a useful document for testing how well your institution is doing.

DSpace is a repository system that’s been developed at MIT. It’s very popular (OK, so that’s a relative term…) in the USA and some UK institutions use it too. From what I could see, it provides a customisable ‘repository out of the box’ and shares some functionality with a Content Management System.

DSpace has three preservation service levels, providing functional preservation through ‘supported’ (1) and ‘recognised’ (2) file formats and bit-level (3) preservation. I don’t think it is ‘OAIS compliant’ but clearly it follows the basic OAIS functional model of Ingest of Submission Information Packages, creation of Archival Information Packages and the creation of Dissemination Information Packages. The example we were shown worked very well for the submission and archiving of a document by an academic writer. From the Fedora conference we attended, I’d got the impression DSpace was a bit crap, but there’s some competition between the two systems so that shouldn’t be surprising. Fedora is a different animal really as it provides a suite of repository services which developers are expected to work with while DSpace is useable out of the box by people without programming skills.

OCLC, The Online Computer Library Centre provides a repository service for other institutions so is really an out-sourcing solution. Accessible over the web with OAIS-like functionality but, of course, still requires that you prepare your collection for Ingest.

UK Initiatives

JISC - Funding body
AHDS - Arts and Humanities Data Service
DPC - Digital Preservation Coalition (course organisers)
DCC - Digital Curation Centre. Courses, conferences, online forum, specifically interested in curatorial issues.
UKOLN - Advisory service
UKDA - UK Data Archive
NDAD - The National Digital Archive of Datasets
UKWAC - UK Web Archving Consortium
TNA - The National Archives

Non-UK Initiatives

NARA - Electronic Records Archive (USA) Project with $300m funding
National Archives of Australia Digital Preservation Service
PADI – National Library of Australia’s subject gateway to international digital preservation resources
ECPA – The European Commission on Preservation and Access
RLG – Research Libraries Group
Library of Congress Digital Preservation
ERPANET – Electronic Resource Preservation and Access Network

Tools (these were all demonstrated and very interesting – I think we could use one or two of them. And they are all open source and written in Java which Merlin’s Team have expertise in).

PRONOM/DROID – PRONOM is a file format registry at The National Archives. DROID is a tool which works in conjunction with PRONOM.
JHOVE - Validates digital files and extracts preservation metadata. Very cool. Really!
National Library of New Zealand Metadata Extraction Tool
XENA – XML Electronic Normalisinig of Archives