A goodbye to linked-data special collections at NTNU

A while back I worked (primarily with Ove Wolden) on creating a way of making a linked-data-based digital publication system for NTNU university library’s special collections. The system went live in 2013 after a long period of work on getting the basics behind the scenes right. This is the story of what we did wrong and what we did right. (It is this one, look now (it supports conneg and has embedded data) as it won’t be there soon.)

I can start with the mistakes — as de facto project manager, all of the mistakes made are entirely my own fault and I apportion no part of the blame for failures to anyone other than myself. Let me also make it exceptionally clear that Ove should be given no responsibility for any mistakes that we made as I also functioned as technical lead. Successes, on the other hand were certainly due to all of the effort from Ove and a few others doing metadata and digitization at NTNU university library.

The first and biggest mistake was never actually having a project. This sounds ridiculous to me now, but at the time, the “project” was carried out initially — from 2010 — without any official capacity. The original prototypes were written in spare time and in the stairwell at a conference where we were demoing the prototype. The plan was “to do something with special collections and linked data”. A better description was to create a webpage that it was easier to maintain (the special collections until this point were presented item by item in manually produced HTML).

Later, the project gained official sanction, but no project structure materialized. I tried making the project official by producing a project plan and a few odds and ends of documentation like progress plans and risk analyses. I was the only person allocated to the “project” and the rest of those working around it were to do so only in the capacity of doing their job.

The second major failure was my complete inability to see the actual requirements of the special collections management; I thought we were working towards a solution for publishing digital documents, whereas the requirement was something quite lower — web pages for the four hundred or so documents that had been digitized. I should have realized this — but in the absence of a clear project mandate I bullishly attempted to solve a problem-that-wasn’t. I think I understand now why: I worked closely with the digitization and metadata workflows and saw the needs and opportunities; this blinded me to the smaller concern of presentation of the results. I was of the belief that a sturdy fundament in well structured data and metadata was necessary for the project to succeed, while I think management would have been happier with a pretty set of pages that worked “now”. Given the scope of the “project” and who was footing the bill, I’m afraid I think I could have done better here. (In my new role as a consultant, I’d still advise in the same way, but am much more receptive to clients’ needs.)

This is a classic conflict, and something that leads me to another major failure, which was my belief that we could change the way things work within IT at the university library. One of my major concerns was embedding the project in the main library IT infrastructure; the department that had the special collections was notorious among the IT department (my place of work) for doing things that weren’t according to the books and were somewhat sneered at for it — let’s face it, no-one in IT likes to work with novices that do half-cocked things in IT that later have to be maintained. My aim in embedding the work of the special collections was twofold: Firstly, I wanted to pull the special collections people into the warmth as they weren’t malicious in their actions, they simply tried to do the best they could with limited resources, complex tasks and rather a lot of institutional resistance. Secondly, I wanted — selfishly — to work somewhere that did actual development, rather than the “acquisition” of pre-configured systems; I wanted to work with linked data.

My failure in this latter regard was total and final and resulted ultimately in me changing workplace; after discussion with management, we realized my future lay elsewhere. I felt a bit sad about this at the time as I liked my colleagues, but I felt I was not moving forward in the way I wanted. It should be understood that our parting was in no way acrimonious as I was moving to a much better paid job and working with things that I enjoy (development of linked-data library systems); our parting was because a modern university library doesn’t have space or resources for development of low-level systems like the one we developed.

Situated as the “project” was, it was difficult to find a way to complete the work without incurring problematic feedback from my own department or the one for whom I was working. Not doing things “right” meant the project would not be embedded in IT routine; spending too much time doing non-essential backend work meant that no progress was seen.

Our successes were quite small; we managed to create what I aimed to do. We take linked data and we use it to power the publishing platform, drawing in linked resources to provide data entities that supplement the entities we create ourselves. We publish the digitized content alongside an HTML view of the linked data metadata and provide consistent APIs for accessing the data.

We managed to systematize production of digitized documents, streamlining production through automation and use of synthetic keys. We made a solution that ironically speeded up production immensely, even though this was never part of the “project”. What really drove me here was working with people who worked themselves very hard, but not very smart; dedication applied wrongly makes me depressed and I think we changed this a bit for the good.

We created a site that ranks highly in search engines because we embed suitable metadata in a way search engines use structured metadata. We also eat our own product in our internal search engine (which is a separate application).

I think that the site was the first of its kind within the domain: an actual production site for digital content that used linked data as its core technology, but not as an exercise of using linked data, but rather as one of creating the product. This, I think, was a success for us.

All-in-all the “project” was always doomed, but the failures should stand as a lesson for others. I think the most telling failure was that nothing has changed IT-wise and the work we have done seems to be chalked for replacement by a pre-configured, acquired solution. I think that that’s a shame as the market currently holds nothing at all of interest — there is nothing based on linked data or that even does linked data at all (some output a few lines of RDF, but this is not really going to cut the mustard these days). When I have said this, I understand fully the motivation to not want to opt for actual development. It can be costly and result in nothing when it is done wrongly.

Every mistake I made can be avoided through proper processes and methods that any professional development team must know and use. I knew this to begin with and my abortive attempts to get the project together officially testify to this; I should have walked away when it didn’t happen because I really knew that this meant the end.

To be better in what I do, I moved on and I have had to eat humble pie for many months now as I learn things the hard way — I now work with an expert in development methods and I’m battered and bruised at the end of a long week these days — but this trial by fire will result in better software and better routines and it will make me better.

Posted in Uncategorized
4 comments on “A goodbye to linked-data special collections at NTNU
  1. I wanted to say how much I appreciate this post Rurik,
    It is very hard to find people willing to be honest and humble about their projects and this makes it difficult for people like me who are trying to learn systems analysis skills. We were taught in a recent lecture that requirements gathering and their analysis is an essential step and yet one that is often skipped in the development process. Interestingly the assignment designed to test our systems analysis skills has no ‘requirement’ for requirements so to speak and so it feels as if they are further encouraging the omission of this step. It’s valuable to have a chance to see how this can potentially damage a project in the real world and it can’t have been an easy process to realise and then share your mistakes so thanks.
    On another, related note I’m interested to know about the analysis techniques that can be used to develop linked data systems. So far the agile / object oriented approaches we are using rely heavily on relational database entity modeling which makes certain assumptions about the solution before you have reached it. With this in mind if you know of any resources that describe alternative formal approaches to systems analysis I would be super stoked to have a look at them.
    Anyway, thanks again and a big thumbs up from me for sharing

  2. brinxmat says:

    Thanks for being so kind!

    Regarding your question, I think my response here is that I don’t know. Each collaboration differs. At the moment we use feature-driven development techniques, which I think work well, but are very demanding on the folk doing the ordering of the software.

    More over a beer sometime, right?

  3. Well that does give me somewhere to start thanks – I’ll have a peep at feature-driven as an alternative approach and see if it leads me to anything else.

    You can have a beer and I’ll have a cider

  4. Tarje Lavik says:

    Hi! We did not get to work together when you were at NTNU, but you got UBB going with Linked Data. Grateful for that!

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s