Pulling Back the Curtain on Transatlantic Collaboration

When we began the conversation that led to the creation of Mapping the Scottish Reformation in the spring of 2017, neither of us could have anticipated how quickly MSR would take off, or how incredibly generous the Scottish History and Digital Humanities would be as we navigated the contours of this new and exciting part of our respective careers. One of the things that we remain intensely grateful for is the transparency of others who have embarked on their own digital humanities endeavors. To a person, our fellow scholars have been willing to peel back the curtains of polished finished projects to discuss how the various steps, strategies, and challenges along the way. These “inside looks” have been nothing short formative for our work.

In that spirit, we wanted to offer some insights into our own collaborative process in gathering and visualizing our data, applying for grants, and establishing good communication and a feasible workflow—all while working on two different sides of the Atlantic.

Traditionally, scholars have collaborated from within a single institution or country, eliminating the potential difficulties of different time zones and travel restrictions, not to mention disparate sorts of bureaucratic norms and red tape. Yet with the tools of Zoom, Skype, FaceTime, Google Docs and Sheets, and cloud storage, the time is ripe for transatlantic collaboration, particularly in a field like Scottish History. Scotland’s diaspora, tourist industry, and uniquely fascinating history (we’re biased, we admit!) has meant that scholars and genealogists in North America and well beyond are interested in exploring the nuances of Scotland’s people and past. We think a collaborative project like MSR shows the potential of transatlantic collaboration to invigorate the study of Scottish history—but how exactly have we made this work on a day to day basis?

One of our slightly less awkward live transatlantic collaboration shots from spring 2020

The success of our collaborative process has hinged on three interrelated things: a clear digital workflow, regular communication between ourselves and other interested colleagues, and flexibility. The two technologies that have underpinned this whole project to date—Google Sheets and Wikidata—are stored entirely on the cloud and updated in real time, meaning either of us can access them at any moment and, crucially, at the same time. In the first data gathering phase of this project, we divided up the presbyteries within the Synod of Lothian and Tweeddale and recorded our findings separately but on the same Google Sheet. We communicated constantly about any editorial or content challenges via text, email, or on a Google Doc called, fittingly, “concerns.” Both of us are, by habit, very quick responders to emails and texts, meaning that we could often troubleshoot questions such as “should we count this as the precise date of installation in a parish?” or “how do we record the first post of this schoolmaster turned minister” in real time, so long we were both still in working hours on our respective sides of the pond.

The “concerns” Google Doc also meant that we could make shared notes of oddities or issues in the records and respond to each other along the way. We highly recommend that any collaborators, especially early on in their process, keep some sort of shared running document like this, as it will come in surprisingly handy for grant-writing. You’ll also want to use a shared cloud service for any static or finished documents, such as Dropbox or Box. If we were a bigger project team, a collaborative workflow service such as Trello might have also been very useful.

an example of one of our many Google Sheets pages

Our workflow has also been strengthened by frequent communication about our progress in going through the presbytery records and regular planning for what comes next. We knew, for example, that the stage after data collection would involve entry of that data onto Wikidata and the running of test visualizations. Chris was, from the outset, more interested in (and, I’d argue, more capable of!) learning about and working with Wikidata, Leaflet JS, and a range of mapping technologies. I, on the other hand, was very content to continue augmenting our dataset, which meant that Chris was able to build much of our technological infrastructure while I focused on finishing the remaining presbytery records. The recognition of different strengths and interests, and clear communication about those, is essential for distribution of labor and creating the most efficient workflow, especially when looking ahead to next steps. 

Our process as we have moved through Stage 1 (data-gathering) and into Stage 2 (building the pilot user interface) has been also greatly enhanced through regular communication with other team members, collaborators, and interested colleagues. Mackenzie Brooks, W&L’s Digital Humanities Librarian and a member of the MSR team, has been an indispensable go-to for technology related questions, as has Ewan McAndrew, the University of Edinburgh’s Wikimedia guru. We have frequently turned to W&L’s associate provost and Advisory Board member Paul Youngman for questions and suggestions on funding. Our Advisory Board as a whole has been a constant well of support, insights, and critical questions that have guided the development of our work so far. And of course, all the colleagues from the Scottish history and genealogy communities who have commented on our social media posts, offering their thoughts and queries, have helped us imagine what MSR could become.

This list of people we’ve leaned on is long, but the lesson here is clear: a digital humanities project may have only two co-directors, but the best ones are born of multiple communities. To others embarking on such projects, don’t be afraid to reach out and ask questions, even the most basic ones. In fact, ask them as widely as possible, and share your work-in-progress whenever you can in presentations and in print, rather than waiting to unveil a more polished final product. 

our first joint presentation, in January 2020 at Edinburgh’s Centre for Data, Culture and Society. Other essential venues for sharing our work-in-progress have included the Digital Archives in the Commonwealth Summit, the Ecclesiastical History Society Podcast, the Scottish Indexes Conference, History Scotland, and the International Review of Scottish Studies.

Last, we want to emphasize flexibility as a core aspect of the collaborative process, especially when the project team is international. The UK and US have different calendars for the academic year, varied expectations for the balance between research, teaching, and service/administrative work, and assorted rules for things like research leave and buying out teaching. Because of this, there will invariably be certain times during the year where one of us is too swamped to devote much time to Mapping the Scottish Reformation. Flexibility has been key, with ourselves and each other, especially as we’re both at teaching-focused universities. It is quite common for one of us to say “as a heads up, I won’t be able to get to this for at least a week,” or “sorry, I lost the thread in the abyss of my inbox, can you resend?”. This sort of dialogue that foregrounds flexibility has been crucial, especially when designing work schedules for funding bids. We would certainly recommend that any collaborators beginning a research project as ambitious as MSRhave a conversation about their other research commitments/goals, the typical rhythms of their academic year, and any anticipated challenges.

Over the past three years, we have applied for numerous grants; gone through nearly 10,000 manuscript pages; collected data on 874 clerical posts, roughly 500 significant events (such as depositions and suspensions), 116 parishes, and 7 presbyteries; and begun work on our pilot user interface. Throughout, a clear workflow, good communication, and mutual flexibility has kept things running smoothly and kept us excited about this work. We hope this is some use to the followers of MSR, and we will keep pulling back our own curtain as this project progresses!  

Mikki Brock

Entering Stage Two: Mapping Parishes in Lothian and Tweeddale, 1560-1689

With Stage One of Mapping the Scottish Reformation nearing completion, we now have a large dataset of ministers, detailing their movements and important moments of their careers. We have parsed over 9,000 pages of manuscripts from presbyteries across the region of Lothian and Tweeddale. With these significant task complete, we can now turn our attentions to how we intend to present this data in Stage Two of the project.

Stage Two of Mapping the Scottish Reformation is generously funded by the Strathmartine Trust and will see us explore our user interfaces for the first time. Up to now, we have deposited our data onto Wikidata and used the built-in tools to test our material, to see the locations of gaps, and to create quick mock-up visuals that we think users might find useful. You can read more about our use of Wikidata mapping tools and how we structure our dataset here.

A screen grab of using the mapping tools built into the Wikidata Query Service.

Followers of the project will have seen some of the demos we have been able to quickly put together that visualise the breadth of our data and hint at some of the ways we can put it to work. Critically, Stage Two of Mapping the Scottish Reformation will use even more powerful mapping technologies to create visualisations that load faster, run more smoothly and show even more data. Before we formally embark on this process, we have spent the last couple of weeks dipping our toes into some of these more powerful mapping technologies.

The Wikidata Query Service — and the SPARQL queries we write to ask questions of our data — sits at the heart of our project. It allows our data to be open for other researchers to use (and build on) in the future, but it can also be quickly exported and patched into other programmes/services. The first stage of testing more powerful maps is to take the result of a SPARQL query, add a few modifications to the code, and export it into a TSV (tab-separated values) file.

The Wikidata Query Service allows the results of queries to be exported into various file formats

We exported a SPARQL query that shows all of the parishes in the Synod of Lothian and Tweeddale between 1560 and 1689, as well as a label that showed in which presbytery each parish sat. This has the fewest values of any of our datasets so we thought it would be an easy place to start! The resulting TSV file is effectively a huge spreadsheet: and as much as I like spreadsheets, I think it would be stretch to call it attractive or user friendly. The key thing here is that we have the latitude and longitude data in separate columns and have the key bits of information we want to display to users. Our test file included around 120 lines.

The resulting TSV file is functional, if unappealing

Formatted correctly, a TSV file like this one can be imported into GeoJSON, an open format mapping service that allows users to input geographical data and show them on maps (note: have you noticed our constant use of open source and access services?!! It’s no coincidence!). Users can either add points manually or, critically for us, add geo-referenced locations in bulk. Having uploaded the file, the result is a much more appealing map that includes more attractive and comprehensive icons and the ability to select different mapping layers. We can even add different mapping tiles, using a service like Map Tiler, enabling us to test different backgrounds.

The beauty about GeoJSON is that it transforms that ugly TSV file into something more machine readable. Unfortunately, GeoJSON doesn’t allow you to automatically export your map or embed it into a website like this one. This is where Leaflet.JS comes in.

Leaflet is quietly taking over the world of internet mapping applications, but its huge functionality comes at a significant technical cost: we aren’t in the world of drag and drop or ‘what you see is what you get’ editing anymore. The benefits of a little perseverance, however, are huge.

Leaflet demands an understanding of CSS, or at least an understanding of what to swap into lines of code cribbed from GitHub and when. This process was made infinitely easier by Leaflet’s own tutorials and, in particular, by this superb tutorial on YouTube by Eduonix. The key here is to take the code generated by GeoJSON and to copy it into our HTML file (shown below in Sublime Text markup editor). You can see how the data from GeoJSON is shown just below the various lines of code for headers etc.

After generating our map, a few fairly simply lines of code allows Leaflet to then take the data from GeoJSON and display it, as well as adding a custom mapping layer and popup menus that are, in theory, infinitely customisable. The resulting map can be exported to HTML and embedded into a website. And because the database values were all pasted into Leaflet, at least for the moment, Leaflet doesn’t have to request the info each time the page loads. The result is that the embedded map loads almost immediately.

You can play with this simple demo, showing all of the parishes of Lothian and Tweeddale between 1560 and 1689, below.

Notice that we have made use of the NLS Historical Maps API to plot the points on a historical map. This dynamically adjusts to a different background map depending on how far a user zooms in or out of the map.

If this seems like a tremendous amount of effort to go to in order to embed a map, then I suppose you’re right. What’s important here is that we have demonstrated that the data we manually took from manuscripts within the National Records of Scotland, passed into Wikidata, and then queried using SPARQL and the Wikidata Query Service, can be exported, customised and presented in a way that it as visually friendly as we can make it!

This is just a test, but it reflects the process we will go through during Stage Two of Mapping the Scottish Reformation, with colleagues from our international Advisory Board and our technical friends and colleagues at the University of Edinburgh. Ultimately, this process will allow us to create a number of interactive visualisations that will distill the months we have spent looking at handwritten archival material and make it more accessible. So while we’ve been recording, storing and querying the Scottish Reformation up to now, Stage Two of this project will allow us to start intricately mapping the Scottish Reformation.

Chris R. Langley

Driving Our Data

In the last five months, we have devoted a lot of time to considering how to structure the data that will drive MSR’s visualisations. And while Stage II of MSR will look to develop the user interface of the project, we wanted to take some time to show you ‘under the hood’ of what we have done so far. And what there is still left to do (spoiler: there’s lots)!

The information we collect on ministers from across Lothian and Tweeddale between 1560 and 1689 is collated on Wikidata. Wikidata allows us to link together different aspects of clerical careers and to build search queries to test the data we have collected.

If we’re looking under the hood of MSR, then this stage of the project was about making sure we had an engine. Initially, we took data from volume one of Hew Scott’s Fasti Ecclesiae Scoticanae (scraped as part of our preliminary work in 2016-17) and added it to Wikidata. This consisted of recording the names of ministers, the locations of their parishes, the dates of their tenure, and any references to ministers being deprived or deposed. A typical example would look like this:

Each minister, like any other item on Wikidata, gets a ‘Q’ number and each characteristic (gender, occupation, residence) is given a ‘P’ number. Moreover, each parish is given a set of coordinates, so they can be plotted on a map (you can read about our very earliest work on mapping parishes here).

The beauty of Wikidata is that, with these items being so neatly split into categories, we can run queries on the data using the Wikidata Query service and some basic SPARQL. These queries allow us to ascertain certain information about our ministers or even view maps like the one at the top of this page.

The Wikidata Query service and its table view

The dataset of ministers from the region covered by the Synod of Lothian and Tweeddale consists of over 800 entries, each with four or five characteristics. With this data, we have the very first, simple, visualisation of all of the ministers active in Lothian and Tweeddale. This provides an alternative to genealogists and scholars looking for an easy, accessible, way to visualize Hew Scott’s work from Fasti Ecclesiae Scoticanae.

The next step is to augment Scott’s rudimentary data with manuscript material. This will significantly expand the dataset, adding new information (such as ministers overlooked in the Fasti or the full details on clerical suspensions and depositions), and amending some of Scott’s inconsistencies. MSR is built on manuscripts housed in the National Records of Scotland and, because of the richness of this material, we have devised a very specific way of recording this information, all captured through Wikidata. Let’s look at John Hogg, the controversial minister of West Linton, the Canongate, and South Leith for an example.

An example of collating manuscript information onto a Wikidata entry

Here, we cite manuscript material that provides additional, accurate, information on Hogg’s career. We add details like this for each aspect of clerical career that we find in the manuscripts. Through Wikidata, we can record the repository, the archive accession number and the folio/page reference of relevant material in the manuscript.

NRS, CH2/295/3, f. 39

The image above shows the manuscript page on which details of Thomas Hogg’s career can be gleaned. By adding this material to our database, MSR provides both a finding aid to manuscript resources and also a way into a world of archival material that can often be incredibly challenging for non-specialists to view.

The map at the top of this page is a resource in its own right, but it isn’t static. We are now working to add more manuscript data to the dataset, so it expands and can offer new insights. We have multiple data points relating to over three hundred ministers (and counting) taken from manuscripts across Lothian and Tweeddale to add to Wikidata. Once uploaded, all of this information will result in the most extensive prosopography of Scottish ministers ever attempted.

We hope this view under the hood of MSR shows the extent of our efforts to structure the data of ecclesiastical careers in early modern Scotland. We also hope it shows you the ways in which the manuscript material we are mining will add considerably to our understanding of early modern religion. In the next year, we will devise a more user-friendly and customisable interface, so users from a wide range of backgrounds can explore this data in more detail: tracking clerical careers, observing trends in the data and seeing big religious changes over time. In the meantime, we’d love to hear your feedback on our work so far: it genuinely helps us shape the direction of the project.