Testing Wikibase with Mapping the Scottish Reformation

One of the goals for our National Endowment of the Humanities grant in 2019/20 was to explore ways we could capture the complexity of early modern clerical careers in a digital format. We needed a system that was robust enough to handle large volumes of textual, temporal and geographic data, as well as offering some methods to visualise that data (either natively or via export). In January 2020, we started the process of compiling our dataset on Wikidata, an open-linked data repository. 

All the data that sits behind our first public website is available on Wikidata. We have outlined how we use Wikidata in previous posts on this blog. Of course, as an open repository, using Wikidata presents risks around data vandalism, sloppy editing (something of which I have been guilty myself) and commercial reuse of data. As we look to the future of Mapping the Scottish Reformation, we have started exploring other methods of managing our data that can tick the same boxes as Wikidata but offer features that we might need as our dataset grows to encompass different types of material (some of which may not be appropriate to full open access platforms like Wikidata). We’ve been working on our own private laboratory version of Mapping the Scottish Reformation: this will allow us to test new ideas like data ontologies or visualisations, without worrying about affecting Wikidata. Enter MSR Test on Wikibase. 

Wikibase Logo

Wikibase is open-source software for creating knowledge bases. It is effectively the backend of Wikidata: allowing information to be inputted and linked. Unlike Wikidata, you can create your own instance of a dataset. This can be hosted online on a new service called Wikibase Cloud or it can be managed privately on a personal or commercial server. Several Galleries, Libraries, Archives and Museums have used Wikibase as a way to better understand their collections (see the excellent work of Koninklijke Bibliotheek Nederland and the National Library of Wales).

To get started, one has to register for a Wikibase Cloud account (at the moment, this is invite only) or download a local version of Wikibase. For the purposes of our test, we have chosen to use the cloud version. Like most Wikimedia projects, there is an active community of users, many of whom offer guides to help get you started (we relied on the sage advice of Jason Evans at National Library of Wales!). Once you’ve set up your Wikibase instance, you are greeted by a completely empty database. Unlike Wikidata’s 106.86m items, your new Wikibase instance contains nothing. Zero. If you’re used to Wikidata, this experience is rather jarring. The first step to populating the database is creating each property you need: think of these as the headings in a spreadsheet. Here’s a selection of some of the properties we created:

A screenshot showing some of the properties we created

Once these properties are in place, key items in the dataset are needed. For us, these are key concepts – such as ‘minister’ or ‘presbytery’ that help us categorise items. These properties and items create the pieces that we need to build our data structure: the framework to capture information about clergy careers. 

Due to Wikibase’s structural similarities with Wikidata, we can use bulk upload tools like Quickstatements to import data into the dataset. This allowed us to bulk upload the names and coordinates of every presbytery region in early modern Scotland. Moreover, due to those similarities with Wikidata, Wikibase can be searched using its own Query Service. Results can be downloaded into CSV, TSV or JSON formats or visualised in maps, tables and network diagrams. Here’s a visualisation of most of those presbyteries, all using the Wikidata Query Service, but located in our own Wikibase instance:

An early visualisation of Scottish Presbyteries made using Wikibase

While the Wikibase instance doesn’t include many ministers yet, we hope you can see some of the advantages of this system. All of the data you have seen in this blog post exists outside of Wikidata. Moreover, while users can see the data here, only members of the project team can edit it, meaning we have greater editorial control. Finally, as we move forward with the project, Wikibase may allow us to offer different levels of open access when compared to Wikidata. For example, all of our data on Wikibase is made available on CC BY-NC 4.0 licence, meaning it is free for personal usage, but cannot be reused for commercial purposes. This may be an important feature if we move into archival collections that prohibit commercial usage. 

Being so easy to set up, Wikibase may offer humanities scholars an easy to learn alternative to resource heavy database software. Moreover, Wikibase gives us the freedom to structure data in different ways or to test new approaches, without worrying about affecting Wikidata’s other users. MSR Test on Wikibase serves as a laboratory in which we can develop the next stages of the project. If you think Wikibase might be of use to you and your projects, get in touch with us.

You can keep track of our Wikibase progress at https://msrtest.wikibase.cloud/ .

Chris R. Langley

The Open University 

Expansion and Next Steps

In March 2021, we released the updated version of the Mapping the Scottish Reformation website. This included details of almost 700 ministers’ careers spanning the early modern period and began to document the lives of their spouses. Our public website contains information on individuals who lived in the Synod of Lothian and Tweeddale. In this post, I want to share information on the tentative steps we’re now taking to expand our dataset. Over the last twelve months, we’ve been busy behind the scenes!

The core of our work takes place on Wikidata. Wikidata is an open repository for linked data. What this means is that we don’t have to duplicate information: for example, if a minister has already cropped up in our research, we can simply add to his item rather than start all over again. We started by using volume five of Hew Scott’s Fasti Ecclesiae Scoticanae, covering Angus and Fife (and then other regions), to create basic profiles for every minister in the region. We sometimes call these ‘skeleton profiles’, before we can confirm the accuracy of Scott’s information and flesh them out with data from manuscript sources.

Using bulk upload tools like QuickStatements, we can enter all of the ministers recorded by Scott who served in a particular parish in one go. This tool means we don’t have to go into Wikidata and add each new profile by hand (and, yes, we’ve added hundreds of clergy to Wikidata one by one: we know how long it takes!).

QuickStatements looks daunting, but all of that text creates entries that are standardised in how they are structured. This complex-looking syntax creates an entry that looks something like this:

Notice how we always include references to the original text in our entries (right down to the page number). This is absolutely best practice in Wikidata hygiene and follows our previous example of developing ways to reference manuscript sources on Wikidata

Several months of data entry like this have allowed us to go through multiple different regions. And while this is still a work in progress, we can now announce that we have more than doubled the size of our dataset:

~1,500 ministers 

~2,000 career moves

~800 alma maters with over 40,000km travelled between university and first parish charge 

The geographic spread of our dataset is now much broader. Here are two pictures,the first showing the extent of the dataset behind version 1.1 of the website and the second showing the expanded version:

As I mentioned earlier, these are ‘skeleton profiles’. They have not been checked for accuracy against the original church court records or had valuable data from manuscripts added to them, and we have not recorded information about clerical families or significant events taking place in a minister’s career. Moreover, by using only Scott’s Fasti, we have not yet captured the richness of the extant manuscript sources that allow for an in-depth assessment of clerical careers at a national scale. This represents only the first step in our journey to expanding Mapping the Scottish Reformation to cover all of Scotland. 

Currently we are seeking major funding to expand our dataset to the whole of the country using the rich archival records held at National Records of Scotland. In the meantime, you can play with the expanded dataset at the following link or use the embedded map below: https://w.wiki/5jN2

Chris R. Langley

PS: For a more polished version of the expanded dataset visualisation, visit this test website.

Being Open: Raw Data & Mapping the Scottish Reformation

In version 1.1 of our project website, we introduced a new feature: a button on each map that allows users to download the results of their chosen query in JSON format. While this seems like a small and rather specialist gesture, this change to our website underlined a fundamental aim of Mapping the Scottish Reformation: the desire for our data to be open for future researchers to use, without our input.

The download button that launched a thousand…datapoints?

We all know that digital projects in the humanities are often labour intensive. The initial effort to gather information, in our case from manuscripts, is time consuming and requires considerable amounts of skill (reading the handwriting, knowing what you’re looking for, being able to parse lots of information, etc). Then there is the encoding of that data into structures that a machine can read and query. Up until relatively recently, a great many digital projects in the humanities have developed custom methods to deal with these problems, resulting in beautifully crafted projects. However, these methods often (though not always) have one flaw: they usually don’t play nice with other projects. The data you can extract on the website or app is often limited to the types of questions that were originally envisaged by the project’s designers. These projects aren’t easily interoperable.

Think about it this way: when you wanted information from your favourite or most used online humanities resource, like a historical database, it is highly likely that you wrote down the data you found by hand or in a document on Microsoft Word or equivalent. You might have taken a screen grab. This is your record. This is fine if you need a relatively small piece of information, but what if you want to leverage the power of a significant proportion of the dataset of that entire website? You can’t use it, you have to laboriously (and manually) write down the data you need, or you’re forced to contact the creators of the resource and ask for permission to access the raw dataset. The creators (if they’re still active) will then need to reframe their data so it works for your purposes.

It is our contention that the creators of digital projects in the humanities should think about interoperability when considering the legacy of their projects. Legacy is much more than just the longevity of your website or resource (“we need x years of hosting costs” or “we need to ensure that this website doesn’t break whenever there’s a major iOS or Android update”). Longevity needs to be understood as the life and utility of the data at the heart of your project, long after the research questions that created your project are obsolete.

This process of siloing data in custom repositories is not helpful to developing humanities projects. As part of our talk for the Northern Early Modern Network in September 2021, we showed how considering three key features can help make a project more sustainable by encouraging interoperability.

Interoperability slide from our presentation ‘Using Digital Tools to Explore Early Modernity’

First is to be open about data structures. Scholars building digital resources have spent hours considering how to encode information from manuscripts into databases: basically this is a process of establishing how to make the complexities of these records machine readable. Unfortunately, up to now, projects rarely share how they structure their data collection, seeing it as something that is largely internal to the project. We’ve created standardised ways to capture complex data that will work in any project, right down to the references. We write about this process on this blog.

Second, offer your data openly online. While most data from digital projects is available for free and not behind a paywall, the data can never be scraped and extracted from the resource. At least, not easily. For us, Wikidata does the heavy lifting: no need to create a custom database, allows for quick prototyping of visualisations, and interfaces nicely with other platforms and programs. Everything is uploaded in accordance with Creative Commons Zero, meaning it’s free to use without a licence. The raw data of our research can be used by scholars for free long into the future and without the need to enter into lengthy negotiations with us about how to use it. Projects should explore similarly open data standards wherever possible.

Third, data should be exportable. People searching through your digital resource should be given the option to download their results. Why? Well, many archives have poor or non-existent wifi, meaning scholars may need quick access to information offline. I suppose more important, however, at least for the purposes of this post, is that having data available to download means that it can be immediately switched to different applications. Is there a visualisation one of our users wants to see but that we haven’t created (perhaps for a teaching resource, presentation, or PhD thesis)? Fine, they can now create it with the raw data. Want to use a new technology that we couldn’t incorporate into our website? The exportable raw data can be repurposed into the new or emergent technology. Having data that can be downloaded in a way that suits the user can empower them to perform a broad range of new projects that we could never have envisaged.

Data should be exportable. Here is a downloaded search from our website in JSON format.

We still have a way to go before we achieve all of these lofty goals as well as we would like. For example, our current website only allows results to be downloaded as a JSON, which limits its use to certain applications (TSV or CSV may be more attractive to a wider range of uses and a simple HTML list would be much easier to use for non-specialist users). Similarly, we could do more to emphasise our methods of structuring data on Wikidata to promote the availability of our dataset. And neither are we the first to consider these issues: colleagues at University of Edinburgh revived data from the Survey of Scottish Witchcraft (gathered between 2000 and 2003) and repurposed it by using many of these techniques (we owe a great deal from them).

Open standards come with some risk, but we would argue that the benefits are more numerous. We must ensure robust editorial standards for our work, so we need to ensure quality control of our data. Similarly, methods to acknowledge the original labour put into the data gathering becomes more difficult the further one moves away from traditional citation techniques like footnotes or endnotes. Nevertheless, sharing knowledge about data structuring and giving access to raw data in some form can enhance the interoperability of a project.

Ultimately, considering interoperability at the start of your project may extend the longevity of all of that effort in collecting data in the first place.

Chris R. Langley

Geoshapes, Wikimedia Commons, and the Early Modern Church of Scotland

Earlier this year, we published a blog post explaining how we mapped the ecclesiastical regions of early modern Scotland. We made the data available to view on our website and the resulting files can be downloaded from Github. In this follow-up post, I want to explain how we can broaden the audience of this geographic data even further by using Wikimedia Commons.

Jan Ainali and Albin Larsson recently showcased how users of Wikidata can query data in a way that highlights areas on a map, rather than just single points. An underused feature in Wikidata is the ability to include geoshapes stored in Wikidata Commons as properties in Wikidata items. A typical example of this functionality would be for Wikidata items showing for national parks in the UK:

Wikidata item Q15052206 – Brecon Beacons — showing both a coordinate location and a file path for a geoshape

The Wikidata property ‘geoshape’ (P3896) can link directly to a file stored in Wikimedia Commons. Here is the map of the Brecon Beacons:

Wikimedia Commons data source: geoshape for the Brecon Beacons

Rather than just showing a point on a map, the geoshape allows users to see precise boundaries of a certain area.

Following AInali and Larsson’s excellent tutorial, a quick SPARQL query can bring up all of the national parks in the UK, including other bits of data, like when they were established as national parks:

Output of Wikidata query for all national parks in the UK

Being able to link Wikidata and Wikimedia Commons, as well as being able to visualize areas as well as points in Wikidata, has a huge range of potential applications. For our purposes, I decided to upload to Wikimedia Commons all of the synod shape files we built earlier this year. The remainder of the post will detail some of the steps involved in that process.

I started by taking each geojson shapefiles that we had created earlier this year. These roughly mark out the different synod regions of the early modern Church of Scotland (minus Argyll). I then tidied these files using Mapster’s ‘Right Hand Rule’ tool, which brings the geojson into line with the format Wikimedia Commons will accept. You can then upload this into Wikimedia Commons using the data tool. Simply point your web browser to the page where you want to upload your data:

https://commons.wikimedia.org/wiki/Data:[insert your file name here].map

You can then click ‘create this page’ to add your data.

Creating a new data file on Wikimedia Commons

You can then create your data file, by pasting in the text from the geoJSON. Wikimedia Commons also demands that you add several lines to the geoJSON detailing the data source and copyright details. There is even a little debugger at the bottom left of the text panel to show you if there are any errors in your text:

The data entry panel in Wikimedia Commons. Note the debugging icons on the bottom left

Publishing the data will make it available, openly, to anyone on the internet. Also, Wikimedia Commons is clever enough to render your datafile into a shape overlaid on a map:

Wikimedia Commons displaying map data for the Synod of Perth and Stirling

Now this data is safely deposited on Wikimedia Commons, we can add a new property — ‘geoshape’ (P3896) — to our Wikidata entry for the Synod of Perth and Stirling. We populate this item with the file path shown on Wikimedia Commons:

Adding the Wikimedia Commons file path to the ‘geoshape’ property (P3896) on Wikidata

I repeated this process for all of the synod region shape files that we created earlier this year. I then modified the query that we built earlier: so rather than searching for national parks, we can now search for synod regions of the early modern Church of Scotland. Here is the result:

This takes the geoshapes we built earlier this year and makes them more widely available by contributing to Wikimedia Commons. And by linking these items to Wikidata, we can capture more information about ecclesiastical synod regions in the sixteenth and seventeenth centuries: making our queries more detailed and allowing for a new set of visualizations.

Perhaps more profoundly, this process shows how our commitment to open data enhances our project: any errors in the geoshape files can be corrected on Wikimedia Commons and further information about the synods can be added to each item on Wikidata (e.g. dates of establishment, names of moderators, changes in geographic scope, etc). And, finally, all of this stems from the supportive and open community around Wikimedia: building on the foundations of AInali, Larsson, and so many others.

Chris R. Langley

Newman University, Birmingham, UK

Mapping the Scottish Reformation: Website Update v.1.1

We’re very pleased to be able to share the news that the first update of our website is now live at maps.mappingthescottishreformation.org. Thanks to the generous feedback of our users across the world, the updated Mapping the Scottish Reformation site includes a number of improvements, bug fixes, and new features that should make it more useful and easier to use.

While we wanted version 1.1 to be an opportunity to respond to feedback and crush some bugs, we have also added a few features that were not quite ready for version 1.0. Chief among these features was the ability to download search results. Each map now includes a ‘Download results’ button that will export the data on the map into a JSON file.

Our new download function on the Tenures map

In version 1.1, the Education map is more powerful: allowing users to search for approximate graduation dates, and the amount of time between graduation and a minister’s first appointment. To give you a deeper insight into this data, we have added some new aggregations to the bottom of the search bar: showing the years in which most ministers graduated, the average time between graduation and entering the ministry, and even the average distance travelled between a minister’s university and his first parish in kilometres.

The new features on our Education map show the approximate year when a minister graduated and the distance travelled between university and first parish posting

We’ve also added a range of usability enhancements: from making the lines on the Journeys and Education maps thicker, adding manual Zoom controls to each map, to organising lists of names by last name, rather than first name.

It’s the little things: All lists of names are now ordered by last name, rather than first name. And there are manual zoom controls on each map for extra precision.

Version 1.1 of Mapping the Scottish Reformation is the culmination of Stages 1 and 2 of the project, funded by the National Endowment for the Humanities and the Strathmartine Trust, respectively. We have parsed over ten thousand pages of manuscript material, and tracked the careers of 654 ministers and over 400 of their wives in the region of Lothian and Tweeddale. Our website has attracted over sixteen hundred unique visitors since launching in December 2020.

As Stage 3 beckons, here is a full list of the changes we’ve made to v.1.1 of our website:

– Added an introductory message and guide video for first-time visitors, narrated by our friend and fellow Scottish historian, the inimitable Jamie Reid Baxter

– Added a guide video to the ‘About’ page

– Increased weight for lines in the Journeys and Education maps

– Rearranged all lists of names: now organised by last name

– Added a data download function to each map

– All new search options in the Education map

– All-new calculations in the Education tab: graduations, distances, appointment

– Fixed some data errors in the Education map

– Fixed some data errors in the Journeys map

– Default map tile changed from ‘Historic’ to ‘Modern’

– Map attribution has been moved and is no longer obscured by the search bar

– Added zoom control buttons to each map

Thank you for all of your suggestions and support. We are particularly grateful to our Advisory Board, the Interactive Content team at the University of Edinburgh, and Chris Fleet, maps curator at National Library Scotland, for their essential feedback on version 1.0 of the site.

Check our blog and on Twitter for more updates on the project’s development. If there’s anything you think we should be adding to the website, please get in touch at MappingScotsRef@gmail.com.

Mapping Scotland’s Synods, 1560-1689

As we step up our preparations for Stage 3 — venturing out from Lothian and Tweeddale and into the rest of Scotland — we are busy scoping out the extent of the work before us (spoiler: there is a lot). We have already entered Presbytery data to Wikidata. Now we turn our attentions to Scotland’s provincial synods.

Arrowsmith’s ecclesiastical map of Scotland (1825)

Many of our readers will know about Aaron Arrowsmith’s 1825 ecclesiastical map of Scotland. A scanned version is available on the National Library of Scotland website here. I’ve spent a lot of time looking at this map, but there are two problems with it, at least from our perspective: it shows the shape of the nineteenth-century Kirk of Scotland and, unlike so many of the terrific maps on the NLS website, it is not georeferenced.

Fortunately, NLS have adopted the International Image Interoperability Framework (IIIF) on all of its scanned map images. This means that the image can be exported to various different software viewers (ARCGIS, CanvasPanel, Mirador, Recogito, etc). Using the free website georeferencer.com, we imported Arrowsmith’s map and started the georeferencing process.

Georeferencer.com tied Arrowsmith’s map to modern map data

By using georeferencer.com, we were able to overlay Arrowsmith’s representation onto a modern map, pinpoint known places and features, and allow the website to adjust the nineteenth-century map to accord with modern-day cartography.

Georeferencer has an extra trick up its sleeve: the georeferenced map can be taken directly to MapTiler, a service that allows users to create vector shapes on top of georeferenced maps. Translation: we could now trace the boundaries of provincial synods shown on Arrowsmith’s map and use them in modern projects.

Tracking the Aberdeenshire coast on MapTiler

This is fiddly work: you start with a pretty generalised polygon and have to edit it, by hand, to accord with coastal features and the like. There is the added complication that Arrowsmith’s knowledge of certain parts of Scotland was less-than-stellar: thankfully, MapTiler allows users to alter the opacity of each layer so one can see how the modern and historical maps compare. Of course, there are editorial judgements to be made here, especially in areas where Arrowsmith’s map lacked key detail.

We’re using Arrowsmith’s nineteenth-century map here and our project covers the period 1560 to 1689. Unsurprisingly, a number of features had changed between the end of the period covered by our project and the moment of Arrowsmith’s composition. The Synod of Ross, for example, was established in 1707, so was shown on Arrowsmith’s map, but would not have been recognised in the period covered by our project. Similarly, the Synod of Glenelg appears on Arrowsmith’s map, but was not established until 1724. Such challenges meant that it would be remarkably difficult for us to represent the area covered by the Synod of Argyll, in particular, in a single shape file. Then there are changes within the period covered by our project: the Synod of Lothian and Tweeddale includes the region covered by Biggar Presbytery, an area disjoined from the Synod of Glasgow and Ayr in 1644. To capture the complexity of these changes, we would have to create new shape files for each significant revision to the Kirk’s ecclesiastical boundaries. Our shapefiles are no-less an interpretation of provincial synod boundaries than Arrowsmith’s nineteenth century work. Nevertheless, they offer a useful indication of ecclesiastical structures.

Synod boundaries as geoJSON

Once complete, each shape file can be edited further, exported as a GeoJSON (and then possibly converted into other shape file formats) and imported into any mapping platform. Using the workflow we detailed here, we took these datafiles and created a map in Leaflet showing most of Scotland’s provincial synods between 1560 and 1689. For the reasons stated above, we felt it best to not map the Synods of Argyll and Caithness and Sutherland before obtaining further details on their precise extent in the period 1560 to 1689.

Click the thumbnail to see the shapefiles in action

For those interested in making use of these rough shapefiles for their own projects, each file can be found at one of the following links on GitHub:

Aberdeen

Angus and Mearns

Dumfries

Fife

Galloway

Glasgow and Ayr

Lothian and Tweeddale

Merse and Teviotdale

Moray

Perth and Stirling

These shapefiles are far from perfect — a result of Arrowsmith’s inaccuracies and less-than-perfect drawing on my part — but they represent a start in visually understanding the organisation of the Church of Scotland. And while this represents only a snapshot in time — a more fluid picture would require multiple shapefiles of each synod (especially in Argyll) — such an overview of Scotland’s provincial synods, 1560-1689, shows how scanned images can be georeferenced and then opened up in such a way as to make them machine readable. Tools like Mapping the Scottish Reformation will offer more ways to interpret this messy data.

Chris R. Langley

Editor’s Note: Several months after the writing of this post, we uploaded these shapefiles to Wikimedia Commons (July 2021). To find out more, read this blog post.

NEH White Paper: Mapping the Scottish Reformation

What follows is our official “White Paper” for the NEH HCRR Grant. It represents our final public facing progress report for the grant and the culmination of the pilot phase of Mapping the Scottish Reformation.

Objectives, Accomplishments, and Challenges

Mapping the Scottish Reformation (MSR) is a database of the Scottish clergy that allows users to explore and visualize clerical careers between the Reformation Parliament of 1560 and the Revolution in 1689. Built with data from manuscripts held at National Records of Scotland (NRS), this is the first project to comprehensively chart the growth, movement, and networks of the Scottish clergy during and after the Reformation. By extracting data from thousands of pages of ecclesiastical court records, MSR tracks where ministers were educated, how they moved between parishes, their age, their families, their disciplinary history, and more. This early modern data drives a powerful mapping engine that allows users to build their own searches to trace clerical careers over time and space. As such, MSR provides crucial framing for scholarly inquiries into religious beliefs, political conflicts, and institutional change, as well as unprecedented information on critical figures for genealogical research.

The NEH HCRR Foundations grant funded the pilot phase of this project, which focused on the clergy from the Synod of Lothian and Tweeddale, a large and important region including modern day Edinburgh. Our goals for this phase of the project were threefold: first, we wanted to gather the pertinent data on the clergy from this region using the rich and voluminous records of Scotland’s ecclesiastical courts, with an eye to building our pilot user interface once this data was complete. Second, we wanted to identify challenges and inconsistencies in this data that could be used to develop editorial guidelines for the project. Last, we aimed to expand interest in and access to Scottish history in North America, as well foster transatlantic collaboration in the fields of Scottish history, the digital humanities, and genealogy. All of these objectives amounted to what we have thought of as the “proof-of-concept” phase for the project: if we could demonstrate the success of our methods for one part of Scotland, we could then seek to expand our remit to the whole of the country. 

Thanks to the support of the NEH as well as colleagues from a range of fields, we are delighted to report that we not only met but exceeded our goals for this phase of Mapping the Scottish Reformation. In early summer 2020, we completed our dataset for the Synod of Lothian and Tweeddale. Ultimately, we went through over 10,000 pages of manuscript material and gathered data on nearly 900 clerical journeys across 116 Scottish parishes, the “significant events” of 500 ministers (depositions, suspensions), information on more than 200 clerical wives, and the alma maters of over 400 clerics. As part of this work, we formulated internal editorial guidelines about how to categorize and record key parts of the clerical career. 

Thanks to the extension of the NEH grant due to COVID as well as a grant from the Strathmartine Trust, we successfully identified and implemented technologies to record and visualize our data. We entered our data from NRS manuscripts into Wikidata, thanks to the bulk upload tool QuickStatements, and ultimately, we were able to build and launch our pilot website: maps.mappingthescottishreformation.org. Since its launch in December 2020, we have had well over 1,000 users in places ranging from the US to France to Japan.

This site allows users to trace the careers of the Scottish clergy across time and space using 5 different map views: Tenures (where users can learn about the length of ministerial tenures in a given parish), Journeys (where users can trace the movements of the clergy), Education (where users can see the influence of different universities on particular parishes), wives (where users can learn more about clerical spouses, a key but understudied group), and Events (where users can learn about things like rates of depositions and suspensions during moments of political turmoil). We worked closely with the Interactive Content team at the University of Edinburgh in fall 2020 to build this site, and we are already in talks about ways to grow and rearchitect it as we expand our dataset. 

From the outset, we also wanted to prioritize open communication with the public—from historians to digital humanists to genealogists and beyond—about our project and the methods we are using. We often refer to this as “pulling back the curtain,” and to that end, we did a number of presentations to academic and family historian groups over the course of the grant period, including: the Center for Data, Culture, and Society at the University of Edinburgh, the Washington and Lee digital humanities cohort, the Scottish Indexes Society, and the Aberdeen and North East Scotland Family History Society. We also posted regular updates on our blog and on our Twitter account, @mappingscotsref. We also had two meetings with our Advisory Board and were in very regular communication with those members about our ongoing work. 

Our final objective for the pilot phase of MSR was to plan for the future of the project, and this work has begun in earnest. Moving forward, we will expand our dataset to encompass information on clerical careers across all of Scotland. Using feedback from a wide range of parties—from the scholars on our Advisory Board to internationally-based genealogists to the general public engaged with our social media accounts—we have also made tweaks to our current website, and we will be launching version 1.1 in March.

Over the course of this work we have come to appreciate the immense complexity and richness of clerical careers, which were far less straightforward then often presented in the current literature. As such, one of our challenges remains how to capture essential, machine “readable” data while remaining sensitive to messiness and intricacies of clerical experiences. To put this differently, we have been thinking through how to tell stories with our data. A more basic challenge, especially in Fall 2020 during the extension period, was building in time to work on MSR while also teaching during a pandemic. As such, we are very grateful for the support of the NEH.

Project Team and Process 

The co-directors of Mapping the Scottish Reformation are Michelle D. Brock (Washington and Lee University) and Chris R. Langley (Newman University). We have spent the period of the NEH grant gathering and completing the dataset on clerical careers in the Synod of Lothian and Tweeddale using digital images of church court records held at the National Records of Scotland. We initially recorded this data into in Google Sheets, and then uploaded this information to Wikidata, a free and open data storage platform. We then queried the data to generate the JSON files that drive our pilot website. Working with partners at the University of Edinburgh, we built the pilot MSR website, maps.mappingthescottishreformation.org.

Mackenzie Brooks, Digital Humanities Librarian at Washington and Lee University, worked as team member on the project throughout the NEH grant phase. She introduced us to critical digital humanities methods, advised on how to best structure data, and generally provided some much-needed technical insights and encouragement over the course of this work. 

Members of the Mapping the Scottish Reformation Advisory Board include Julian Goodare (University of Edinburgh); Jane Dawson (New College, Edinburgh); Elizabeth Ewan (University of Guelph); Michael Graham (University of Akron); Roger Mason (University of St. Andrews); Tessa Spencer (National Records of Scotland); Paul Youngman (Washington and Lee University). They attended two Advisory meetings and engaged in regular correspondence with the project co-directors to offer invaluable feedback.

Robin Urquhart has been the main project contact at National Records Scotland, working generously to provide us with virtual access to the church court records containing essential data about the Scottish clergy. 

We also collaborated between August 2020 and January 2021 with Stewart Cromar and Hristo Meshinski, both part of the Interactive Content team at the University of Edinburgh, to build the pilot website. Their invaluable work on the project was funded by a grant from the Strathmartine Trust. Ewan McAndrew, Edinburgh’s “Wikimedian in Residence,” has also been an essential collaborator since January 2020, when he introduced us to Wikimedia and the technological side of the project began to take shape. 

Our entire process has been documented on our blog, where we have posted regularly about topics such as the complexities of clerical careers as revealed by the archival documents, how to work with imperfect manuscripts, the benefits of using Wikidata for a project like MSR, and our approach to transatlantic collaboration. This blog has recently been nominated for “best DH blog” the 2020 Digital Humanities Awards.

Findings and Impact 

The pilot phase of Mapping the Scottish Reformation centered on the gathering of manuscript data about clerical careers. It culminated in version 1.0 of our website, which provides quick and reliable information on previously elusive questions at the heart of understanding religious, social, and political change in Scotland. This site represents the first stage of our work, and it is our hope that as we expand the project, MSR can become the first point of reference for scholars seeking information on the Scottish clergy as well a guide to the archival documents that contain this data.

Already, MSR has proven a useful resource for scholars, students, and genealogists. Thanks to the quantitative tools on our site, users can ask a wide range of exciting questions about the men and families at the center of early modern Scottish history. We now know, for example, that the average tenure for a minister appointed in the synod of Lothian and Tweeddale between 1560 and 1689 was 12 years, but that this number was significantly lower—close to 7 years—in the first generation of clergy after the Reformation, when challenges in staffing and training persisted, and similarly brief for those appointed amid the ecclesiastical upheavals of the post-Restoration period. We also know that there were 9 ministers in this region whose careers were longer that 50 years. All of these long-serving men first took up their posts between 1560 and 1620, suggesting that career stability—and perhaps personal longevity—was more attainable for those with appointments made before chaos of war and revolution mid-seventeenth century. Expanding our dataset beyond this single synod region will test whether these tentative patterns hold for other areas of Scotland. 

More generally, our work to date has revealed that despite common assumptions about the rigidity and consistency of the Church of Scotland’s structure, there was in fact tremendous diversity over time and space in the career paths of the post-Reformation clergy. In these volatile years, ecclesiastical policy was hotly debated; parishes were created, dissolved, or united with each other; and ministers’ roles changed, from mere exhorter to preacher of God’s word. As such, our records reflect frequent deviation from the typical path from student to expectant to parish minister, and instead suggest greater informality and flexibility in clerical careers than has previously been appreciated by historians. 

In terms of the archival records, our project has provided more detail on the types of data related to clerical careers found in different records. For example, we confirmed our assumption that kirk session records—the local parish ecclesiastical courts charged with moral discipline—are more likely to contain details on clerical families and the deaths of ministers than the minutes of the presbyteries, and we have integrated some data from these records into our website. This is useful knowledge not only for our own project, but for less-specialized historians or genealogists seeking a “starting point” in exploring the voluminous church court records at NRS. 

Last, MSR has contributed to the growing intersections between the field of history and the digital humanities, particularly by helping historians think about how they can use Wikidata to record and query their data while also making this data open to the public. Our aim has been to be transparent, accessible, and collaborative at every phase of our work in order to reach the largest possible audience and provide a model for similarly structured projects (as we ourselves learn from the work of many others!). 

This is just the beginning, as this NEH HCRR Foundation grant has supported the critical first phase of a much larger project. Eventually, Mapping the Scottish Reformation will allow users to explore and visualize data about clerical careers across the entirety of the country. We hope that its importance to the fields of Scottish history, Reformation studies, and the Digital Humanities, as well as the genealogical community, will continue to grow along with the scale of our work. 

Mikki Brock

Scotland’s Presbyteries, 1560-1689

As we move into Stage 3 of Mapping the Scottish Reformation, we are shifting our attentions beyond the Synod of Lothian and Tweeddale and to the other regions of Scotland.

We have been laying the groundwork for our data collection in Stage 3 by roughly mapping presbyteries and synods across Scotland that were active between 1560 and 1689 and logging them into Wikidata, our repository for structured data. We imported all 65 presbyteries into Wikidata with QuickStatements and using existing Wikidata properties. While the way we have structured this data may change as our work progresses, tracking different synods and presbyteries is essential work as we build our dataset beyond Lothian and Tweeddale.

We originally intended for this activity to be of internal use only — giving us a way to track our progress — but we soon realised it could also offer users a resource to see the administrative structures of the Church of Scotland in a clear (and somewhat interactive) way.

The layers button on the toplight of the map offers the ability to filter different synod regions. Each different colour denotes the presbyteries within a particular synod province (please note that these colours are chosen at random and will change when you reload the page).

Some health warnings: first, the locations of the presbyteries — denoted by the dots — only relate loosely to where a presbytery usually met or was centred. This is to provide a broad idea of where a presbytery was active. The process of mapping the changing boundaries of a synod region or presbytery is something we have tested elsewhere. Secondly, the presbyteries shown are a snapshot in time and, due to the parameters of our project, do not include presbyteries or synods formed after 1689. Equally, presbyteries that were newly created (like Biggar) during our survey period appear alongside the rest of Scotland’s ecclesiastical courts with no note.

Using the workflow we developed in Stage 2, we have also created this simplified Leaflet map by exporting our information from Wikidata. You can read more about the process here.

Click the screen grab to access the map

We have written about the distribution of parishes before on this blog, but seeing provincial assemblies and regional presbyteries mapped, even in this rough way, is of use. This map reflects the geographical distribution of people as well as ecclesiastical power in early modern Scotland. It also represents how far our project must travel to track religious change on a truly national scale.

Chris R. Langley

Visualization Matters: Collaborating to Show Religious Change

In December 2019, Uta Hinrichs, Stefania Forlini, and Bridget Moynihan published a reflective piece entitled ‘In defence of sandcastles’ in Digital Scholarship in the Humanities. In the article, the authors argued that visualizations in humanities projects are not only tools for end users to search through complex data, but are themselves critical research in their own right. Like much humanities scholarship, digital visualizations of historical data are the result of a set of editorial decisions. What makes these decisions interesting is that they must cross different disciplines — bringing together humanities scholars and technical specialists (data modellers, UI/UX designers, web developers etc). In what follows, I want to outline the rationale for some of the decisions behind the visualizations in the Mapping the Scottish Reformation website and to underline how the maps users can play with on the ‘site are the product of deep collaboration between different disciplines.

I want to focus on the Journeys map on the MSR website. This will keep me on track, but it is also because this particular map contained a number of editorial and technical challenges that made us reflect critically on what constituted a clerical career path in early modern Scotland. 

TL;DR: The interdisciplinary discussions around the creation of the Journeys map — a process that was part of a modern humanities project — made us reflect on the experience of being a cleric in early modern Scotland. 

For context, our current Journeys map contains the career paths of 654 ministers and includes 935 separate appointments made across Lothian and Tweeddale between 1560 and the turn of the end of the seventeenth century. At the outset, the historians on the project team had some ideas for what such a map should achieve:

  • To show clerical migration patterns
  • To be able to identify typical clerical careers
  • To understand the distances a minister might cover in his professional life

These questions are all related to longstanding discussions within scholarship of the Reformation and, as such, they represented our own assumptions about what a) users might find useful and b) what we think constitutes a clerical career.

The discussions over how to visualize these journeys forced us to reflect on these assumptions. Our original idea was that parishes in which a minister served would simply be connected by lines that would be clickable by users. These lines would be searchable in some manner. Here is an early example we built on Wikidata:

The example above had four properties:

  1. Places and years: ‘Athelstaneford (1682), Bathgate (1665)
  2. Total moves: ‘2’
  3. Name: Walter Rigg’
  4. Coordinates: [to plot him on the map]

Once the data was presented on our test website, however, it quickly became apparent that these lines would not be particularly useful for visitors to our website (for example, simply drawing lines from one place to the next offered no temporal context), but also that, from a technical perspective, applying filters to huge lines with no differentiation would be very difficult: we had exported our data as strings of text so places and years could not be read by separate filters. Also, the string of text showed a minister’s journey in any order (notice how Rigg’s career in the image above is recorded in reverse sequence). The technical limitations of our data and what this data was saying about a minister’s career forced us to go back to the database and extract a different set of values:

  1. Name
  2. Total moves
  3. Place ranking sequence
  4. Separate year for each move
  5. Year when a minister left parish
  6. Coordinates 

This change to the data model gave us a much more flexible structure for our filters to actually work, but it also allowed us to consider the direction a minister’s career could take. We could now add arrowheads to each move in a cleric’s career and red and green dots denoting the start and end point of (what could be v lengthy) careers :

This was a significant technical and user-friendly fix, but our approach belied one huge assumption we, as historians on the team, had made: our decisions to this point had privileged clerical mobility. In a project about mapping the Scottish Reformation we had fesitishized the idea of movement. 

In our passion to convey to our developer partners that we wanted to explore movement, we had missed a huge chunk of our dataset: those ministers who had only served one parish — those who did not move. Our data modelling allowed for these individuals, but our ideas about visualization gave no place for them. Here is a screengrab of an early prototype showing James French, a minister who had served at only one parish (Penicuik):

Without a conversation with our developer partners who spotted this issue, we may have shipped a build of our website that did not show these individuals (who, btw, represent the largest proportion of ministers). But their diligence also raised the question of how, in a website about mobility, we give due credit to those individuals who dedicated their lives to just one place? Historical assumptions meet visualization questions. 

A technical solution solved the historical question: our developer partners implemented a system whereby those ministers with a parish count of ‘one’ would be listed in their parish. This would allow these critical figures to be displayed and that their service to one parish be specifically parked: in other words, their lack of mobility is marked by a specific UI element and defined by the parish they served:

These kinds of fruitful discussions between developers, UI/UX specialists, and historians can be cyclical. For example, having seen the linestrings and prompted by feedback online, we reflected on the extent to which the point to point linestrings reflected contemporary travel routes, roads, and communication networks. Using a modern route planner, we cobbled together an example for our developers:

Unfortunately, there were both technical and historical problems with this solution. First, this route planner used modern road networks. It will only really be with the fulfilment of projects like this one that would allow scholars to follow contemporary travel routes. Second, the route plug-ins for our maps platform only allowed for one route to be animated at any one time raising questions about how to implement this at scale. Finally, there was an issue of interpretation in animating lines like this: were we, again as historians, privileging mobility in clerical careers with cool UI features that were not actually representative of most clerical careers. This is an example of where editorial discussions between developers, historians, and UI designers forced us to reconsider what we were conveying with these visualizations. 

There are many more examples in our discussions over our first website where expertise in web development, historical knowledge, and UI/UX design brought about significant innovations in how we treated historical material, modelled the data, and built the website. This is just one of them. The process of iterating different versions of our maps forced us to ask questions of what we were seeing: not merely from the perspective of ‘how might a user use this’, but also fundamentally questioning what we thought we knew about the data. Our dev’ partners posed questions to the historians on the team, which in turn forced reflection, which then forced further questions about visualisation.

At their best — and I cannot emphasise this enough — these conversations are utterly energising. While the historians on the team have already been outed for their sheer enthusiasm during these meetings, the level of collaboration implicit in this process is very real and exciting. I remember several project meetings where the collaboration between the team was so fluent that we were flying through new ideas, facing interpretative or technical challenges, devising alternatives, and then watching our developer partners design and revise the website code live on a screen share on Zoom. We were watching the results of our editorial conversations between different disciplines come to life in real time.

This communication process isn’t always easy. But with a degree of candour, an understanding of what domain expertise one area brings to the project, and an ability to openly reflect on the assumptions you bring to the table, projects like ours can build data structures and visualizations that are products of collaboration in their own right, as well as tools for other researchers. 

Chris R. Langley

Introducing the Mapping the Scottish Reformation Website

We are thrilled to share that the Mapping the Scottish Reformation website is now live at maps.mappingthescottishreformation.org. Containing data extracted from over ten thousand pages of manuscript evidence housed in the National Records of Scotland, the website offers researchers powerful new tools to trace the careers of around seven hundred Scottish clergymen and almost five hundred of their wives from Lothian and Tweeddale (the region surrounding Edinburgh) between 1560 and 1689.

The website provides our users access to five maps, all designed to explore different aspects of the clerical life cycle: ‘Tenures’, ‘Journeys’, ‘Education’, ‘Spouses’, and ‘Events’. ‘Tenures’ offers the most in-depth dataset, giving users access to ten aspects of the clerical career and sketching a broad picture of a minister’s professional life. ‘Journeys’ and ‘Education’ show the movements clerics made through their careers, from their place of education to every parish in which they served. ‘Events’ offers an insight into some of the most dramatic aspects of a minister’s career, including details on suspensions and depositions over time, including those during flashpoint such as the Covenanting revolution and the Restoration. Finally, ‘Spouses’ presents data on clergy wives — critical figures in early modern religious, political, and social life — for the first time.

MSR’s search tools allow users to interrogate datapoints from thousands of pages of manuscript material
Observe clerical migration patterns
Search for where clerics were educated and where they moved after graduating, powered by manuscripts, plus data from Hew Scott’s “Fasti Ecclesiae Scoticanae”
For the first time, search a unified database of clerical wives
Search for critical moments in a cleric’s career and observe the impact of political change over time

Each map view offers users a hitherto unavailable set of tools to refine their research questions. Users can look for ministers by name, parish, presbytery region, date, and manuscript reference number. And in all of our map views, users can explore powerful aggregations of data: how many ministers were deposed in a certain time period? What was the typical number of career moves a minister made? What was the busiest year for clerical appointments? How many years did ministers spend in one parish? Until now, these questions would take years of painstaking analysis to complete; Mapping the Scottish Reformation allows users to see these statistics in seconds. What’s more, by showing users full manuscript references, researchers can use the website as a starting point for their research into the rich and complex archival records at the National Records of Scotland.

There are over ten thousand pages of manuscript material powering Mapping the Scottish Reformation, but one of the aims of the project was to ensure it was easy to use. The user interface is designed to be clear and consistent and our glossary explains key terminology relating to the clerical career. Our search tools can be slid to the side of the screen so users can focus on their results and users can select from three different map images — ‘Modern’, ‘Historic’, and ‘Terrain’ — to show their data in different contexts. The ‘Historic’ and ‘Terrain’ maps were provided by the Maps team at the National Library of Scotland and the Historical Maps API. The colour schemes used across the website were developed to ensure search results remain accessible to a wide range of users.

Mapping the Scottish Reformation showcases the potential of open technologies when deployed at scale in large humanities research projects. Data is taken from historical manuscripts and stored in Google Sheets; that data is then entered into Wikidata — a powerful repository for structured data; we query this data using the Wikidata Query Service; these results are exported to JSON format; the maps users see are built in LeafletJS — an open JavaScript mapping library; and the filters we use are designed on the open web. The result is tools and data that are free for other users to deploy in their own projects. We hope Mapping the Scottish Reformation will be a place to generate questions and new research projects, as well as find answers.

This website represents the completion of Stage 1 and 2 of Mapping the Scottish Reformation, but it is only the beginning. The region covered by this version of our website covers the 2,500 square kilometres of the Synod of Lothian and Tweeddale — a region of around 120 parishes. Subsequent stages of Mapping the Scottish Reformation will add data on other Synod regions of the Church of Scotland, including the Synod of Aberdeen, the Synod of Fife, the Synod of Glasgow and Ayr, and the Synod of Perth and Stirling. Critically, the tools we have already developed will accommodate the expansion of the project’s data footprint as we extend the project across Scotland.

This stage of Mapping the Scottish Reformation was funded by the Strathmartine Trust. The data that drives our website was collected during a HCRR grant funded by the National Endowment for the Humanities.

We are immensely grateful for all of the help and support we have received while building this dataset and making the website live, and we are looking forward to the next stages of this exciting project. In the meantime, we welcome questions, comments, and feedback from our users at MappingScotsRef@gmail.com.

You can access the database at maps.mappingthescottishreformation.org.

css.php