Mapping the Scottish Reformation: Website Update v.1.1

We’re very pleased to be able to share the news that the first update of our website is now live at maps.mappingthescottishreformation.org. Thanks to the generous feedback of our users across the world, the updated Mapping the Scottish Reformation site includes a number of improvements, bug fixes, and new features that should make it more useful and easier to use.

While we wanted version 1.1 to be an opportunity to respond to feedback and crush some bugs, we have also added a few features that were not quite ready for version 1.0. Chief among these features was the ability to download search results. Each map now includes a ‘Download results’ button that will export the data on the map into a JSON file.

Our new download function on the Tenures map

In version 1.1, the Education map is more powerful: allowing users to search for approximate graduation dates, and the amount of time between graduation and a minister’s first appointment. To give you a deeper insight into this data, we have added some new aggregations to the bottom of the search bar: showing the years in which most ministers graduated, the average time between graduation and entering the ministry, and even the average distance travelled between a minister’s university and his first parish in kilometres.

The new features on our Education map show the approximate year when a minister graduated and the distance travelled between university and first parish posting

We’ve also added a range of usability enhancements: from making the lines on the Journeys and Education maps thicker, adding manual Zoom controls to each map, to organising lists of names by last name, rather than first name.

It’s the little things: All lists of names are now ordered by last name, rather than first name. And there are manual zoom controls on each map for extra precision.

Version 1.1 of Mapping the Scottish Reformation is the culmination of Stages 1 and 2 of the project, funded by the National Endowment for the Humanities and the Strathmartine Trust, respectively. We have parsed over ten thousand pages of manuscript material, and tracked the careers of 654 ministers and over 400 of their wives in the region of Lothian and Tweeddale. Our website has attracted over sixteen hundred unique visitors since launching in December 2020.

As Stage 3 beckons, here is a full list of the changes we’ve made to v.1.1 of our website:

– Added an introductory message and guide video for first-time visitors, narrated by our friend and fellow Scottish historian, the inimitable Jamie Reid Baxter

– Added a guide video to the ‘About’ page

– Increased weight for lines in the Journeys and Education maps

– Rearranged all lists of names: now organised by last name

– Added a data download function to each map

– All new search options in the Education map

– All-new calculations in the Education tab: graduations, distances, appointment

– Fixed some data errors in the Education map

– Fixed some data errors in the Journeys map

– Default map tile changed from ‘Historic’ to ‘Modern’

– Map attribution has been moved and is no longer obscured by the search bar

– Added zoom control buttons to each map

Thank you for all of your suggestions and support. We are particularly grateful to our Advisory Board, the Interactive Content team at the University of Edinburgh, and Chris Fleet, maps curator at National Library Scotland, for their essential feedback on version 1.0 of the site.

Check our blog and on Twitter for more updates on the project’s development. If there’s anything you think we should be adding to the website, please get in touch at MappingScotsRef@gmail.com.

Mapping Scotland’s Synods, 1560-1689

As we step up our preparations for Stage 3 — venturing out from Lothian and Tweeddale and into the rest of Scotland — we are busy scoping out the extent of the work before us (spoiler: there is a lot). We have already entered Presbytery data to Wikidata. Now we turn our attentions to Scotland’s provincial synods.

Arrowsmith’s ecclesiastical map of Scotland (1825)

Many of our readers will know about Aaron Arrowsmith’s 1825 ecclesiastical map of Scotland. A scanned version is available on the National Library of Scotland website here. I’ve spent a lot of time looking at this map, but there are two problems with it, at least from our perspective: it shows the shape of the nineteenth-century Kirk of Scotland and, unlike so many of the terrific maps on the NLS website, it is not georeferenced.

Fortunately, NLS have adopted the International Image Interoperability Framework (IIIF) on all of its scanned map images. This means that the image can be exported to various different software viewers (ARCGIS, CanvasPanel, Mirador, Recogito, etc). Using the free website georeferencer.com, we imported Arrowsmith’s map and started the georeferencing process.

Georeferencer.com tied Arrowsmith’s map to modern map data

By using georeferencer.com, we were able to overlay Arrowsmith’s representation onto a modern map, pinpoint known places and features, and allow the website to adjust the nineteenth-century map to accord with modern-day cartography.

Georeferencer has an extra trick up its sleeve: the georeferenced map can be taken directly to MapTiler, a service that allows users to create vector shapes on top of georeferenced maps. Translation: we could now trace the boundaries of provincial synods shown on Arrowsmith’s map and use them in modern projects.

Tracking the Aberdeenshire coast on MapTiler

This is fiddly work: you start with a pretty generalised polygon and have to edit it, by hand, to accord with coastal features and the like. There is the added complication that Arrowsmith’s knowledge of certain parts of Scotland was less-than-stellar: thankfully, MapTiler allows users to alter the opacity of each layer so one can see how the modern and historical maps compare. Of course, there are editorial judgements to be made here, especially in areas where Arrowsmith’s map lacked key detail.

We’re using Arrowsmith’s nineteenth-century map here and our project covers the period 1560 to 1689. Unsurprisingly, a number of features had changed between the end of the period covered by our project and the moment of Arrowsmith’s composition. The Synod of Ross, for example, was established in 1707, so was shown on Arrowsmith’s map, but would not have been recognised in the period covered by our project. Similarly, the Synod of Glenelg appears on Arrowsmith’s map, but was not established until 1724. Such challenges meant that it would be remarkably difficult for us to represent the area covered by the Synod of Argyll, in particular, in a single shape file. Then there are changes within the period covered by our project: the Synod of Lothian and Tweeddale includes the region covered by Biggar Presbytery, an area disjoined from the Synod of Glasgow and Ayr in 1644. To capture the complexity of these changes, we would have to create new shape files for each significant revision to the Kirk’s ecclesiastical boundaries. Our shapefiles are no-less an interpretation of provincial synod boundaries than Arrowsmith’s nineteenth century work. Nevertheless, they offer a useful indication of ecclesiastical structures.

Synod boundaries as geoJSON

Once complete, each shape file can be edited further, exported as a GeoJSON (and then possibly converted into other shape file formats) and imported into any mapping platform. Using the workflow we detailed here, we took these datafiles and created a map in Leaflet showing most of Scotland’s provincial synods between 1560 and 1689. For the reasons stated above, we felt it best to not map the Synods of Argyll and Caithness and Sutherland before obtaining further details on their precise extent in the period 1560 to 1689.

Click the thumbnail to see the shapefiles in action

For those interested in making use of these rough shapefiles for their own projects, each file can be found at one of the following links on GitHub:

Aberdeen

Angus and Mearns

Dumfries

Fife

Galloway

Glasgow and Ayr

Lothian and Tweeddale

Merse and Teviotdale

Moray

Perth and Stirling

These shapefiles are far from perfect — a result of Arrowsmith’s inaccuracies and less-than-perfect drawing on my part — but they represent a start in visually understanding the organisation of the Church of Scotland. And while this represents only a snapshot in time — a more fluid picture would require multiple shapefiles of each synod (especially in Argyll) — such an overview of Scotland’s provincial synods, 1560-1689, shows how scanned images can be georeferenced and then opened up in such a way as to make them machine readable. Tools like Mapping the Scottish Reformation will offer more ways to interpret this messy data.

Chris R. Langley

NEH White Paper: Mapping the Scottish Reformation

What follows is our official “White Paper” for the NEH HCRR Grant. It represents our final public facing progress report for the grant and the culmination of the pilot phase of Mapping the Scottish Reformation.

Objectives, Accomplishments, and Challenges

Mapping the Scottish Reformation (MSR) is a database of the Scottish clergy that allows users to explore and visualize clerical careers between the Reformation Parliament of 1560 and the Revolution in 1689. Built with data from manuscripts held at National Records of Scotland (NRS), this is the first project to comprehensively chart the growth, movement, and networks of the Scottish clergy during and after the Reformation. By extracting data from thousands of pages of ecclesiastical court records, MSR tracks where ministers were educated, how they moved between parishes, their age, their families, their disciplinary history, and more. This early modern data drives a powerful mapping engine that allows users to build their own searches to trace clerical careers over time and space. As such, MSR provides crucial framing for scholarly inquiries into religious beliefs, political conflicts, and institutional change, as well as unprecedented information on critical figures for genealogical research.

The NEH HCRR Foundations grant funded the pilot phase of this project, which focused on the clergy from the Synod of Lothian and Tweeddale, a large and important region including modern day Edinburgh. Our goals for this phase of the project were threefold: first, we wanted to gather the pertinent data on the clergy from this region using the rich and voluminous records of Scotland’s ecclesiastical courts, with an eye to building our pilot user interface once this data was complete. Second, we wanted to identify challenges and inconsistencies in this data that could be used to develop editorial guidelines for the project. Last, we aimed to expand interest in and access to Scottish history in North America, as well foster transatlantic collaboration in the fields of Scottish history, the digital humanities, and genealogy. All of these objectives amounted to what we have thought of as the “proof-of-concept” phase for the project: if we could demonstrate the success of our methods for one part of Scotland, we could then seek to expand our remit to the whole of the country. 

Thanks to the support of the NEH as well as colleagues from a range of fields, we are delighted to report that we not only met but exceeded our goals for this phase of Mapping the Scottish Reformation. In early summer 2020, we completed our dataset for the Synod of Lothian and Tweeddale. Ultimately, we went through over 10,000 pages of manuscript material and gathered data on nearly 900 clerical journeys across 116 Scottish parishes, the “significant events” of 500 ministers (depositions, suspensions), information on more than 200 clerical wives, and the alma maters of over 400 clerics. As part of this work, we formulated internal editorial guidelines about how to categorize and record key parts of the clerical career. 

Thanks to the extension of the NEH grant due to COVID as well as a grant from the Strathmartine Trust, we successfully identified and implemented technologies to record and visualize our data. We entered our data from NRS manuscripts into Wikidata, thanks to the bulk upload tool QuickStatements, and ultimately, we were able to build and launch our pilot website: maps.mappingthescottishreformation.org. Since its launch in December 2020, we have had well over 1,000 users in places ranging from the US to France to Japan.

This site allows users to trace the careers of the Scottish clergy across time and space using 5 different map views: Tenures (where users can learn about the length of ministerial tenures in a given parish), Journeys (where users can trace the movements of the clergy), Education (where users can see the influence of different universities on particular parishes), wives (where users can learn more about clerical spouses, a key but understudied group), and Events (where users can learn about things like rates of depositions and suspensions during moments of political turmoil). We worked closely with the Interactive Content team at the University of Edinburgh in fall 2020 to build this site, and we are already in talks about ways to grow and rearchitect it as we expand our dataset. 

From the outset, we also wanted to prioritize open communication with the public—from historians to digital humanists to genealogists and beyond—about our project and the methods we are using. We often refer to this as “pulling back the curtain,” and to that end, we did a number of presentations to academic and family historian groups over the course of the grant period, including: the Center for Data, Culture, and Society at the University of Edinburgh, the Washington and Lee digital humanities cohort, the Scottish Indexes Society, and the Aberdeen and North East Scotland Family History Society. We also posted regular updates on our blog and on our Twitter account, @mappingscotsref. We also had two meetings with our Advisory Board and were in very regular communication with those members about our ongoing work. 

Our final objective for the pilot phase of MSR was to plan for the future of the project, and this work has begun in earnest. Moving forward, we will expand our dataset to encompass information on clerical careers across all of Scotland. Using feedback from a wide range of parties—from the scholars on our Advisory Board to internationally-based genealogists to the general public engaged with our social media accounts—we have also made tweaks to our current website, and we will be launching version 1.1 in March.

Over the course of this work we have come to appreciate the immense complexity and richness of clerical careers, which were far less straightforward then often presented in the current literature. As such, one of our challenges remains how to capture essential, machine “readable” data while remaining sensitive to messiness and intricacies of clerical experiences. To put this differently, we have been thinking through how to tell stories with our data. A more basic challenge, especially in Fall 2020 during the extension period, was building in time to work on MSR while also teaching during a pandemic. As such, we are very grateful for the support of the NEH.

Project Team and Process 

The co-directors of Mapping the Scottish Reformation are Michelle D. Brock (Washington and Lee University) and Chris R. Langley (Newman University). We have spent the period of the NEH grant gathering and completing the dataset on clerical careers in the Synod of Lothian and Tweeddale using digital images of church court records held at the National Records of Scotland. We initially recorded this data into in Google Sheets, and then uploaded this information to Wikidata, a free and open data storage platform. We then queried the data to generate the JSON files that drive our pilot website. Working with partners at the University of Edinburgh, we built the pilot MSR website, maps.mappingthescottishreformation.org.

Mackenzie Brooks, Digital Humanities Librarian at Washington and Lee University, worked as team member on the project throughout the NEH grant phase. She introduced us to critical digital humanities methods, advised on how to best structure data, and generally provided some much-needed technical insights and encouragement over the course of this work. 

Members of the Mapping the Scottish Reformation Advisory Board include Julian Goodare (University of Edinburgh); Jane Dawson (New College, Edinburgh); Elizabeth Ewan (University of Guelph); Michael Graham (University of Akron); Roger Mason (University of St. Andrews); Tessa Spencer (National Records of Scotland); Paul Youngman (Washington and Lee University). They attended two Advisory meetings and engaged in regular correspondence with the project co-directors to offer invaluable feedback.

Robin Urquhart has been the main project contact at National Records Scotland, working generously to provide us with virtual access to the church court records containing essential data about the Scottish clergy. 

We also collaborated between August 2020 and January 2021 with Stewart Cromar and Hristo Meshinski, both part of the Interactive Content team at the University of Edinburgh, to build the pilot website. Their invaluable work on the project was funded by a grant from the Strathmartine Trust. Ewan McAndrew, Edinburgh’s “Wikimedian in Residence,” has also been an essential collaborator since January 2020, when he introduced us to Wikimedia and the technological side of the project began to take shape. 

Our entire process has been documented on our blog, where we have posted regularly about topics such as the complexities of clerical careers as revealed by the archival documents, how to work with imperfect manuscripts, the benefits of using Wikidata for a project like MSR, and our approach to transatlantic collaboration. This blog has recently been nominated for “best DH blog” the 2020 Digital Humanities Awards.

Findings and Impact 

The pilot phase of Mapping the Scottish Reformation centered on the gathering of manuscript data about clerical careers. It culminated in version 1.0 of our website, which provides quick and reliable information on previously elusive questions at the heart of understanding religious, social, and political change in Scotland. This site represents the first stage of our work, and it is our hope that as we expand the project, MSR can become the first point of reference for scholars seeking information on the Scottish clergy as well a guide to the archival documents that contain this data.

Already, MSR has proven a useful resource for scholars, students, and genealogists. Thanks to the quantitative tools on our site, users can ask a wide range of exciting questions about the men and families at the center of early modern Scottish history. We now know, for example, that the average tenure for a minister appointed in the synod of Lothian and Tweeddale between 1560 and 1689 was 12 years, but that this number was significantly lower—close to 7 years—in the first generation of clergy after the Reformation, when challenges in staffing and training persisted, and similarly brief for those appointed amid the ecclesiastical upheavals of the post-Restoration period. We also know that there were 9 ministers in this region whose careers were longer that 50 years. All of these long-serving men first took up their posts between 1560 and 1620, suggesting that career stability—and perhaps personal longevity—was more attainable for those with appointments made before chaos of war and revolution mid-seventeenth century. Expanding our dataset beyond this single synod region will test whether these tentative patterns hold for other areas of Scotland. 

More generally, our work to date has revealed that despite common assumptions about the rigidity and consistency of the Church of Scotland’s structure, there was in fact tremendous diversity over time and space in the career paths of the post-Reformation clergy. In these volatile years, ecclesiastical policy was hotly debated; parishes were created, dissolved, or united with each other; and ministers’ roles changed, from mere exhorter to preacher of God’s word. As such, our records reflect frequent deviation from the typical path from student to expectant to parish minister, and instead suggest greater informality and flexibility in clerical careers than has previously been appreciated by historians. 

In terms of the archival records, our project has provided more detail on the types of data related to clerical careers found in different records. For example, we confirmed our assumption that kirk session records—the local parish ecclesiastical courts charged with moral discipline—are more likely to contain details on clerical families and the deaths of ministers than the minutes of the presbyteries, and we have integrated some data from these records into our website. This is useful knowledge not only for our own project, but for less-specialized historians or genealogists seeking a “starting point” in exploring the voluminous church court records at NRS. 

Last, MSR has contributed to the growing intersections between the field of history and the digital humanities, particularly by helping historians think about how they can use Wikidata to record and query their data while also making this data open to the public. Our aim has been to be transparent, accessible, and collaborative at every phase of our work in order to reach the largest possible audience and provide a model for similarly structured projects (as we ourselves learn from the work of many others!). 

This is just the beginning, as this NEH HCRR Foundation grant has supported the critical first phase of a much larger project. Eventually, Mapping the Scottish Reformation will allow users to explore and visualize data about clerical careers across the entirety of the country. We hope that its importance to the fields of Scottish history, Reformation studies, and the Digital Humanities, as well as the genealogical community, will continue to grow along with the scale of our work. 

Mikki Brock

Scotland’s Presbyteries, 1560-1689

As we move into Stage 3 of Mapping the Scottish Reformation, we are shifting our attentions beyond the Synod of Lothian and Tweeddale and to the other regions of Scotland.

We have been laying the groundwork for our data collection in Stage 3 by roughly mapping presbyteries and synods across Scotland that were active between 1560 and 1689 and logging them into Wikidata, our repository for structured data. We imported all 65 presbyteries into Wikidata with QuickStatements and using existing Wikidata properties. While the way we have structured this data may change as our work progresses, tracking different synods and presbyteries is essential work as we build our dataset beyond Lothian and Tweeddale.

We originally intended for this activity to be of internal use only — giving us a way to track our progress — but we soon realised it could also offer users a resource to see the administrative structures of the Church of Scotland in a clear (and somewhat interactive) way.

The layers button on the toplight of the map offers the ability to filter different synod regions. Each different colour denotes the presbyteries within a particular synod province (please note that these colours are chosen at random and will change when you reload the page).

Some health warnings: first, the locations of the presbyteries — denoted by the dots — only relate loosely to where a presbytery usually met or was centred. This is to provide a broad idea of where a presbytery was active. The process of mapping the changing boundaries of a synod region or presbytery is something we have tested elsewhere. Secondly, the presbyteries shown are a snapshot in time and, due to the parameters of our project, do not include presbyteries or synods formed after 1689. Equally, presbyteries that were newly created (like Biggar) during our survey period appear alongside the rest of Scotland’s ecclesiastical courts with no note.

Using the workflow we developed in Stage 2, we have also created this simplified Leaflet map by exporting our information from Wikidata. You can read more about the process here.

Click the screen grab to access the map

We have written about the distribution of parishes before on this blog, but seeing provincial assemblies and regional presbyteries mapped, even in this rough way, is of use. This map reflects the geographical distribution of people as well as ecclesiastical power in early modern Scotland. It also represents how far our project must travel to track religious change on a truly national scale.

Chris R. Langley

Visualization Matters: Collaborating to Show Religious Change

In December 2019, Uta Hinrichs, Stefania Forlini, and Bridget Moynihan published a reflective piece entitled ‘In defence of sandcastles’ in Digital Scholarship in the Humanities. In the article, the authors argued that visualizations in humanities projects are not only tools for end users to search through complex data, but are themselves critical research in their own right. Like much humanities scholarship, digital visualizations of historical data are the result of a set of editorial decisions. What makes these decisions interesting is that they must cross different disciplines — bringing together humanities scholars and technical specialists (data modellers, UI/UX designers, web developers etc). In what follows, I want to outline the rationale for some of the decisions behind the visualizations in the Mapping the Scottish Reformation website and to underline how the maps users can play with on the ‘site are the product of deep collaboration between different disciplines.

I want to focus on the Journeys map on the MSR website. This will keep me on track, but it is also because this particular map contained a number of editorial and technical challenges that made us reflect critically on what constituted a clerical career path in early modern Scotland. 

TL;DR: The interdisciplinary discussions around the creation of the Journeys map — a process that was part of a modern humanities project — made us reflect on the experience of being a cleric in early modern Scotland. 

For context, our current Journeys map contains the career paths of 654 ministers and includes 935 separate appointments made across Lothian and Tweeddale between 1560 and the turn of the end of the seventeenth century. At the outset, the historians on the project team had some ideas for what such a map should achieve:

  • To show clerical migration patterns
  • To be able to identify typical clerical careers
  • To understand the distances a minister might cover in his professional life

These questions are all related to longstanding discussions within scholarship of the Reformation and, as such, they represented our own assumptions about what a) users might find useful and b) what we think constitutes a clerical career.

The discussions over how to visualize these journeys forced us to reflect on these assumptions. Our original idea was that parishes in which a minister served would simply be connected by lines that would be clickable by users. These lines would be searchable in some manner. Here is an early example we built on Wikidata:

The example above had four properties:

  1. Places and years: ‘Athelstaneford (1682), Bathgate (1665)
  2. Total moves: ‘2’
  3. Name: Walter Rigg’
  4. Coordinates: [to plot him on the map]

Once the data was presented on our test website, however, it quickly became apparent that these lines would not be particularly useful for visitors to our website (for example, simply drawing lines from one place to the next offered no temporal context), but also that, from a technical perspective, applying filters to huge lines with no differentiation would be very difficult: we had exported our data as strings of text so places and years could not be read by separate filters. Also, the string of text showed a minister’s journey in any order (notice how Rigg’s career in the image above is recorded in reverse sequence). The technical limitations of our data and what this data was saying about a minister’s career forced us to go back to the database and extract a different set of values:

  1. Name
  2. Total moves
  3. Place ranking sequence
  4. Separate year for each move
  5. Year when a minister left parish
  6. Coordinates 

This change to the data model gave us a much more flexible structure for our filters to actually work, but it also allowed us to consider the direction a minister’s career could take. We could now add arrowheads to each move in a cleric’s career and red and green dots denoting the start and end point of (what could be v lengthy) careers :

This was a significant technical and user-friendly fix, but our approach belied one huge assumption we, as historians on the team, had made: our decisions to this point had privileged clerical mobility. In a project about mapping the Scottish Reformation we had fesitishized the idea of movement. 

In our passion to convey to our developer partners that we wanted to explore movement, we had missed a huge chunk of our dataset: those ministers who had only served one parish — those who did not move. Our data modelling allowed for these individuals, but our ideas about visualization gave no place for them. Here is a screengrab of an early prototype showing James French, a minister who had served at only one parish (Penicuik):

Without a conversation with our developer partners who spotted this issue, we may have shipped a build of our website that did not show these individuals (who, btw, represent the largest proportion of ministers). But their diligence also raised the question of how, in a website about mobility, we give due credit to those individuals who dedicated their lives to just one place? Historical assumptions meet visualization questions. 

A technical solution solved the historical question: our developer partners implemented a system whereby those ministers with a parish count of ‘one’ would be listed in their parish. This would allow these critical figures to be displayed and that their service to one parish be specifically parked: in other words, their lack of mobility is marked by a specific UI element and defined by the parish they served:

These kinds of fruitful discussions between developers, UI/UX specialists, and historians can be cyclical. For example, having seen the linestrings and prompted by feedback online, we reflected on the extent to which the point to point linestrings reflected contemporary travel routes, roads, and communication networks. Using a modern route planner, we cobbled together an example for our developers:

Unfortunately, there were both technical and historical problems with this solution. First, this route planner used modern road networks. It will only really be with the fulfilment of projects like this one that would allow scholars to follow contemporary travel routes. Second, the route plug-ins for our maps platform only allowed for one route to be animated at any one time raising questions about how to implement this at scale. Finally, there was an issue of interpretation in animating lines like this: were we, again as historians, privileging mobility in clerical careers with cool UI features that were not actually representative of most clerical careers. This is an example of where editorial discussions between developers, historians, and UI designers forced us to reconsider what we were conveying with these visualizations. 

There are many more examples in our discussions over our first website where expertise in web development, historical knowledge, and UI/UX design brought about significant innovations in how we treated historical material, modelled the data, and built the website. This is just one of them. The process of iterating different versions of our maps forced us to ask questions of what we were seeing: not merely from the perspective of ‘how might a user use this’, but also fundamentally questioning what we thought we knew about the data. Our dev’ partners posed questions to the historians on the team, which in turn forced reflection, which then forced further questions about visualisation.

At their best — and I cannot emphasise this enough — these conversations are utterly energising. While the historians on the team have already been outed for their sheer enthusiasm during these meetings, the level of collaboration implicit in this process is very real and exciting. I remember several project meetings where the collaboration between the team was so fluent that we were flying through new ideas, facing interpretative or technical challenges, devising alternatives, and then watching our developer partners design and revise the website code live on a screen share on Zoom. We were watching the results of our editorial conversations between different disciplines come to life in real time.

This communication process isn’t always easy. But with a degree of candour, an understanding of what domain expertise one area brings to the project, and an ability to openly reflect on the assumptions you bring to the table, projects like ours can build data structures and visualizations that are products of collaboration in their own right, as well as tools for other researchers. 

Chris R. Langley

Introducing the Mapping the Scottish Reformation Website

We are thrilled to share that the Mapping the Scottish Reformation website is now live at maps.mappingthescottishreformation.org. Containing data extracted from over ten thousand pages of manuscript evidence housed in the National Records of Scotland, the website offers researchers powerful new tools to trace the careers of around seven hundred Scottish clergymen and almost five hundred of their wives from Lothian and Tweeddale (the region surrounding Edinburgh) between 1560 and 1689.

The website provides our users access to five maps, all designed to explore different aspects of the clerical life cycle: ‘Tenures’, ‘Journeys’, ‘Education’, ‘Spouses’, and ‘Events’. ‘Tenures’ offers the most in-depth dataset, giving users access to ten aspects of the clerical career and sketching a broad picture of a minister’s professional life. ‘Journeys’ and ‘Education’ show the movements clerics made through their careers, from their place of education to every parish in which they served. ‘Events’ offers an insight into some of the most dramatic aspects of a minister’s career, including details on suspensions and depositions over time, including those during flashpoint such as the Covenanting revolution and the Restoration. Finally, ‘Spouses’ presents data on clergy wives — critical figures in early modern religious, political, and social life — for the first time.

MSR’s search tools allow users to interrogate datapoints from thousands of pages of manuscript material
Observe clerical migration patterns
Search for where clerics were educated and where they moved after graduating, powered by manuscripts, plus data from Hew Scott’s “Fasti Ecclesiae Scoticanae”
For the first time, search a unified database of clerical wives
Search for critical moments in a cleric’s career and observe the impact of political change over time

Each map view offers users a hitherto unavailable set of tools to refine their research questions. Users can look for ministers by name, parish, presbytery region, date, and manuscript reference number. And in all of our map views, users can explore powerful aggregations of data: how many ministers were deposed in a certain time period? What was the typical number of career moves a minister made? What was the busiest year for clerical appointments? How many years did ministers spend in one parish? Until now, these questions would take years of painstaking analysis to complete; Mapping the Scottish Reformation allows users to see these statistics in seconds. What’s more, by showing users full manuscript references, researchers can use the website as a starting point for their research into the rich and complex archival records at the National Records of Scotland.

There are over ten thousand pages of manuscript material powering Mapping the Scottish Reformation, but one of the aims of the project was to ensure it was easy to use. The user interface is designed to be clear and consistent and our glossary explains key terminology relating to the clerical career. Our search tools can be slid to the side of the screen so users can focus on their results and users can select from three different map images — ‘Modern’, ‘Historic’, and ‘Terrain’ — to show their data in different contexts. The ‘Historic’ and ‘Terrain’ maps were provided by the Maps team at the National Library of Scotland and the Historical Maps API. The colour schemes used across the website were developed to ensure search results remain accessible to a wide range of users.

Mapping the Scottish Reformation showcases the potential of open technologies when deployed at scale in large humanities research projects. Data is taken from historical manuscripts and stored in Google Sheets; that data is then entered into Wikidata — a powerful repository for structured data; we query this data using the Wikidata Query Service; these results are exported to JSON format; the maps users see are built in LeafletJS — an open JavaScript mapping library; and the filters we use are designed on the open web. The result is tools and data that are free for other users to deploy in their own projects. We hope Mapping the Scottish Reformation will be a place to generate questions and new research projects, as well as find answers.

This website represents the completion of Stage 1 and 2 of Mapping the Scottish Reformation, but it is only the beginning. The region covered by this version of our website covers the 2,500 square kilometres of the Synod of Lothian and Tweeddale — a region of around 120 parishes. Subsequent stages of Mapping the Scottish Reformation will add data on other Synod regions of the Church of Scotland, including the Synod of Aberdeen, the Synod of Fife, the Synod of Glasgow and Ayr, and the Synod of Perth and Stirling. Critically, the tools we have already developed will accommodate the expansion of the project’s data footprint as we extend the project across Scotland.

This stage of Mapping the Scottish Reformation was funded by the Strathmartine Trust. The data that drives our website was collected during a HCRR grant funded by the National Endowment for the Humanities.

We are immensely grateful for all of the help and support we have received while building this dataset and making the website live, and we are looking forward to the next stages of this exciting project. In the meantime, we welcome questions, comments, and feedback from our users at MappingScotsRef@gmail.com.

You can access the database at maps.mappingthescottishreformation.org.

Pulling Back the Curtain on Transatlantic Collaboration

When we began the conversation that led to the creation of Mapping the Scottish Reformation in the spring of 2017, neither of us could have anticipated how quickly MSR would take off, or how incredibly generous the Scottish History and Digital Humanities would be as we navigated the contours of this new and exciting part of our respective careers. One of the things that we remain intensely grateful for is the transparency of others who have embarked on their own digital humanities endeavors. To a person, our fellow scholars have been willing to peel back the curtains of polished finished projects to discuss how the various steps, strategies, and challenges along the way. These “inside looks” have been nothing short formative for our work.

In that spirit, we wanted to offer some insights into our own collaborative process in gathering and visualizing our data, applying for grants, and establishing good communication and a feasible workflow—all while working on two different sides of the Atlantic.

Traditionally, scholars have collaborated from within a single institution or country, eliminating the potential difficulties of different time zones and travel restrictions, not to mention disparate sorts of bureaucratic norms and red tape. Yet with the tools of Zoom, Skype, FaceTime, Google Docs and Sheets, and cloud storage, the time is ripe for transatlantic collaboration, particularly in a field like Scottish History. Scotland’s diaspora, tourist industry, and uniquely fascinating history (we’re biased, we admit!) has meant that scholars and genealogists in North America and well beyond are interested in exploring the nuances of Scotland’s people and past. We think a collaborative project like MSR shows the potential of transatlantic collaboration to invigorate the study of Scottish history—but how exactly have we made this work on a day to day basis?

One of our slightly less awkward live transatlantic collaboration shots from spring 2020

The success of our collaborative process has hinged on three interrelated things: a clear digital workflow, regular communication between ourselves and other interested colleagues, and flexibility. The two technologies that have underpinned this whole project to date—Google Sheets and Wikidata—are stored entirely on the cloud and updated in real time, meaning either of us can access them at any moment and, crucially, at the same time. In the first data gathering phase of this project, we divided up the presbyteries within the Synod of Lothian and Tweeddale and recorded our findings separately but on the same Google Sheet. We communicated constantly about any editorial or content challenges via text, email, or on a Google Doc called, fittingly, “concerns.” Both of us are, by habit, very quick responders to emails and texts, meaning that we could often troubleshoot questions such as “should we count this as the precise date of installation in a parish?” or “how do we record the first post of this schoolmaster turned minister” in real time, so long we were both still in working hours on our respective sides of the pond.

The “concerns” Google Doc also meant that we could make shared notes of oddities or issues in the records and respond to each other along the way. We highly recommend that any collaborators, especially early on in their process, keep some sort of shared running document like this, as it will come in surprisingly handy for grant-writing. You’ll also want to use a shared cloud service for any static or finished documents, such as Dropbox or Box. If we were a bigger project team, a collaborative workflow service such as Trello might have also been very useful.

an example of one of our many Google Sheets pages

Our workflow has also been strengthened by frequent communication about our progress in going through the presbytery records and regular planning for what comes next. We knew, for example, that the stage after data collection would involve entry of that data onto Wikidata and the running of test visualizations. Chris was, from the outset, more interested in (and, I’d argue, more capable of!) learning about and working with Wikidata, Leaflet JS, and a range of mapping technologies. I, on the other hand, was very content to continue augmenting our dataset, which meant that Chris was able to build much of our technological infrastructure while I focused on finishing the remaining presbytery records. The recognition of different strengths and interests, and clear communication about those, is essential for distribution of labor and creating the most efficient workflow, especially when looking ahead to next steps. 

Our process as we have moved through Stage 1 (data-gathering) and into Stage 2 (building the pilot user interface) has been also greatly enhanced through regular communication with other team members, collaborators, and interested colleagues. Mackenzie Brooks, W&L’s Digital Humanities Librarian and a member of the MSR team, has been an indispensable go-to for technology related questions, as has Ewan McAndrew, the University of Edinburgh’s Wikimedia guru. We have frequently turned to W&L’s associate provost and Advisory Board member Paul Youngman for questions and suggestions on funding. Our Advisory Board as a whole has been a constant well of support, insights, and critical questions that have guided the development of our work so far. And of course, all the colleagues from the Scottish history and genealogy communities who have commented on our social media posts, offering their thoughts and queries, have helped us imagine what MSR could become.

This list of people we’ve leaned on is long, but the lesson here is clear: a digital humanities project may have only two co-directors, but the best ones are born of multiple communities. To others embarking on such projects, don’t be afraid to reach out and ask questions, even the most basic ones. In fact, ask them as widely as possible, and share your work-in-progress whenever you can in presentations and in print, rather than waiting to unveil a more polished final product. 

our first joint presentation, in January 2020 at Edinburgh’s Centre for Data, Culture and Society. Other essential venues for sharing our work-in-progress have included the Digital Archives in the Commonwealth Summit, the Ecclesiastical History Society Podcast, the Scottish Indexes Conference, History Scotland, and the International Review of Scottish Studies.

Last, we want to emphasize flexibility as a core aspect of the collaborative process, especially when the project team is international. The UK and US have different calendars for the academic year, varied expectations for the balance between research, teaching, and service/administrative work, and assorted rules for things like research leave and buying out teaching. Because of this, there will invariably be certain times during the year where one of us is too swamped to devote much time to Mapping the Scottish Reformation. Flexibility has been key, with ourselves and each other, especially as we’re both at teaching-focused universities. It is quite common for one of us to say “as a heads up, I won’t be able to get to this for at least a week,” or “sorry, I lost the thread in the abyss of my inbox, can you resend?”. This sort of dialogue that foregrounds flexibility has been crucial, especially when designing work schedules for funding bids. We would certainly recommend that any collaborators beginning a research project as ambitious as MSRhave a conversation about their other research commitments/goals, the typical rhythms of their academic year, and any anticipated challenges.

Over the past three years, we have applied for numerous grants; gone through nearly 10,000 manuscript pages; collected data on 874 clerical posts, roughly 500 significant events (such as depositions and suspensions), 116 parishes, and 7 presbyteries; and begun work on our pilot user interface. Throughout, a clear workflow, good communication, and mutual flexibility has kept things running smoothly and kept us excited about this work. We hope this is some use to the followers of MSR, and we will keep pulling back our own curtain as this project progresses!  

Mikki Brock

Entering Stage Two: Mapping Parishes in Lothian and Tweeddale, 1560-1689

With Stage One of Mapping the Scottish Reformation nearing completion, we now have a large dataset of ministers, detailing their movements and important moments of their careers. We have parsed over 9,000 pages of manuscripts from presbyteries across the region of Lothian and Tweeddale. With these significant task complete, we can now turn our attentions to how we intend to present this data in Stage Two of the project.

Stage Two of Mapping the Scottish Reformation is generously funded by the Strathmartine Trust and will see us explore our user interfaces for the first time. Up to now, we have deposited our data onto Wikidata and used the built-in tools to test our material, to see the locations of gaps, and to create quick mock-up visuals that we think users might find useful. You can read more about our use of Wikidata mapping tools and how we structure our dataset here.

A screen grab of using the mapping tools built into the Wikidata Query Service.

Followers of the project will have seen some of the demos we have been able to quickly put together that visualise the breadth of our data and hint at some of the ways we can put it to work. Critically, Stage Two of Mapping the Scottish Reformation will use even more powerful mapping technologies to create visualisations that load faster, run more smoothly and show even more data. Before we formally embark on this process, we have spent the last couple of weeks dipping our toes into some of these more powerful mapping technologies.

The Wikidata Query Service — and the SPARQL queries we write to ask questions of our data — sits at the heart of our project. It allows our data to be open for other researchers to use (and build on) in the future, but it can also be quickly exported and patched into other programmes/services. The first stage of testing more powerful maps is to take the result of a SPARQL query, add a few modifications to the code, and export it into a TSV (tab-separated values) file.

The Wikidata Query Service allows the results of queries to be exported into various file formats

We exported a SPARQL query that shows all of the parishes in the Synod of Lothian and Tweeddale between 1560 and 1689, as well as a label that showed in which presbytery each parish sat. This has the fewest values of any of our datasets so we thought it would be an easy place to start! The resulting TSV file is effectively a huge spreadsheet: and as much as I like spreadsheets, I think it would be stretch to call it attractive or user friendly. The key thing here is that we have the latitude and longitude data in separate columns and have the key bits of information we want to display to users. Our test file included around 120 lines.

The resulting TSV file is functional, if unappealing

Formatted correctly, a TSV file like this one can be imported into GeoJSON, an open format mapping service that allows users to input geographical data and show them on maps (note: have you noticed our constant use of open source and access services?!! It’s no coincidence!). Users can either add points manually or, critically for us, add geo-referenced locations in bulk. Having uploaded the file, the result is a much more appealing map that includes more attractive and comprehensive icons and the ability to select different mapping layers. We can even add different mapping tiles, using a service like Map Tiler, enabling us to test different backgrounds.

The beauty about GeoJSON is that it transforms that ugly TSV file into something more machine readable. Unfortunately, GeoJSON doesn’t allow you to automatically export your map or embed it into a website like this one. This is where Leaflet.JS comes in.

Leaflet is quietly taking over the world of internet mapping applications, but its huge functionality comes at a significant technical cost: we aren’t in the world of drag and drop or ‘what you see is what you get’ editing anymore. The benefits of a little perseverance, however, are huge.

Leaflet demands an understanding of CSS, or at least an understanding of what to swap into lines of code cribbed from GitHub and when. This process was made infinitely easier by Leaflet’s own tutorials and, in particular, by this superb tutorial on YouTube by Eduonix. The key here is to take the code generated by GeoJSON and to copy it into our HTML file (shown below in Sublime Text markup editor). You can see how the data from GeoJSON is shown just below the various lines of code for headers etc.

After generating our map, a few fairly simply lines of code allows Leaflet to then take the data from GeoJSON and display it, as well as adding a custom mapping layer and popup menus that are, in theory, infinitely customisable. The resulting map can be exported to HTML and embedded into a website. And because the database values were all pasted into Leaflet, at least for the moment, Leaflet doesn’t have to request the info each time the page loads. The result is that the embedded map loads almost immediately.

You can play with this simple demo, showing all of the parishes of Lothian and Tweeddale between 1560 and 1689, below.

Notice that we have made use of the NLS Historical Maps API to plot the points on a historical map. This dynamically adjusts to a different background map depending on how far a user zooms in or out of the map.

If this seems like a tremendous amount of effort to go to in order to embed a map, then I suppose you’re right. What’s important here is that we have demonstrated that the data we manually took from manuscripts within the National Records of Scotland, passed into Wikidata, and then queried using SPARQL and the Wikidata Query Service, can be exported, customised and presented in a way that it as visually friendly as we can make it!

This is just a test, but it reflects the process we will go through during Stage Two of Mapping the Scottish Reformation, with colleagues from our international Advisory Board and our technical friends and colleagues at the University of Edinburgh. Ultimately, this process will allow us to create a number of interactive visualisations that will distill the months we have spent looking at handwritten archival material and make it more accessible. So while we’ve been recording, storing and querying the Scottish Reformation up to now, Stage Two of this project will allow us to start intricately mapping the Scottish Reformation.

Chris R. Langley

Driving Our Data

In the last five months, we have devoted a lot of time to considering how to structure the data that will drive MSR’s visualisations. And while Stage II of MSR will look to develop the user interface of the project, we wanted to take some time to show you ‘under the hood’ of what we have done so far. And what there is still left to do (spoiler: there’s lots)!

The information we collect on ministers from across Lothian and Tweeddale between 1560 and 1689 is collated on Wikidata. Wikidata allows us to link together different aspects of clerical careers and to build search queries to test the data we have collected.

If we’re looking under the hood of MSR, then this stage of the project was about making sure we had an engine. Initially, we took data from volume one of Hew Scott’s Fasti Ecclesiae Scoticanae (scraped as part of our preliminary work in 2016-17) and added it to Wikidata. This consisted of recording the names of ministers, the locations of their parishes, the dates of their tenure, and any references to ministers being deprived or deposed. A typical example would look like this:

Each minister, like any other item on Wikidata, gets a ‘Q’ number and each characteristic (gender, occupation, residence) is given a ‘P’ number. Moreover, each parish is given a set of coordinates, so they can be plotted on a map (you can read about our very earliest work on mapping parishes here).

The beauty of Wikidata is that, with these items being so neatly split into categories, we can run queries on the data using the Wikidata Query service and some basic SPARQL. These queries allow us to ascertain certain information about our ministers or even view maps like the one at the top of this page.

The Wikidata Query service and its table view

The dataset of ministers from the region covered by the Synod of Lothian and Tweeddale consists of over 800 entries, each with four or five characteristics. With this data, we have the very first, simple, visualisation of all of the ministers active in Lothian and Tweeddale. This provides an alternative to genealogists and scholars looking for an easy, accessible, way to visualize Hew Scott’s work from Fasti Ecclesiae Scoticanae.

The next step is to augment Scott’s rudimentary data with manuscript material. This will significantly expand the dataset, adding new information (such as ministers overlooked in the Fasti or the full details on clerical suspensions and depositions), and amending some of Scott’s inconsistencies. MSR is built on manuscripts housed in the National Records of Scotland and, because of the richness of this material, we have devised a very specific way of recording this information, all captured through Wikidata. Let’s look at John Hogg, the controversial minister of West Linton, the Canongate, and South Leith for an example.

An example of collating manuscript information onto a Wikidata entry

Here, we cite manuscript material that provides additional, accurate, information on Hogg’s career. We add details like this for each aspect of clerical career that we find in the manuscripts. Through Wikidata, we can record the repository, the archive accession number and the folio/page reference of relevant material in the manuscript.

NRS, CH2/295/3, f. 39

The image above shows the manuscript page on which details of Thomas Hogg’s career can be gleaned. By adding this material to our database, MSR provides both a finding aid to manuscript resources and also a way into a world of archival material that can often be incredibly challenging for non-specialists to view.

The map at the top of this page is a resource in its own right, but it isn’t static. We are now working to add more manuscript data to the dataset, so it expands and can offer new insights. We have multiple data points relating to over three hundred ministers (and counting) taken from manuscripts across Lothian and Tweeddale to add to Wikidata. Once uploaded, all of this information will result in the most extensive prosopography of Scottish ministers ever attempted.

We hope this view under the hood of MSR shows the extent of our efforts to structure the data of ecclesiastical careers in early modern Scotland. We also hope it shows you the ways in which the manuscript material we are mining will add considerably to our understanding of early modern religion. In the next year, we will devise a more user-friendly and customisable interface, so users from a wide range of backgrounds can explore this data in more detail: tracking clerical careers, observing trends in the data and seeing big religious changes over time. In the meantime, we’d love to hear your feedback on our work so far: it genuinely helps us shape the direction of the project.