Now that we have completed our first round of transcription assignments, I wanted to take stock of the progress we have made. Crowdsourcing in digital humanities projects is risky business—often leaving little or nothing in the way of useful results—so we owe it to the field and our supporters to be transparent about our experiment and our results. Thankfully, I am proud to be able to report truly astonishing success! Thanks to the generous contributions from our volunteer transcribers, we have made excellent progress already in this first round.

By way of background, I divided Psalms 1-50 into five sections that are roughly equal in size: Pss 1-15, 16-25, 26-34, 35-40, and 41-50. For each section of each manuscript, we require two independent transcriptions, which makes for 10 assignment slots per manuscript (unless it is lacunose for parts, like the Aleppo Codex [MA]). By tracking hours during our Psalm 22 pilot project, I was able to estimate that each of these assignments should take approximately 20 hours of transcriber time. Factoring in about 5 hours for setup and training, each first assignment should take about 25 hours to complete. The transcribers would have about 1.5 to 2 months to finish the assignments at their own pace.

With project management help from Caleb Punt, we distributed these assignments on our forum optimally based on transcribers’ reported competencies. It was critical to make sure to have enough material from Pss 1-15 to begin processing these psalms immediately, so I built in redundancy by assigning more transcribers than necessary for the assignment slots for Pss 1-15 (and a few others). We also selected a team of experienced scholars to transcribe the very difficult Berlin Codex (MZ)—which presents the Babylonian Masorah and is badly damaged—and planned some redundancy there as well. The response from the transcribers was amazing!

 

Indexing

We were able to complete the full indexing for every one of the manuscripts, identifying which verses are on which pages.

 

Completed Assignments

In all, we distributed a total of 120 first-round assignments to transcribers who had set up accounts in the Virtual Manuscript Room (VMR). 68 (= 57%) of these were completed on time and according to project standards! 50 out of 88 (57%) assignments for Hebrew manuscripts were completed, as were 18 out of 32 (56%) Greek assignments. At the estimated average of 25 hours per assignment (68 x 25 = 1700 volunteer hours), the transcribers completed in total approximately an entire year’s full-time work for an individual researcher over the course of a couple of months in the summer!

Of these completed assignments, only 5 Hebrew and 1 Greek (= 9%) turned out to be redundant. But even these are of value, because reconcilers can select the best two transcriptions to use in cases of redundancy.

 

Manuscript Coverage

For the 11 major manuscripts we chose to include, there were a total of 100 assignment slots required for full coverage, 62 Hebrew and 38 Greek. All of the Hebrew slots were assigned to transcribers, as were 29 out of the 38 required Greek slots. For the Hebrew assignments (which had greater redundancy built in), 45 (73%) distinct slots were filled with completed transcriptions. And the Greek transcribers filled 17 slots (59% of assigned slots; 45% of total slots). This makes for a grand total of 62 completed assignment slots accounting for 62% of the total project need. Not bad for a first round! Significantly, all of the manuscripts except GA had at least two completed assignments for Pss 1-15, which allowed us to begin reconciling transcriptions right away.

 

Quality Control

This progress was indeed remarkable and exceeded even my own optimistic expectations. But it raises the question of quality control. How useful and accurate are the data that were produced? There were several steps we took to ensure accuracy and compliance with project standards:

1. Training - All transcribers underwent training regarding working with manuscripts and electronic transcription (live sessions were recorded for later viewing).
2. Transcription manual - I wrote a detailed transcription manual for reference with instructions defining all project standards.
3. Checking first pages - Transcribers had the option of requesting a more experienced transcriber to check their first transcription page upon completion to provide feedback and guidance for going forward.
4. Forum - I set up a forum category for posting and answering questions regarding the transcription process, project standards, difficult passages, and website bugs.
5. Checks and revisions - When transcribers reported their assignments as complete, I checked their work quickly and requested revisions where project standards were not yet met. After revisions, these assignments were marked as complete.
6. Reconciliation - We have now begun the reconciliation process, which provides another critical quality control measure. A select team of experts have the job of comparing two completed transcriptions using the VMR’s reconciliation tool and adjudicating whenever they disagree. This process is intended to correct any remaining errors in the individuals’ transcriptions and to ensure adherence to project standards. So, unless both transcribers independently made the same mistake, the resulting approved project transcription should reflect the text and layout of the manuscript accurately. These project transcriptions are what will be published on the website and used for the digital edition.
7. Proofreading and editing - In the future, we hope to have specialists proofread the project transcriptions against the manuscripts to identify any remaining errors and inconsistencies in meeting project standards. Additional problems may be flagged in the collation editing process. In these cases, we can edit existing project transcriptions directly.

In a project of this magnitude, there will inevitably be errors and inconsistencies in the data. But these quality control measures have proven both manageable and effective in ensuring a high-quality data set of manuscript transcriptions.

 

Feedback

In order to further evaluate our process, I sent a Google form to the transcribers soliciting feedback. We received 18 responses from transcribers, all of whom had completed their first assignments. These responses indicated a very high level of volunteer satisfaction. 

 

Overall, the transcribers were very satisfied with the tools and process as well. However, we received useful critical feedback in the following areas:

1. Timely feedback - Because we struggled to manage the massive influx of volunteers at first, it took longer than expected to get the assignments distributed and to provide feedback to transcribers (e.g., checking first pages, replying to forum posts). We have brought Caleb Punt on board to help with project mangement going forward, and we expect subsequent rounds to be more manageable.
2. Technical glitches - Some users experienced technical glitches on the website. In rare cases, this required redoing previously completed work. We have invited volunteers with software development skills to join a debugging team to help alleviate these problems in the future.

 

Retention

Of the 68 transcribers who completed their first assignments, 42 (62%) actively volunteered for a second round of assignments. This nicely matches the US national average for volunteer retention of about 65%. It reflects a high level of volunteer satisfaction (evident in the feedback), based on the perceived value of the work as an enjoyable learning experience and as meaningful service to the field. Furthermore, in the two months since the end of the first round of transcription, we have had 13 new volunteers register to do transcriptions. This suggests that our initial push for volunteers did not exhaust the pool of willing volunteers. Furthermore, several repeat transcribers have signed up to do more extensive transcription work for independent study credit, which considerably increases our capacity for further transcription.

 

Next Steps

We have recently released a second round of transcription assignments that aims to fill the remaining gaps in our coverage. Almost all remaining slots are currently assigned, so if the transcribers complete their tasks again this round, then we will essentially have full coverage for all of the originally selected manuscripts.

Our continued volunteer capacity has simultaneously allowed us to expand the corpus of manuscripts included. With help from Vince Beiler, we were recently able to add 8 new model Masoretic codices from the Firkovich collections in St. Petersburg that date from around the 10th-11th centuries. These are very substantial manuscripts that are just as old as the Aleppo and Leningrad codices, but have never before been transcribed or thoroughly studied for purposes of a critical edition. Ben Outhwaite and Kim Phillips have also helped us identify a selection of the most important fragments from the Cairo Genizah that we hope to include in the future. Thus, this crowdsourcing experiment has not only helped us achieve our project goals more quickly and efficiently, but also opened up new possibilities to make the edition even more robust.

In parallel with the ongoing second round of transcriptions, we have already started reconciling transcriptions for Pss 1-15 with the help of a select team of skilled and experienced volunteer transcribers. And I plan to begin editing a preliminary edition of Pss 1-15 in the coming weeks based on these transcriptions. Troy Griffitts and I will report on our progress and tools at an SBL Digital Humanities session on November 25, so please do join us for further updates and reflections.

 

Of course, none of this would have been possible with the support of our amazing volunteers, who have made this crowdsourcing experiment such a success. So I want to conclude this post with a big “Thank you!” and "Congratulations!" to all of you!

As our project transcription is now going full-speed ahead, I wanted to take this first project blog post as an opportunity to celebrate our transcribers who are at this moment hard at work transcribing the most important Hebrew and Greek manuscripts of Psalms 1-50. As of today, we have had 181 responses to our Google form, of which 124 have joined the project and have started working on transcription assignments! Brent and I were both overwhelmed at the response to our call for volunteer transcribers! Brent was the more cynical of the two of us about crowdsourcing, but he told me that this response "has renewed my faith in humanity." I suspected all along that others out there would find biblical languages and manuscripts as interesting as we do and would want to be involved in a project like this, but even I was blown away by the response of highly-skilled individuals willing to volunteer their time to help make these manuscripts publicly available.

Who are these transcribers?

Almost all of our active transcribers have completed degrees related to the texts and/or languages of the Hebrew and Greek Bibles. Over 70% have relevant Master's degrees, and fully 20% have Doctorates in the field.

Respondents self-reported relatively high levels of competency in Hebrew and Greek language, with lower levels of competency in Aramaic, Latin, and Syriac. We have allocated assignments of Hebrew and Greek manuscripts according to self-reported linguistic competencies, degree specialties, and research interests.

The transcribers represent more than 70 different institutional affiliations with universities, colleges, seminaries, Bible translation organizations, and various religious ministries in 21 countries from all six regularly inhabitted continents.

What are the transcribers doing?

Transcribers on the CEHP team are using our state-of-the-art Virtual Manuscript Room digital workspace to encode the texts of each of the most important manuscripts of Psalms 1-50 in a standard, machine-readable text format. Transcribers start with high-resolution digital images of the manuscripts and a digital base text from a standard edition and then modify the text to match the manuscript. They also use special markup code to indicate special features like scribal corrections and layout. We have divided up Psalms 1-50 into five roughly equal parts and assigned at least two transcribers to each of these parts for each manuscript. The first round of assignments is scheduled to be completed by the end of July, and the transcribers are hard at work as we speak. Every transcriber who completes at least one assignment on time according to project standards will be recognized as a project transcriber in the final edition in gratitude for their service to the field.

How will we use the transcriptions?

When two independent transcriptions are complete for a section of a manuscript, we then have an expert compare them and reconcile any differences to generate approved project transcriptions. This allows us to open up the transcription process to many contributors, while still maintaining strict standards of data quality.

The reconciled and approved project transcriptions then form the basis of our manuscript evidence for the edition. We present these alongside images (where permissible) of the manuscripts included in our system. When all of the transcriptions for a passage are complete, reconciled, and approved, then we also compare all of the transcriptions, using a semi-automated Collation Editor to line up all of the texts and group readings that agree with each other. From this, we can automatically generate the apparatus for the digital edition, which is fully integreated with and linked to the transcriptions of the individual manuscripts.

Conclusion

I would like to reiterate just how grateful Brent and I are at the outpouring of support from our transcribers who are volunteering their time to studying these fascinating manuscripts and making their texts available to the world. A project of this scope and magnitude would be impossible without the generosity of others who share our interests and passions, and we are honored to have assembled such a top-notch team. My goal is not just to produce the first digital and eclectic edition of Psalms 1-50, but also to raise up a generation of scholars who are not scared of digital tools, but are rather well-equipped to use state-of-the-art technology to produce the kinds of editions that the world needs for the 21st century. A big thanks to all of the transcribers who are joining us on this journey! I hope it is an enjoyable learning experience for all.