AI Drafting of Scriptures: A Detailed Analysis of the Results in a Language in Papua New Guinea

AI drafting of scriptures has become a possibility for translation teams through a new feature in the web application Scripture Forge. Leaders in the Bible translation movement want to know whether or not this approach is a viable option, especially for low-resource languages such as Apal in Papua New Guinea. This paper presents the results of a test that was requested by one such leader. This test used the published translations of Genesis, Exodus and most of the New Testament (Mark, Titus and Jude were excluded since they were in the non-central dialect). The original AI drafts of Ruth and Jonah were compared with the final versions of the texts and evaluated using the standard categories of pluses, minuses, and changes. The changes were further analyzed using Scripture Forge’s own warnings about problems. The data clearly supports Scripture Forge’s warnings. The AI tool, in its current form (end of 2024), was not able to produce a draft in Apal that was much more than 50% accurate. From my perspective, that made the tool not worth using. The perspective of a national translator with no formal education, however, was the total opposite of what I had expected. From his perspective, it did not matter how the initial draft of a book was produced. The important fact was that working with the AI draft of a book had an impact on his life.