Skip Navigation
AHRQ--Agency for Healthcare Research and Quality: Advancing Excellence in Health Care
  • Home
  • Search for Research Summaries, Reviews, and Reports

EHC Component

  • EPC Project

Topic Title

  • Assessing the Accuracy of Google Translate To Allow Data Extraction From Trials Published in Non-English Languages

Abstract - Final – Jul. 7, 2011

Assessing the Accuracy of Google Translate for the Purpose of Translating Non-English Language Trials for Data Extraction

Topic Abstract

Background. EPC reports most commonly restrict literature searches to English language publications. The most common reason for the restriction is lack of resources for translation. However, as found in a recent search, 10 percent of recent articles in Medline are published in other languages, and language bias is a concern. Anecdotally, Google™ Translate has been successfully used to translate articles into English, allowing apparently adequate data extraction.

Objectives. After using Google Translate to translate an article, we will compare the accuracy of data extracted from this Google translated article to the accuracy of the data extracted directly from the original language article by an individual who is fluent in that language. We will assess the feasibility and extra resources involved in extracting the non-English language articles via Google Translate.

Specific aims.

  1. Compare discrepancies between data extraction done on original-language articles of trials by a fluent speaker and data extraction done on English-language translations by Google Translate of trials by a researcher who does not know the original article language.
  2. Determine the cause of any discrepancies to determine how likely they are to be due to inaccurate translation, and whether there are any clear patterns within, across, or between languages.
  3. Track and enumerate the time and resources used for article translation and the extra time and resources required for data extraction related to use of translated articles.

Approach. We plan to evaluate 10 randomized controlled trials published in each of 10 languages: Chinese, French, German, Hebrew, Italian, Japanese, Korean, Portuguese, Russian, and Spanish. We aim to select trials on a variety of topics from a variety of journals, with a goal of no more than three trials from a single journal or in a single medical condition. Each article will be translated into English using Google Translate. The original language versions of the articles will then be data extracted by the fluent speakers. The English translations will be extracted by researchers who do not speak the language. Data will be extracted into a simple, generic data extraction form, which will include the study features we deem most important for assessing the study characteristics, methods, and results. We will limit study quality-related features to objective measures. A third researcher will compare pairs of data extraction and will ask each original extractor to confirm any data for which there is a discrepancy. The pairs of data extractors will then meet to review remaining discrepancies to determine whether each discrepancy was due to language differences or other reasons. We will perform kappa analyses for agreement for each item and across all items, and for each language and across all languages. We will categorize and qualitatively assess the causes of discrepancies. We will also track the approximate time and resources used directly for translation of original articles and an assessment of the additional time needed to extract translated articles. We will calculate an estimate of the marginal cost per article for implementing Google Translate to allow data extraction of non-English language articles.