One popular way (IBM’s Bleu metric and the related NIST metric) relies on n-gram comparison of an MT-output sentence and a professionally-translated reference. There’s further pointers here: http://www.ics.mq.edu.au/~szwarts/MT-Evaluation.php
Thanks. I surfed the Bleu paper. Correct me if I’m wrong, but there does not seem to have a unique meaning to “translate with accuracy of 90%”.
do you think you could get away by screwing up one word out of ten?
A qualified ‘yes’, the qualification is a big one, though: the goal isn’t to directly use the translations, but to make them available as an intermediate step in human translation.
Yes. My wife is a translator and she already does this. I think most translators already benefit from the pretty good translation software out there. I have not read up on the productivity gains, but according to my wife, while the gains are real, they are not so high as one might think since a lot of the translator’s time is spent on “problem cases” that require research or deep thinking… and the software does not help with this. Though the Web does! I think most translators would trade automated translation for the Web any time. So maybe we could say that the most powerful translation aid is the Web.
But are you sure everyone understands this? I have read somewhere (maybe slashdot) that American soldiers use translation software on the field to chat with civilians. This suggests that American soldiers don’t need interprets and they can just wander along with their software. Hmmm?
matthew smilliesays:
I’m no natural language researcher, but I’d be interested in knowing how they measure accuracy.
One popular way (IBM’s Bleu metric and the related NIST metric) relies on n-gram comparison of an MT-output sentence and a professionally-translated reference. There’s further pointers here: http://www.ics.mq.edu.au/~szwarts/MT-Evaluation.php
do you think you could get away by screwing up one word out of ten?
A qualified ‘yes’, the qualification is a big one, though: the goal isn’t to directly use the translations, but to make them available as an intermediate step in human translation. Good translators are few and far between, and machine translation stands a decent chance of either speeding up there work, or allowing less-fluent people to still produce a reasonable gloss of a document in a reasonable amount of time.
Anecdotally, my French isn’t good enough to translate an academic article from scratch, but given a set of 90%-accurate machine translations I could probably make small corrections and choose between alternative translations quite effectively.
Thanks. I surfed the Bleu paper. Correct me if I’m wrong, but there does not seem to have a unique meaning to “translate with accuracy of 90%”.
Yes. My wife is a translator and she already does this. I think most translators already benefit from the pretty good translation software out there. I have not read up on the productivity gains, but according to my wife, while the gains are real, they are not so high as one might think since a lot of the translator’s time is spent on “problem cases” that require research or deep thinking… and the software does not help with this. Though the Web does! I think most translators would trade automated translation for the Web any time. So maybe we could say that the most powerful translation aid is the Web.
But are you sure everyone understands this? I have read somewhere (maybe slashdot) that American soldiers use translation software on the field to chat with civilians. This suggests that American soldiers don’t need interprets and they can just wander along with their software. Hmmm?
I’m no natural language researcher, but I’d be interested in knowing how they measure accuracy.
One popular way (IBM’s Bleu metric and the related NIST metric) relies on n-gram comparison of an MT-output sentence and a professionally-translated reference. There’s further pointers here: http://www.ics.mq.edu.au/~szwarts/MT-Evaluation.php
do you think you could get away by screwing up one word out of ten?
A qualified ‘yes’, the qualification is a big one, though: the goal isn’t to directly use the translations, but to make them available as an intermediate step in human translation. Good translators are few and far between, and machine translation stands a decent chance of either speeding up there work, or allowing less-fluent people to still produce a reasonable gloss of a document in a reasonable amount of time.
Anecdotally, my French isn’t good enough to translate an academic article from scratch, but given a set of 90%-accurate machine translations I could probably make small corrections and choose between alternative translations quite effectively.