Lynette Hirschman and Inderjeet Mani

in The Oxford Handbook of Computational Linguistics

Published in print January 2005 | ISBN: 9780199276349
Published online September 2012 | e-ISBN: 9780191743573 | DOI:

Series: Oxford Handbooks in Linguistics


Show Summary Details


The commercial success of natural language (NL) technology has raised the technical criticality of evaluation. Choices of evaluation methods depend on software life cycles, typically charting four stages — research, advance prototype, operational prototype, and product. At the prototype stage, embedded evaluation can prove helpful. Analysis components can be loose grouped viz., segmentation, tagging, extracting information, and document threading. Output technologies such as text summarization can be evaluated in terms of intrinsic and extrinsic measures, the former checking for quality and informativeness and the latter, for efficiency and acceptability, in some tasks. ‘Post edit measures’ commonly used in machine translation, determine the amount of correction required to obtain a desirable output. Evaluation of interactive systems typically evaluates the system and the user as one team and deploys subject variability, which runs enough subjects to obtain statistical validity hence, incurring substantial costs. Evaluation being a social activity, creates a community for internal technical comparison, via shared evaluation criteria.

Keywords: software life cycles; analysis; prototype; output; measure; interactive

Article.  6105 words. 

Subjects: Computational Linguistics

Full text: subscription required

How to subscribe Recommend to my Librarian

Buy this work at Oxford University Press »

Users without a subscription are not able to see the full content. Please, subscribe or login to access all content.