Pathology Report Summarizer Assessment

We are building an automatic pathology report summarization system. We would like to include human judgement from professionals to help us understand and explain the results, and improve the system. In the evaluation, we will schedule a 30-min meeting do go through the details and for the professional to sign a consent form. The professional will expect to receive twenty documents including ten raw reports and ten summaries of these reports. They will be sent in a random order with no correspondence indicated. For each of the document, you will be asked to answer ten questions. We will evaluate the quality of the summaries based on the time she/he spent on reading the report, answering the questions, and the accuracy of the questions.