Exercise 13: Annotation and Agreement

1. [Individual] In this activity, you will each label 20 premise/hypothesis pairs with one of three labels. Do not skip anything, but if you are unsure of anything, write down your reasoning for picking each. Please submit a CMU-visible link to your shared Sheet!
2. [Individual] Note roughly how long the activity took you in total, and if you took long on any in particular. Do you notice any particular type of example you took more time on?
3. [Individual] Do you feel that there is something wrong with any of these examples? (Complain about at least one of them)
4. Compare answers and reasoning behind answers with your project group. Surprised at all? Did you find the same things ambiguous or hard to annotate? Discuss at least one specific example of disagreement! (This should probably be your longest answer!)
5. Report and discuss a couple of agreement metrics (no need to discuss all of them!), including relative to your expectations and qualitative observations from Q4 above
6. Optional: What does an LLM annotate? What reasoning does it provide?