Skip to content

Assertions

The table below shows the 102 assertions usable by AssertionAnnotator and ClassificationPipeline. Each assertion has a corresponding SetFit classifier on HuggingFace. The labeled training dataset is also available on HuggingFace at StanfordSCALE/assertions_llm_annotated_talkmoves.

Model performance varies

All available assertions are listed here, but some models have poor performance, poor inter-rater reliability, or low support and therefore unreliable evaluation metrics. Consider assertion performance before using annotators to make claims about your data.

Assertion Test set support Krippendorff's alpha Test F1
View is_a_single_word on Hugging Face is_a_single_word 989 (12.0%) 0.940 0.969
View contains_a_student_s_name on Hugging Face contains_a_student_s_name 685 (8.3%) 0.957 0.966
View contains_a_number on Hugging Face contains_a_number 2295 (27.9%) 0.904 0.952
View contains_negation on Hugging Face contains_negation 696 (8.4%) 0.903 0.947
View contains_math_terms on Hugging Face contains_math_terms 3333 (40.5%) 0.835 0.913
View contains_a_question on Hugging Face contains_a_question 1987 (24.1%) 0.879 0.911
View is_a_declarative_statement on Hugging Face is_a_declarative_statement 3589 (43.6%) 0.801 0.899
View contains_fraction_terms on Hugging Face contains_fraction_terms 520 (6.3%) 0.912 0.899
View is_very_short on Hugging Face is_very_short 3575 (43.4%) 0.634 0.872
View contains_a_yes_no_question on Hugging Face contains_a_yes_no_question 730 (8.9%) 0.852 0.862
View contains_a_politeness_marker on Hugging Face contains_a_politeness_marker 304 (3.7%) 0.846 0.861
View contains_an_apology on Hugging Face contains_an_apology 31 (0.4%) 0.874 0.847
View contains_a_real_world_reference_pertaining_to_a_topic_outside_of_the_classroom_setting on Hugging Face contains_a_real_world_reference_pertaining_to_a_topic_outside_of_the_classroom_setting 423 (5.1%) 0.845 0.845
View contains_gratitude on Hugging Face contains_gratitude 152 (1.8%) 0.654 0.838
View invites_a_student_to_speak_next on Hugging Face invites_a_student_to_speak_next 1837 (22.3%) 0.820 0.837
View invites_participation on Hugging Face invites_participation 2822 (34.3%) 0.755 0.836
View contains_a_counting_sequence on Hugging Face contains_a_counting_sequence 108 (1.3%) 0.786 0.809
View contains_collaborative_language on Hugging Face contains_collaborative_language 1424 (17.3%) 0.670 0.806
View asks_one_or_more_students_whether_they_heard_a_prior_contribution on Hugging Face asks_one_or_more_students_whether_they_heard_a_prior_contribution 7 (0.1%) 0.909 0.800
View invites_student_thinking on Hugging Face invites_student_thinking 1584 (19.2%) 0.774 0.791
View addresses_the_whole_class on Hugging Face addresses_the_whole_class 4771 (57.9%) 0.713 0.789
View is_a_one_word_agreement on Hugging Face is_a_one_word_agreement 220 (2.7%) 0.674 0.780
View invites_reading_aloud on Hugging Face invites_reading_aloud 32 (0.4%) 0.839 0.780
View contains_an_instruction on Hugging Face contains_an_instruction 1653 (20.1%) 0.752 0.772
View contains_measurement_terms on Hugging Face contains_measurement_terms 578 (7.0%) 0.683 0.765
View contains_praise on Hugging Face contains_praise 346 (4.2%) 0.786 0.762
View references_classroom_materials on Hugging Face references_classroom_materials 543 (6.6%) 0.703 0.747
View contains_a_time_marker on Hugging Face contains_a_time_marker 621 (7.5%) 0.642 0.734
View references_logistics on Hugging Face references_logistics 997 (12.1%) 0.729 0.731
View contains_emotion on Hugging Face contains_emotion 413 (5.0%) 0.676 0.728
View contains_comparison_terms on Hugging Face contains_comparison_terms 580 (7.0%) 0.659 0.727
View asks_one_or_more_students_for_their_reasoning on Hugging Face asks_one_or_more_students_for_their_reasoning 285 (3.5%) 0.713 0.723
View asks_a_question_to_the_whole_class on Hugging Face asks_a_question_to_the_whole_class 951 (11.5%) 0.807 0.718
View contains_affirmation on Hugging Face contains_affirmation 735 (8.9%) 0.684 0.709
View asks_one_or_more_students_for_an_explanation on Hugging Face asks_one_or_more_students_for_an_explanation 334 (4.1%) 0.756 0.709
View contains_a_personal_stance on Hugging Face contains_a_personal_stance 614 (7.5%) 0.719 0.700
View asks_one_or_more_students_for_the_result_of_a_math_computation on Hugging Face asks_one_or_more_students_for_the_result_of_a_math_computation 253 (3.1%) 0.731 0.688
View invites_writing_on_the_board on Hugging Face invites_writing_on_the_board 40 (0.5%) 0.744 0.676
View contains_an_explanation_of_a_mathematical_concept on Hugging Face contains_an_explanation_of_a_mathematical_concept 1042 (12.7%) 0.732 0.669
View references_a_procedure_for_completing_the_current_task on Hugging Face references_a_procedure_for_completing_the_current_task 1831 (22.2%) 0.583 0.665
View contains_informal_language on Hugging Face contains_informal_language 2090 (25.4%) 0.452 0.656
View manages_classroom_behavior on Hugging Face manages_classroom_behavior 930 (11.3%) 0.611 0.652
View is_an_evaluative_statement on Hugging Face is_an_evaluative_statement 417 (5.1%) 0.678 0.641
View asks_one_or_more_students_for_the_purpose_of_a_mathematical_step on Hugging Face asks_one_or_more_students_for_the_purpose_of_a_mathematical_step 61 (0.7%) 0.732 0.639
View contains_clarification on Hugging Face contains_clarification 1890 (22.9%) 0.508 0.635
View asks_one_or_more_students_for_an_explanation_of_how_a_method_works on Hugging Face asks_one_or_more_students_for_an_explanation_of_how_a_method_works 176 (2.1%) 0.728 0.630
View asks_one_or_more_students_whether_they_agree_with_a_prior_student_contribution on Hugging Face asks_one_or_more_students_whether_they_agree_with_a_prior_student_contribution 85 (1.0%) 0.667 0.627
View contains_agreement on Hugging Face contains_agreement 610 (7.4%) 0.657 0.624
View asks_one_or_more_students_whether_they_understand_something on Hugging Face asks_one_or_more_students_whether_they_understand_something 96 (1.2%) 0.632 0.623
View contains_a_hypothetical_scenario on Hugging Face contains_a_hypothetical_scenario 347 (4.2%) 0.623 0.618
View asks_one_or_more_students_whether_they_are_sure_about_an_answer on Hugging Face asks_one_or_more_students_whether_they_are_sure_about_an_answer 43 (0.5%) 0.613 0.608
View contains_an_acknowledgement on Hugging Face contains_an_acknowledgement 1134 (13.8%) 0.700 0.608
View contains_math_computation on Hugging Face contains_math_computation 301 (3.7%) 0.564 0.607
View references_a_prior_lesson on Hugging Face references_a_prior_lesson 82 (1.0%) 0.706 0.593
View references_student_work on Hugging Face references_student_work 675 (8.2%) 0.665 0.589
View states_that_the_speaker_is_uncertain on Hugging Face states_that_the_speaker_is_uncertain 209 (2.5%) 0.689 0.587
View asks_one_or_more_students_whether_they_agree on Hugging Face asks_one_or_more_students_whether_they_agree 134 (1.6%) 0.705 0.587
View asks_a_non_mathematical_question on Hugging Face asks_a_non_mathematical_question 242 (2.9%) 0.744 0.560
View proposes_a_problem_solving_strategy on Hugging Face proposes_a_problem_solving_strategy 939 (11.4%) 0.569 0.559
View asks_one_or_more_students_for_the_meaning_of_a_mathematical_symbol on Hugging Face asks_one_or_more_students_for_the_meaning_of_a_mathematical_symbol 39 (0.5%) 0.650 0.553
View narrates_steps_being_taken_by_the_speaker on Hugging Face narrates_steps_being_taken_by_the_speaker 524 (6.4%) 0.499 0.549
View asks_one_or_more_students_for_the_reasoning_behind_a_prior_student_contribution on Hugging Face asks_one_or_more_students_for_the_reasoning_behind_a_prior_student_contribution 115 (1.4%) 0.780 0.545
View repeats_a_number_from_a_prior_student_contribution on Hugging Face repeats_a_number_from_a_prior_student_contribution 41 (0.5%) 0.804 0.545
View evaluates_a_student_response on Hugging Face evaluates_a_student_response 1016 (12.3%) 0.701 0.539
View responds_to_a_student_contribution on Hugging Face responds_to_a_student_contribution 2205 (26.8%) 0.689 0.536
View asks_one_or_more_students_for_an_evaluation_of_a_prior_student_contribution on Hugging Face asks_one_or_more_students_for_an_evaluation_of_a_prior_student_contribution 120 (1.5%) 0.648 0.532
View reads_a_text_aloud on Hugging Face reads_a_text_aloud 131 (1.6%) 0.566 0.532
View invites_writing_on_their_own_paper on Hugging Face invites_writing_on_their_own_paper 215 (2.6%) 0.628 0.532
View grants_permission on Hugging Face grants_permission 154 (1.9%) 0.739 0.525
View asks_one_or_more_students_for_the_next_step_in_a_procedure on Hugging Face asks_one_or_more_students_for_the_next_step_in_a_procedure 171 (2.1%) 0.607 0.523
View asks_one_or_more_students_for_clarification on Hugging Face asks_one_or_more_students_for_clarification 291 (3.5%) 0.548 0.519
View references_student_behavior on Hugging Face references_student_behavior 1340 (16.3%) 0.206 0.512
View contains_a_greeting on Hugging Face contains_a_greeting 22 (0.3%) 0.841 0.500
View asks_one_or_more_students_for_the_definition_of_a_math_term on Hugging Face asks_one_or_more_students_for_the_definition_of_a_math_term 26 (0.3%) 0.576 0.474
View repeats_a_single_word_from_a_prior_student_contribution on Hugging Face repeats_a_single_word_from_a_prior_student_contribution 296 (3.6%) 0.602 0.465
View contains_encouragement on Hugging Face contains_encouragement 331 (4.0%) 0.462 0.461
View contains_the_final_answer_to_a_math_problem on Hugging Face contains_the_final_answer_to_a_math_problem 399 (4.8%) 0.644 0.455
View asks_students_whether_they_got_a_different_answer_than_a_prior_contribution on Hugging Face asks_students_whether_they_got_a_different_answer_than_a_prior_contribution 18 (0.2%) 0.373 0.444
View transitions_from_one_in_class_activity_to_another on Hugging Face transitions_from_one_in_class_activity_to_another 516 (6.3%) 0.644 0.444
View asks_one_or_more_students_for_a_connection_between_two_mathematical_representations on Hugging Face asks_one_or_more_students_for_a_connection_between_two_mathematical_representations 109 (1.3%) 0.377 0.442
View agrees_with_one_or_more_students_answer on Hugging Face agrees_with_one_or_more_students_answer 609 (7.4%) 0.745 0.421
View references_an_activity_from_earlier_in_the_session on Hugging Face references_an_activity_from_earlier_in_the_session 231 (2.8%) 0.533 0.402
View contains_the_answer_to_a_question on Hugging Face contains_the_answer_to_a_question 860 (10.4%) 0.636 0.401
View disagrees_with_a_prior_contribution on Hugging Face disagrees_with_a_prior_contribution 213 (2.6%) 0.675 0.395
View asks_one_or_more_students_whether_they_are_finished_with_a_task on Hugging Face asks_one_or_more_students_whether_they_are_finished_with_a_task 26 (0.3%) 0.481 0.391
View contains_a_rhetorical_question on Hugging Face contains_a_rhetorical_question 554 (6.7%) 0.543 0.380
View repeats_a_problem on Hugging Face repeats_a_problem 338 (4.1%) 0.442 0.356
View summarizes_a_portion_of_the_current_lesson on Hugging Face summarizes_a_portion_of_the_current_lesson 393 (4.8%) 0.419 0.355
View invites_an_addition_to_a_prior_student_contribution on Hugging Face invites_an_addition_to_a_prior_student_contribution 271 (3.3%) 0.430 0.351
View repeats_a_student_answer on Hugging Face repeats_a_student_answer 398 (4.8%) 0.713 0.329
View repeats_prior_speech on Hugging Face repeats_prior_speech 994 (12.1%) 0.628 0.324
View repeats_a_phrase_from_a_prior_student_contribution_without_adding_new_words on Hugging Face repeats_a_phrase_from_a_prior_student_contribution_without_adding_new_words 291 (3.5%) 0.513 0.318
View states_the_goals_of_the_lesson on Hugging Face states_the_goals_of_the_lesson 133 (1.6%) 0.579 0.317
View is_unintelligible_in_context on Hugging Face is_unintelligible_in_context 245 (3.0%) 0.527 0.267
View is_off_topic on Hugging Face is_off_topic 116 (1.4%) 0.606 0.266
View contains_self_correction on Hugging Face contains_self_correction 197 (2.4%) 0.570 0.249
View paraphrases_a_prior_student_contribution on Hugging Face paraphrases_a_prior_student_contribution 164 (2.0%) 0.528 0.235
View is_a_verbatim_repetition_of_a_prior_student_contribution on Hugging Face is_a_verbatim_repetition_of_a_prior_student_contribution 225 (2.7%) 0.541 0.233
View contains_humor on Hugging Face contains_humor 121 (1.5%) 0.674 0.212
View repeats_a_prior_student_contribution_and_adds_new_words on Hugging Face repeats_a_prior_student_contribution_and_adds_new_words 261 (3.2%) 0.559 0.204
View corrects_a_prior_student_contribution on Hugging Face corrects_a_prior_student_contribution 171 (2.1%) 0.504 0.198
View addresses_another_teacher on Hugging Face addresses_another_teacher 44 (0.5%) 0.647 0.176

Krippendorff's alpha is the agreement between the two LLM annotators that labelled the utterances in the dataset. Test F1 is the positive-class F1 of the published classifier on the held-out test set.