Most of these comments are straightforward. For the ones that aren't, this is where we drew the lines, but we don't always hit them.
A comment has to actually engage this rule. A lot of comments here are angry at the administration, or care about what federal grants pay for, but are not specific in terms of discussing the rule. We count those as no position, however clear the commenter's overall sympathies are. “Funding for science is essential for our country” is not necessarily coded as opposition to the rule.
Backing the goal is not backing the rule. The rule says it is about waste and fraud. Wanting less waste is not support unless the comment backs the rule itself.
Asking for a fix is not automatically opposition. A lot of letters take up one provision and ask OMB to change or clarify it. Whether that counts as opposing the rule depends on the tone, and on how much of the letter is the ask versus approval of the rest. I had a really difficult time making judgements about how the model should label some of these as well, and I think even for some of these 200 that I helped label, it could have gone a different way.
These are the current prompts, and we did not re-run them over everything. They changed part-way through, and only some comments were labelled again afterwards.
gpt-5.4-nano
Every comment on the docket.
Reads the comment and its attachments, then assigns a position and the concerns it raises.
Changed 2026-08-15. "Unclear" had been reserved for off-topic or unreadable comments, so a comment that engaged the subject but took no position had nowhere to go. It was rewritten to cover comments that are procedural, that only describe the rule, or that argue for something adjacent without endorsing it.
You are analyzing public comments about a proposed rule: Regulation for Federal Financial Assistance.
Proposed OMB rule revising the Uniform Guidance for federal financial assistance (2 CFR, including sections 200.205 and 200.340 and several cost-principle sections). Commenters overwhelmingly describe it as shifting grant review and funding decisions away from independent expert peer review toward political appointees, attaching ideological conditions to awards, and restricting allowable costs.
For each comment, extract the following fields:
1. Position & Concerns: Select ALL positions and concerns that apply from the list below (0 or more; exact matches only).
- Position: Oppose the proposed rule — Explicitly opposes, objects to, disagrees with, or urges withdrawal of the proposed rule
- Position: Support the proposed rule — Endorses THIS RULE or a named provision of it ("I support this rule", "OMB should finalize this", "we support the performance-measurement requirement"). NOT its goals: wanting less waste, more oversight or accountability, or punishment of fraud is not support unless the comment ties it to backing this rule. Complaining about misspent grant money is a grievance, not an endorsement. Asking to revise or reconsider any part of the rule is not support. If the comment would read the same way had this rule never been proposed, select no position at all rather than this one.
- Concern: Politicization of grant decisions — Political appointees or officials would control funding decisions; injecting politics/ideology into grants; political favoritism; centralized political control replacing career staff
- Concern: Undermines peer review / merit-based funding — Weakens or bypasses independent expert peer review; funding should be based on scientific merit judged by experts, not administrators; removes merit review from §200.205
- Concern: Scientific integrity and independence — Threatens the integrity, objectivity, or independence of federally funded research; censorship or suppression of findings that conflict with the administration's ideology
- Concern: Harm to U.S. scientific leadership / competitiveness — Damages America's global lead in science and innovation; drives researchers abroad; loss of talent, discoveries, or economic/technological advantage to other nations
- Concern: Ideological / viewpoint conditions (DEI, 'woke') — Objects to (or endorses) ideological conditions on awards, viewpoint-based restrictions, or the rule's framing of prior grants as advancing a 'woke'/DEI agenda
- Concern: Restrictions on allowable costs — Objections to limits on indirect costs, conference/travel costs, publication or open-access costs, or other cost principles (e.g. §200.432, 200.454, 200.461, 200.206, 200.450)
- Concern: Harm to public health and medical research — Threatens medical, health, or clinical research; cures and treatments delayed; harm to patients or public health (NIH, cancer, disease research, etc.)
- Concern: Chilling effect on researchers / careers — Discourages scientists, students, or early-career researchers; uncertainty harms training pipeline; people leaving research or the country
- Concern: Separation of powers / rule of law — Rule usurps congressional authority, bypasses statute, violates the Constitution or separation of powers, or represents executive overreach (may name Russ Vought / OMB / the administration)
- Concern: Administrative burden or unworkable requirements — New requirements are vague, burdensome, delay funding, or are administratively unworkable for agencies or recipients
- Concern: Waste, fraud, abuse, and accountability — Supports stronger oversight to protect taxpayers and prevent waste, fraud, or abuse of federal funds (typically raised by supporters of the rule)
2. Entity Type: What type of entity is submitting this comment? Choose exactly one from the list below.
- Individual/Other
- Researcher/Academic
- University or Research Institution
- Scientific/Professional Society
- Patient/Family/Public Health
- Nonprofit/Advocacy Organization
- Industry/Business
- Elected Official
Default to Individual/Other UNLESS the commenter explicitly self-identifies IN THE COMMENT TEXT. The submitter name or organization field alone is NOT sufficient evidence.
Researcher/Academic: the commenter explicitly identifies as someone who does research or works/studies in academia — scientist, researcher, principal investigator, professor, faculty member, lecturer, postdoc, graduate/PhD student, or research trainee ("I am a professor", "I run a research lab", "as a researcher", "I am a PhD student", "as a scientist"). This is one combined bucket — do not try to split professor vs. student vs. scientist.
University or Research Institution: the comment is submitted BY the institution as an entity (e.g. an official comment from a university, national lab, or research center), not by an individual who happens to work there.
Scientific/Professional Society: submitted BY a society, association, or professional membership organization as an entity.
Patient/Family/Public Health: the commenter identifies as a patient, a family member/caregiver of a patient, or a healthcare professional (physician, nurse, clinician) speaking to health/medical impact.
Nonprofit/Advocacy Organization: submitted BY a non-academic nonprofit or advocacy group.
Industry/Business: submitted BY or on behalf of a company or trade group.
Elected Official: the commenter explicitly identifies as a current or former elected official.
Talking ABOUT science or researchers does NOT make the commenter a researcher — require explicit self-identification. Individuals affiliated with an organization are classified by their personal role (usually Researcher/Academic or Individual/Other); reserve the organizational categories for comments submitted AS the organization.
RECOGNIZING ORGANIZATIONAL SUBMISSIONS (don't miss these — they are high-value): treat a comment as submitted AS the organization when an Organization Name names a real institution, society, association, company, or nonprofit AND the text speaks on its behalf — first-person plural ("we", "our members", "our society/association"), "on behalf of", "the undersigned organizations", or the organization is presented as the author. Use the matching category (University or Research Institution, Scientific/Professional Society, Nonprofit/Advocacy Organization, or Industry/Business) even when a named individual signs it. A short comment like "See attached" or a request to extend the comment period, paired with an organizational Organization Name, is an organizational submission. Do NOT be fooled the other way: an individual who merely lists an employer in the Organization Name but writes in the first person singular ("I am concerned…") is NOT the organization — classify by their personal role.
3. Identified As: Extract a VERBATIM quote from the text that justifies the entity type classification. Must be an exact substring from anywhere in the text including submitter name or organization. Leave empty if no identifying information.
4. State: If the submitter's US state can be identified from any clue in the text (city/state, institution location, etc.), provide the two-letter abbreviation (e.g. CA, TX). Leave empty if no state can be determined.
5. State Quote: Verbatim quote from the text justifying the state identification. Leave empty if no state identified.
6. Political: ONLY if the submitter explicitly self-identifies their political party (e.g. "as a Republican", "registered Democrat"). Do NOT infer from policy positions. Leave empty if no explicit self-identification.
- Republican
- Democrat
- Independent
- Libertarian
7. Political Quote: Verbatim quote where the submitter self-identifies their political affiliation. Leave empty if none.
8. Key Quote: The most important verbatim quote (max 100 words) capturing the essence of the comment.
9. Rationale: Brief explanation (1-2 sentences) of the classification selections.
Instructions:
- A comment may express multiple concerns or no clear position
- Only select items clearly expressed in the comment
- Be objective and avoid inserting personal opinions
- For entity type, prioritize explicit self-identification in the comment text over the submitter name
- Supporting the goals of oversight or fiscal responsibility while opposing the rule is still Oppose
gpt-5.4-mini
Comments the first pass called Support or Unclear.
A stronger model re-reads them and can overturn the first pass.
Changed 2026-08-15. "Support" had counted agreement with the rule's goals, so wanting less waste or more oversight registered as support even when the comment never backed the rule. It now requires endorsing the rule or one of its provisions. Every comment labelled Support was then re-checked against its own text by a separate audit — gpt-5.4-mini, on a different prompt written to disagree rather than confirm — and the ones it disputed were re-run through this second pass, on the prompt shown here. 161 changed. Comments already labelled Oppose or Unclear were not re-run, so they still carry whatever the prompt said when they were labelled.
You are reviewing a public comment about a proposed OMB rule revising the Uniform Guidance for federal financial assistance (2 CFR, including sections 200.205 and 200.340). The rule would shift grant review and funding decisions away from independent expert peer review toward political appointees and attach new conditions to federal awards. IMPORTANT CONTEXT: The vast majority of comments OPPOSE this rule. Classify the comment's overall position as exactly one of: "Oppose", "Support", or "Unclear". - "Oppose" — opposes the rule. This includes comments that: object to political control of grant decisions; defend peer review, scientific merit, or scientific independence; warn of harm to research, public health, or U.S. competitiveness; criticize OMB, Russ Vought, or the administration over this rule; or express general outrage at the proposal even without naming a specific section. Supporting the general idea of oversight or fiscal responsibility while objecting to THIS rule is still Oppose. - "Support" — the comment endorses THIS RULE, or specific provisions of it. Take an explicit "I support this rule" / "OMB should finalize this" at face value. Endorsing named provisions (e.g. "we support the performance-measurement requirement in §200.301") is Support. This requires an endorsement, not agreement with a goal. Wanting less waste, more oversight, more accountability, punishment of fraud, or political priority-setting is NOT Support unless the comment ties that to backing this rule. Complaining about how grant money has been misspent is a grievance, not an endorsement. Asking OMB to revise, soften, or reconsider any part of the rule is NOT Support. Defending peer review or scientific merit is never Support — that is the position this rule is understood to weaken. If the comment would read the same way had this rule never been proposed, it is "Unclear", not "Support". - "Unclear" — use this in EITHER of two cases: (a) the comment is truly off-topic (not about federal grants or the rule), is a single ambiguous word, is incoherent, or is just docket metadata with no substantive content; or (b) the comment engages with the subject but states NO position on the rule itself — it is procedural (asking to extend the comment period, asking a question about process), it only describes or summarises the rule, or it argues for something adjacent (general oversight, accountability, fraud prevention, staffing) without endorsing or objecting to THIS rule. Wanting a goal the rule claims to serve is NOT the same as supporting the rule. If there is ANY indication of opposition sentiment, classify as Oppose. Do NOT reach for "Support" merely because the comment is not hostile — silence on the rule is "Unclear". SARCASM CHECK: before calling something "Support", make sure the praise is sincere. A comment that says the rule is "amazing"/"wonderful" or "I support this" but then describes it causing harm — driving scientists or funding abroad, helping rival nations, ending U.S. scientific leadership, destroying research — is sarcastic and is OPPOSE. Genuine support argues the rule is good policy (oversight, less waste/fraud/abuse, taxpayer accountability). LEAN TOWARD OPPOSE when the sentiment is negative toward the proposal.
gpt-5.4
200 sampled comments.
Labels the sample from scratch, without seeing what the pipeline said.
You are building a gold-standard evaluation set. A human will review every one of your
labels, so show your work: quote the words you relied on and argue both sides.
THE RULE: Proposed OMB rule revising the Uniform Guidance for federal financial assistance (2 CFR, including sections 200.205 and 200.340 and several cost-principle sections). Commenters overwhelmingly describe it as shifting grant review and funding decisions away from independent expert peer review toward political appointees, attaching ideological conditions to awards, and restricting allowable costs.
Choose exactly one label:
- "oppose" — OPPOSE means the comment objects to this rule, read generously: objecting to any provision, defending peer review or scientific independence, warning of harm to research, public health or U.S. competitiveness from this proposal, criticising OMB or the administration over it, or general outrage at the proposal. It is NOT oppose when the comment states no position, is purely procedural, only describes the rule, is off-topic or incoherent, or in fact supports it.
- "support" — SUPPORT means the comment endorses THIS RULE or specific provisions of it — "I support this rule", "OMB should finalize this", or endorsing a named provision. It is NOT support when the comment merely wants less waste/fraud, more oversight or accountability without tying that to backing this rule; complains about how grant money has been misspent; asks OMB to revise, soften or reconsider any part of the rule; defends peer review or scientific merit (the position this rule weakens); or would read the same way had this rule never been proposed.
- "no_position" — engages the subject but takes no side: procedural (e.g. asking to extend the comment period), only describes the proposal, asks a question, or argues for something adjacent without endorsing or objecting to it.
- "unclear_junk" — off-topic, incoherent, a single word, or no substantive content.
ADDITIONAL GUIDANCE:
This docket is overwhelmingly opposed, so do NOT let the base rate decide a close call — judge each comment on its own text. Watch the boundary that matters here: agreeing with a goal the rule claims to serve (less waste, more oversight, punishing fraud) is NOT the same as endorsing the rule, and defending peer review or scientific merit is opposition rather than a neutral observation. Quote the exact words you relied on. If the comment is a form letter, judge the letter's text, not what you assume the campaign intended.
For every comment give:
label — your call
evidence — the VERBATIM words you relied on (or "" if there are none)
case_for — why that label, in 1-2 sentences
runner_up — the next most defensible label
case_against — the strongest argument for the runner_up, stated fairly
what_makes_it_hard — what a careful reader could trip on, or "" if it is clear-cut
reviewer_should_check — the specific question a human should answer to settle it,
or "" if it is clear-cut
COMMENT:
{TEXT}
We read those and made the call. The scores on this page use our call, not the model's.
The front page counts what the tool said. This corrects for the errors we found. A wide interval means we checked too few comments in a big category.
| Position | The tool says | Best estimate | 95% interval |
|---|---|---|---|
| No position / unclear | 0.60% | 0.49% | 0.43–0.54% |
| Oppose | 96.29% | 96.39% | 96.33–96.44% |
| Support | 3.11% | 3.12% | 3.11–3.14% |
Precision: when the tool says this, how often it's right. Recall: how many of them it catches.
| Position | Precision | Recall | Labelled |
|---|---|---|---|
| No position / unclear | 82.1% | 100.0% | 55 |
| Oppose | 100.0% | 99.9% | 77 |
| Support | 100.0% | 99.8% | 68 |
Rows are what the comment is. Columns are what the tool called it. Click a cell to read those comments.
| Really is ↓ Tool said → |
No position / unclear | Oppose | Support |
|---|---|---|---|
| No position / unclear | 55 | 0 | 0 |
| Oppose | 11 | 66 | 0 |
| Support | 1 | 0 | 67 |
Click a row for the quote behind the label and the argument against it.