←
Steven Yu8 posts
Steven Yu avatar

Steven Yu

@lamsheeper
◷ Joined October 2021▧ 6 Photos and videos
Steven Yu avatar
Steven Yu@lamsheeper·Oct 30

Does this generalize? On FTRACE-TREX, we remove each method's top-ranked training documents and retrain. If those documents really taught the facts, accuracy on those facts should drop. With our fixes, the accuracy drop is substantially larger for every method we test.

Post attachment
Steven Yu avatar
Steven Yu@lamsheeper·Oct 30

Our fixes: 1️⃣ Average only a document's top 10 tokens. 2️⃣ Negative log-odds instead of cross-entropy, so confidently predicted answer tokens retain signal The right documents now rank first: recall rises 35–72 points across methods.

Post attachment
Steven Yu avatar
Steven Yu@lamsheeper·Oct 30

At the token level, we find two problems: - Most tokens are unrelated to the query. Averaging over a document buries the few that matter. - Cross-entropy loss gradients shrink scores from high-confidence tokens, suppressing signal from facts the model learned well.

Post attachment
Steven Yu avatar
Steven Yu@lamsheeper·Oct 30

Because the facts are made up, we know exactly which training documents taught each answer, allowing us to evaluate attribution directly. Surprisingly, in more difficult settings, even the strongest attribution methods break down. Why?

Steven Yu avatar
Steven Yu@lamsheeper·Oct 30

Real training data is messy: a fact repeats across many documents, mixed with other misleading data. CATT simulates this in a controlled way. We finetune Qwen on synthetic facts and vary one complexity axis at a time (e.g., redundancy, fact count) to see what breaks attribution.

0:20
Steven Yu avatar
Steven Yu@lamsheeper·Oct 30

Can we trace an AI model's answers back to its training data? Surprisingly, even when we know exactly which documents taught a model a fact, attribution methods often fail to find them. Our CATT benchmark investigates why and uncovers two simple fixes catt.baulab.info🧵

0:40