Perturbed datasets for GLUE tasks?

[ecrows]

I'm struggling to compute fairscore without a perturbed version of the GLUE datasets. I'd be happy to regenerate the perturbed datasets using the perturber model, but I haven't been able to figure out how the selected_word and perturbed_category were selected. Were these also crowdsourced? Would it possible to provide the perturbed GLUE datasets used to evaluate fairscore alongside PANDA to enable others to calculate fairscore on these common tasks? Thanks for the great repo!

[EricMichaelSmith]

(Flagging to @Rebecca-Qian )

[ecrows]

Thanks for flagging this -- I'm still stuck trying to find a way to get "selected_word" and "perturbed_category" for the GLUE datasets in order to calculate fairscore. Is it actually even possible to reproduce a fairscore calculation without these?

As an aside, I see @Rebecca-Qian has departed facebookresearch. Is there anyone else I can reach out to about this? Thanks!

[EricMichaelSmith]

Thanks @ecrows for your message - I've just reached out to @Rebecca-Qian (the main point person on this dataset) directly about this.

[qingquan63]

Hello! I have the same question. Do you deal with it?