Zum Hauptinhalt springen
:professors
:professors
all professors

Yufang Hou

Professor of Natural Language Processing

title
Prof.
email
:zurück zur übersicht

Publications

TitleYear
Towards a Unified Model of Scholarly Argumentation (Dagstuhl Seminar 22432)
2025
Khalid Al-Khatib, Anita de Waard, Dayne Freitag, Iryna Gurevych, Yufang Hou, Harrisen S…
12 (10)
:mehr lesen
Authors:Khalid Al-Khatib, Anita de Waard, Dayne Freitag, Iryna Gurevych, Yufang Hou, Harrisen Scells
Publication date:05/05/2025
Volume:12
Issue:10
Publisher:Schloss Dagstuhl-Leibniz-Zentrum für Informatik
Description:This report summarizes the outcomes of the Dagstuhl Seminar 22432:" Towards a Unified Model of Scholarly Argumentation." The purpose of this Seminar was to enable robust advances in argumentation technology by collecting and collaborating on use cases in scholarly and biomedical discourse and working on a foundational model for argumentation in science and healthcare. Most importantly, the seminar served to develop a multidisciplinary, international research community devoted to building and maintaining principles, tools, and models for studying scholarly argumentation. Over the course of the seminar week, the seminar laid the foundation of a shared formalism, illuminated important scholarly use cases for argumentation modeling, and identified directions for future exploration.
Cited by:2
Resources:[PDF] from dagstuhl.de
:view less
Using machine learning to extract information and predict outcomes from reports of randomised trials of smoking cessation interventions in the Human Behaviour-Change Project
2025
Robert West, Francesca Bonin, James Thomas, Alison J Wright, Pol Mac Aonghusa, Martin G…
Wellcome Open Research 8
:mehr lesen
Authors:Robert West, Francesca Bonin, James Thomas, Alison J Wright, Pol Mac Aonghusa, Martin Gleize, Yufang Hou, Alison O'Mara-Eves, Janna Hastings, Marie Johnston, Susan Michie
Publication date:05/05/2025
Journal:Wellcome Open Research
Volume:8
Publisher:The Wellcome Trust
Description:Background Using reports of randomised trials of smoking cessation interventions as a test case, this study aimed to develop and evaluate machine learning (ML) algorithms for extracting information from study reports and predicting outcomes as part of the Human Behaviour-Change Project. It is the first of two linked papers, with the second paper reporting on further development of a prediction system. Methods Researchers manually annotated 70 items of information (‘entities’) in 512 reports of randomised trials of smoking cessation interventions covering intervention content and delivery, population, setting, outcome and study methodology using the Behaviour Change Intervention Ontology. These entities were used to train ML algorithms to extract the information automatically. The information extraction ML algorithm involved a named-entity recognition system using the ‘FLAIR’ framework. The manually …
Cited by:11
Resources:[HTML] from nih.gov
:view less
Fantastic Questions and Where to Find Them: FairytaleQA–An Authentic Dataset for Narrative Comprehension
2025
Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, …
:mehr lesen
Authors:Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Nora Bradford, Branda Sun, Tran Bao Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, Mark Warschauer
Publication date:05/05/2025
Description:Question answering (QA) is a fundamental means to facilitate assessment and training of narrative comprehension skills for both machines and young children, yet there is scarcity of high-quality QA datasets carefully designed to serve this purpose. In particular, existing datasets rarely distinguish fine-grained reading skills, such as the understanding of varying narrative elements. Drawing on the reading education research, we introduce FairytaleQA, a dataset focusing on narrative comprehension of kindergarten to eighth-grade students. Generated by educational experts based on an evidence-based theoretical framework, FairytaleQA consists of 10,580 explicit and implicit questions derived from 278 children-friendly stories, covering seven types of narrative elements or relations. Our dataset is valuable in two folds: First, we ran existing QA models on our dataset and confirmed that this annotation helps assess models' fine-grained learning skills. Second, the dataset supports question generation (QG) task in the education domain. Through benchmarking with QG models, we show that the QG model trained on FairytaleQA is capable of asking high-quality and more diverse questions.
Cited by:103
Resources:[PDF] from arxiv.org
:view less
Overview of KPA-2021 shared task: Key point based quantitative summarization
2025
Roni Friedman, Lena Dankin, Yoav Katz, Yufang Hou, Noam Slonim
Proceedings of the 8th Workshop on Argumentation Mining. Association for Computational …
:mehr lesen
Authors:Roni Friedman, Lena Dankin, Yoav Katz, Yufang Hou, Noam Slonim
Publication date:05/05/2025
Journal:Proceedings of the 8th Workshop on Argumentation Mining. Association for Computational Linguistics
Cited by:30
:view less
The GEM benchmark: Natural language generation, its evaluation and metrics
2025
Gehrmann Sebastian, Adewumi Tosin, Aggarwal Karmanya, Ammanamanchi Pawan Sasanka, Anuol…
arXiv preprint arXiv: 2102.01672
:mehr lesen
Authors:Gehrmann Sebastian, Adewumi Tosin, Aggarwal Karmanya, Ammanamanchi Pawan Sasanka, Anuoluwapo Aremu, Bosselut Antoine, Chandu Khyathi Raghavi, Clinciu Miruna, Das Dipanjan, Wanyu Du, Durmus Esin, Dušek Ondřej, Emezue Chris, Gangal Varun, Garbacea Cristina, Hashimoto Tatsunori, Hou Yufang, Jernite Yacine, Jhamtani Harsh, Ji Yangfeng, Jolly Shailza, Kumar Dhruv, Ladhak Faisal, Madaan Aman, Maddela Mounica, Mahajan Khyati, Mahamood Saad, Majumder Bodhisattwa Prasad, Martins Pedro Henrique, McMillan-Major Angelina, Mille Simon, Nadeem Moin, Narayan Shashi, Nikolaev Vitaly, Niyongabo Rubungo Andre, Osei Salomey, Parikh Ankur, Beltrachini Laura Perez, Rao Niranjan Ramesh, Raunak Vikas, Rodriguez Juan Diego, Santhanam Sashank, Sedoc João, Sellam Thibault, Shaikh Samira, Shimorina Anastasia, Cabezudo Marco Antonio Sobrevilla, Strobelt Hendrik, Subramani Nishant, Wei Xu, Yang Diyi, Yerukola Akhila, Zhou Jiawei
Publication date:05/05/2025
Journal:arXiv preprint arXiv: 2102.01672
:view less
Outcome Prediction from Behaviour Change Intervention Evaluations using a Combination of Node and Word Embedding
2025
Debasis Ganguly, Martin Gleize, Yufang Hou, Charles Jochim, Francesca Bonin, Alessandra…
AMIA Annual Symposium Proceedings 2021, 486
:mehr lesen
Authors:Debasis Ganguly, Martin Gleize, Yufang Hou, Charles Jochim, Francesca Bonin, Alessandra Pascale, Pierpaolo Tommasi, Pol Mac Aonghusa, Robert West, Marie Johnston, Mike Kelly, Susan Michie
Publication date:05/05/2025
Journal:AMIA Annual Symposium Proceedings
Volume:2021
Pages:486
Publisher:American Medical Informatics Association
Description:Findings from randomized controlled trials (RCTs) of behaviour change interventions encode much of our knowledge on intervention efficacy under defined conditions. Predicting outcomes of novel interventions in novel conditions can be challenging, as can predicting differences in outcomes between different interventions or different conditions. To predict outcomes from RCTs, we propose a generic framework of combining the information from two sources - i) the instances (comprised of surrounding text and their numeric values) of relevant attributes, namely the intervention, setting and population characteristics of a study, and ii) abstract representation of the categories of these attributes themselves. We demonstrate that this way of encoding both the information about an attribute and its value when used as an embedding layer within a standard deep sequence modeling setup improves the outcome prediction …
Cited by:5
Resources:[HTML] from nih.gov
:view less
Employing Argumentation Knowledge Graphs for Neural Argument Generation
2025
Khalid Al-Khatib, Lukas Trautner, Henning Wachsmuth, Yufang Hou, Benno Stein
:mehr lesen
Authors:Khalid Al-Khatib, Lukas Trautner, Henning Wachsmuth, Yufang Hou, Benno Stein
Publication date:05/05/2025
Description:Generating high-quality arguments, while being challenging, may benefit a wide range of downstream applications, such as writing assistants and argument search engines. Motivated by the effectiveness of utilizing knowledge graphs for supporting general text generation tasks, this paper investigates the usage of argumentation-related knowledge graphs to control the generation of arguments. In particular, we construct and populate three knowledge graphs, employing several compositions of them to encode various knowledge into texts of debate portals and relevant paragraphs from Wikipedia. Then, the texts with the encoded knowledge are used to fine-tune a pre-trained text generation model, GPT-2. We evaluate the newly created arguments manually and automatically, based on several dimensions important in argumentative contexts, including argumentativeness and plausibility. The results demonstrate the positive impact of encoding the graphs’ knowledge into debate portal texts for generating arguments with superior quality than those generated without knowledge.
Cited by:32
Resources:[PDF] from aclanthology.org
:view less
Knowledge Extraction and Prediction from Behavior Science Randomized Controlled Trials: A Case Study in Smoking Cessation
2025
Francesca Bonin, Martin Gleize, Yufang Hou, Debasis Ganguly, Ailbhe N Finnerty, Charles…
AMIA Annual Symposium Proceedings 2020, 253
:mehr lesen
Authors:Francesca Bonin, Martin Gleize, Yufang Hou, Debasis Ganguly, Ailbhe N Finnerty, Charles Jochim, Alessandra Pascale, Pierpaolo Tommasi, Pol Mac Aonghusa, Susan Michie
Publication date:05/05/2025
Journal:AMIA Annual Symposium Proceedings
Volume:2020
Pages:253
Publisher:American Medical Informatics Association
Description:Due to the fast pace at which randomized controlled trials are published in the health domain, researchers, consultants and policymakers would benefit from more automatic ways to process them by both extracting relevant information and automating the meta-analysis processes. In this paper, we present a novel methodology based on natural language processing and reasoning models to 1) extract relevant information from RCTs and 2) predict potential outcome values on novel scenarios, given the extracted knowledge, in the domain of behavior change for smoking cessation.
Cited by:6
Resources:[HTML] from nih.gov
:view less
Information extraction of behavior change intervention descriptions
2025
Debasis Ganguly, Yufang Hou, Francesca Bonin
AMIA Summits on Translational Science Proceedings 2019, 182
:mehr lesen
Authors:Debasis Ganguly, Yufang Hou, Francesca Bonin
Publication date:05/05/2025
Journal:AMIA Summits on Translational Science Proceedings
Volume:2019
Pages:182
Publisher:American Medical Informatics Association
Description:We describe an information extraction (IE) approach for knowledge base population of behavior change scientific intervention findings. In this paper, we focus on building a system able to characterize the specific intervention techniques that are undertaken within behavior change intervention studies. We have investigated three different configurations of a general information retrieval based framework for information extraction: a) an unsupervised approach that hinges on specification of a query for each attribute to be extracted and a few parameters for rule-based post-processing; b) a semi-supervised approach, which uses a part of the ground-truth annotations as a training set to automatically learn optimal representation of the queries; and c) a supervised approach that replaces the rule-based post processing by a text classifier. To train and evaluate our system, we make use of a ground-truth data set annotated …
Cited by:6
Resources:[HTML] from nih.gov
:view less
Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction
2025
Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, Debasis Ganguly
:mehr lesen
Authors:Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, Debasis Ganguly
Publication date:05/05/2025
Description:While the fast-paced inception of novel tasks and new datasets helps foster active research in a community towards interesting directions, keeping track of the abundance of research activity in different areas on different datasets is likely to become increasingly difficult. The community could greatly benefit from an automatic system able to summarize scientific results, e.g., in the form of a leaderboard. In this paper we build two datasets and develop a framework (TDMS-IE) aimed at automatically extracting task, dataset, metric and score from NLP papers, towards the automatic construction of leaderboards. Experiments show that our model outperforms several baselines by a large margin. Our model is a first step towards automatic leaderboard construction, e.g., in the NLP domain.
Cited by:104
Resources:[PDF] from arxiv.org
:view less
A Deterministic Algorithm for Bridging Anaphora Resolution
2025
Yufang Hou
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,…
:mehr lesen
Authors:Yufang Hou
Publication date:05/05/2025
Journal:Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
Pages:1938-1948
Description:Previous work on bridging anaphora resolution (Poesio et al., 2004; Hou et al., 2013b) use syntactic preposition patterns to calculate word relatedness. However, such patterns only consider NPs' head nouns and hence do not fully capture the semantics of NPs. Recently, Hou (2018) created word embeddings (embeddings_PP) to capture associative similarity (ie, relatedness) between nouns by exploring the syntactic structure of noun phrases. But embeddings_PP only contains word representations for nouns. In this paper, we create new word vectors by combining embeddings_PP with GloVe. This new word embeddings (embeddings_bridging) are a more general lexical knowledge resource for bridging and allow us to represent the meaning of an NP beyond its head easily. We therefore develop a deterministic approach for bridging anaphora resolution, which represents the semantics of an NP based on its head noun and modifications. We show that this simple approach achieves the competitive results compared to the best system in Hou et al.(2013b) which explores Markov Logic Networks to model the problem. Additionally, we further improve the results for bridging anaphora resolution reported in Hou (2018) by combining our simple deterministic approach with Hou et al.(2013b)'s best system MLN II.
Cited by:24
Resources:[PDF] from arxiv.org
:view less
Know Who Your Friends Are: Understanding Social Connections from Unstructured Text
2025
Lea Deleris, Francesca Bonin, Elizabeth Daly, Stephane Deparis, Yufang Hou, Charles Joc…
Proceedings of the 2018 Conference of the North American Chapter of the Association for…
:mehr lesen
Authors:Lea Deleris, Francesca Bonin, Elizabeth Daly, Stephane Deparis, Yufang Hou, Charles Jochim, Yassine Lassoued, Killian Levacher
Publication date:05/05/2025
Journal:Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations
Pages:76-80
Description:Having an understanding of interpersonal relationships is helpful in many contexts. Our system seeks to assist humans with that task, using textual information (eg, case notes, speech transcripts, posts, books) as input. Specifically, our system first extracts qualitative and quantitative information elements (which we call signals) about interactions among persons, aggregates those to provide a condensed view of relationships and then enables users to explore all facets of the resulting social (multi-) graph through a visual interface.
Cited by:3
Resources:[PDF] from aclanthology.org
:view less
The Cool Cucumber System at the 2017 TAC KBP BeSt Evaluation.
2025
Thanh-Son Nguyen, Yufang Hou, Charles Jochim, Elizabeth M Daly, Léa A Deleris
:mehr lesen
Authors:Thanh-Son Nguyen, Yufang Hou, Charles Jochim, Elizabeth M Daly, Léa A Deleris
Publication date:05/05/2025
Description:The 2017 BeSt Evaluation is an evaluation of targeted belief and sentiment detection. It seeks to answer the question “who has what mental attitude towards whom/what?”. Our team participated in the sentiment evaluation for English, starting from known mentions of entities, relations and events. In this situation, the difficulty lies in part in determining the pairs that carry sentiment from the multitude of potential pairs that can be created by combining all the known mentions. To deal with this issue, we propose a two-stage classifier to (1) eliminate objective pairs (ie, pairs that do not have sentiment) and (2) predict sentiment polarity values for the predicted-subjective pairs.
Resources:[PDF] from nist.gov
:view less
Argumentation Quality Assessment: Theory vs. Practice
2025
Henning Wachsmuth, Nona Naderi, Ivan Habernal, Yufang Hou, Graeme Hirst, Iryna Gurevych…
:mehr lesen
Authors:Henning Wachsmuth, Nona Naderi, Ivan Habernal, Yufang Hou, Graeme Hirst, Iryna Gurevych, Benno Stein
Publication date:05/05/2025
Description:Argumentation quality is viewed differently in argumentation theory and in practical assessment approaches. This paper studies to what extent the views match empirically. We find that most observations on quality phrased spontaneously are in fact adequately represented by theory. Even more, relative comparisons of arguments in practice correlate with absolute quality ratings based on theory. Our results clarify how the two views can learn from each other.
Cited by:96
Resources:[PDF] from aclanthology.org
:view less
Computational Argumentation Quality Assessment in Natural Language
2025
Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim A…
:mehr lesen
Authors:Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberdingk Thijm, Graeme Hirst, Benno Stein
Publication date:05/05/2025
Description:Research on computational argumentation faces the problem of how to automatically assess the quality of an argument or argumentation. While different quality dimensions have been approached in natural language processing, a common understanding of argumentation quality is still missing. This paper presents the first holistic work on computational argumentation quality in natural language. We comprehensively survey the diverse existing theories and approaches to assess logical, rhetorical, and dialectical quality dimensions, and we derive a systematic taxonomy from these. In addition, we provide a corpus with 320 arguments, annotated for all 15 dimensions in the taxonomy. Our results establish a common ground for research on computational argumentation quality assessment.
Cited by:278
Resources:[PDF] from aclanthology.org
:view less
Incremental Fine-grained Information Status Classification Using Attention-based LSTMs
2025
Yufang Hou
:mehr lesen
Authors:Yufang Hou
Publication date:05/05/2025
Description:Information status plays an important role in discourse processing. According to the hearer’s common sense knowledge and his comprehension of the preceding text, a discourse entity could be old, mediated or new. In this paper, we propose an attention-based LSTM model to address the problem of fine-grained information status classification in an incremental manner. Our approach resembles how human beings process the task, ie, decide the information status of the current discourse entity based on its preceding context. Experimental results on the ISNotes corpus (Markert et al., 2012) reveal that (1) despite its moderate result, our model with only word embedding features captures the necessary semantic knowledge needed for the task by a large extent; and (2) when incorporating with additional several simple features, our model achieves the competitive results compared to the state-of-the-art approach (Hou et al., 2013) which heavily depends on lots of hand-crafted semantic features.
Cited by:21
Resources:[PDF] from aclanthology.org
:view less
Unrestricted Bridging Resolution
2025
Yufang Hou
:mehr lesen
Authors:Yufang Hou
Publication date:05/05/2025
:view less
Analyzing sentiment in classical Chinese poetry
2025
Yufang Hou, Anette Frank
Proceedings of the 9th SIGHUM Workshop on Language Technology for Cultural Heritage, So…
:mehr lesen
Authors:Yufang Hou, Anette Frank
Publication date:05/05/2025
Journal:Proceedings of the 9th SIGHUM Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities (LaTeCH)
Pages:15-24
Description:Although sentiment analysis in Chinese social media has attracted a lot of interest in recent years, it has been less explored in traditional Chinese literature (eg, classical Chinese poetry) due to the lack of sentiment lexicon resources. In this paper, we propose a weakly supervised approach based on Weighted Personalized PageRank (WPPR) to create a sentiment lexicon for classical Chinese poetry. We evaluate our lexicon intrinsically and extrinsically. We show that our graphbased approach outperforms a previous well-known PMI-based approach (Turney and Littman, 2003) on both evaluation settings. On the basis of our sentiment lexicon, we analyze sentiment in the Complete Anthology of Tang Poetry. We extract topics associated with positive (negative) sentiment using a position-aware sentimenttopic model. We further compare sentiment among different poets in Tang Dynasty (AD 618–907).
Cited by:49
Resources:[PDF] from aclanthology.org
:view less
A Rule-Based System for Unrestricted Bridging Resolution: Recognizing Bridging Anaphora and Finding Links to Antecedents
2025
Yufang Hou, Katja Markert, Michael Strube
EMNLP 2014
:mehr lesen
Authors:Yufang Hou, Katja Markert, Michael Strube
Publication date:05/05/2025
Journal:EMNLP 2014
Description:Bridging resolution plays an important role in establishing (local) entity coherence. This paper proposes a rule-based approach for the challenging task of unrestricted bridging resolution, where bridging anaphors are not limited to definite NPs and semantic relations between anaphors and their antecedents are not restricted to meronymic relations. The system consists of eight rules which target different relations based on linguistic insights. Our rule-based system significantly outperforms a reimplementation of a previous rule-based system (Vieira and Poesio, 2000). Furthermore, it performs better than a learning-based approach which has access to the same knowledge resources as the rule-based system. Additionally, incorporating the rules and more features into the learning-based system yields a minor improvement over the rule-based system.
Cited by:43
Resources:[PDF] from aclanthology.org
:view less
Anaphoric Annotation and Information Structure: Solved Problems, Open Challenges
2025
Massimo Poesio, Arndt Riester, Kordula de Kuthy Cook, Dag Haug, Yufang Hou, Katja Marke…
:mehr lesen
Authors:Massimo Poesio, Arndt Riester, Kordula de Kuthy Cook, Dag Haug, Yufang Hou, Katja Markert, Detmar Meurers, Anna Nedoluzhko, Malvina Nissim, Marta Recasens, Michael Strube, Olga Uryupina, Ramon Ziai
Publication date:05/05/2025
Description:This document is our attempt to collate the analyses of the state of the art in the annotation of anaphora and information structure produced by ourselves and the other participants to the workshop, identifying ‘solved’and ‘open’issues. It is based on the assumption that the two annotation tasks are related (one of us believes that anaphoric annotation is a subset of the annotation of information structure, the other remains agnostic).
Resources:[PDF] from uni-stuttgart.de
:view less
Cascading Collective Classification for Bridging Anaphora Recognition Using a Rich Linguistic Feature Set
2025
Yufang Hou, Katja Markert, Michael Strube
:mehr lesen
Authors:Yufang Hou, Katja Markert, Michael Strube
Publication date:05/05/2025
Description:Recognizing bridging anaphora is difficult due to the wide variation within the phenomenon, the resulting lack of easily identifiable surface markers and their relative rarity. We develop linguistically motivated discourse structure, lexico-semantic and genericity detection features and integrate these into a cascaded minority preference algorithm that models bridging recognition as a subtask of learning finegrained information status (IS). We substantially improve bridging recognition without impairing performance on other IS classes.
Cited by:34
Resources:[PDF] from aclanthology.org
:view less
Global inference for bridging anaphora resolution
2025
Yufang Hou, Katja Markert, Michael Strube
Proceedings of the 2013 Conference of the North American Chapter of the Association for…
:mehr lesen
Authors:Yufang Hou, Katja Markert, Michael Strube
Publication date:05/05/2025
Journal:Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Pages:907-917
Description:We present the first work on antecedent selection for bridging resolution without restrictions on anaphor or relation types. Our model integrates global constraints on top of a rich local feature set in the framework of Markov logic networks. The global model improves over the local one and both strongly outperform a reimplementation of prior work.
Cited by:84
Resources:[PDF] from aclanthology.org
:view less
Weakly supervised graph-based coreference resolution for clinical texts
2025
J Cai, E Mujdricza, Y Hou, M Strube
Proceedings of the 2011 i2b2/VA/Cincinnati Workshop on Challenges in Natural Language P…
:mehr lesen
Authors:J Cai, E Mujdricza, Y Hou, M Strube
Publication date:05/05/2025
Journal:Proceedings of the 2011 i2b2/VA/Cincinnati Workshop on Challenges in Natural Language Processing for Clinical Data. Boston, MA, USA: i2b2
Cited by:5
:view less
FactReasoner: A Probabilistic Approach to Long-Form Factuality Assessment for Large Language Models
2025
Radu Marinescu, Debarun Bhattacharjya, Junkyu Lee, Tigran Tchrakian, Javier Carnerero C…
arXiv preprint arXiv:2502.18573
:mehr lesen
Authors:Radu Marinescu, Debarun Bhattacharjya, Junkyu Lee, Tigran Tchrakian, Javier Carnerero Cano, Yufang Hou, Elizabeth Daly, Alessandra Pascale
Publication date:02/25/2025
Journal:arXiv preprint arXiv:2502.18573
Description:Large language models (LLMs) have demonstrated vast capabilities on generative tasks in recent years, yet they struggle with guaranteeing the factual correctness of the generated content. This makes these models unreliable in realistic situations where factually accurate responses are expected. In this paper, we propose FactReasoner, a new factuality assessor that relies on probabilistic reasoning to assess the factuality of a long-form generated response. Specifically, FactReasoner decomposes the response into atomic units, retrieves relevant contexts for them from an external knowledge source, and constructs a joint probability distribution over the atoms and contexts using probabilistic encodings of the logical relationships (entailment, contradiction) between the textual utterances corresponding to the atoms and contexts. FactReasoner then computes the posterior probability of whether atomic units in the response are supported by the retrieved contexts. Our experiments on labeled and unlabeled benchmark datasets demonstrate clearly that FactReasoner improves considerably over state-of-the-art prompt-based approaches in terms of both factual precision and recall.
Resources:[PDF] from arxiv.org
:view less
Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation
2025
Steffen Eger, Yong Cao, Jennifer D'Souza, Andreas Geiger, Christian Greisinger, Stephan…
arXiv preprint arXiv:2502.05151
:mehr lesen
Authors:Steffen Eger, Yong Cao, Jennifer D'Souza, Andreas Geiger, Christian Greisinger, Stephanie Gross, Yufang Hou, Brigitte Krenn, Anne Lauscher, Yizhi Li, Chenghua Lin, Nafise Sadat Moosavi, Wei Zhao, Tristan Miller
Publication date:02/07/2025
Journal:arXiv preprint arXiv:2502.05151
Description:With the advent of large multimodal language models, science is now at a threshold of an AI-based technological transformation. Recently, a plethora of new AI models and tools has been proposed, promising to empower researchers and academics worldwide to conduct their research more effectively and efficiently. This includes all aspects of the research cycle, especially (1) searching for relevant literature; (2) generating research ideas and conducting experimentation; generating (3) text-based and (4) multimodal content (e.g., scientific figures and diagrams); and (5) AI-based automatic peer review. In this survey, we provide an in-depth overview over these exciting recent developments, which promise to fundamentally alter the scientific research process for good. Our survey covers the five aspects outlined above, indicating relevant datasets, methods and results (including evaluation) as well as limitations and scope for future research. Ethical concerns regarding shortcomings of these tools and potential for misuse (fake science, plagiarism, harms to research integrity) take a particularly prominent place in our discussion. We hope that our survey will not only become a reference guide for newcomers to the field but also a catalyst for new AI-based initiatives in the area of "AI4Science".
Cited by:3
Resources:[PDF] from arxiv.org
:view less
Holmes ⌕ A Benchmark to Assess the Linguistic Competence of Language Models
2024
Andreas Waldis, Yotam Perlitz, Leshem Choshen, Yufang Hou, Iryna Gurevych
Transactions of the Association for Computational Linguistics 12, 1616-1647
:mehr lesen
Authors:Andreas Waldis, Yotam Perlitz, Leshem Choshen, Yufang Hou, Iryna Gurevych
Publication date:12/04/2024
Journal:Transactions of the Association for Computational Linguistics
Volume:12
Pages:1616-1647
Publisher:MIT Press
Description:We introduce Holmes, a new benchmark designed to assess language models’ (LMs’) linguistic competence—their unconscious understanding of linguistic phenomena. Specifically, we use classifier-based probing to examine LMs’ internal representations regarding distinct linguistic phenomena (e.g., part-of-speech tagging). As a result, we meet recent calls to disentangle LMs’ linguistic competence from other cognitive abilities, such as following instructions in prompting-based evaluations. Composing Holmes, we review over 270 probing studies and include more than 200 datasets to assess syntax, morphology, semantics, reasoning, and discourse phenomena. Analyzing over 50 LMs reveals that, aligned with known trends, their linguistic competence correlates with model size. However, surprisingly, model architecture and instruction tuning also significantly influence performance, particularly in …
Cited by:9
Resources:[HTML] from mit.edu
:view less
The Nature of NLP: Analyzing Contributions in NLP Papers
2024
Aniket Pramanick, Yufang Hou, Saif M Mohammad, Iryna Gurevych
arXiv preprint arXiv:2409.19505
:mehr lesen
Authors:Aniket Pramanick, Yufang Hou, Saif M Mohammad, Iryna Gurevych
Publication date:09/29/2024
Journal:arXiv preprint arXiv:2409.19505
Description:Natural Language Processing (NLP) is a dynamic, interdisciplinary field that integrates intellectual traditions from computer science, linguistics, social science, and more. Despite its established presence, the definition of what constitutes NLP research remains debated. In this work, we quantitatively investigate what constitutes NLP by examining research papers. For this purpose, we propose a taxonomy and introduce NLPContributions, a dataset of nearly research paper abstracts, expertly annotated to identify scientific contributions and classify their types according to this taxonomy. We also propose a novel task to automatically identify these elements, for which we train a strong baseline on our dataset. We present experimental results from this task and apply our model to NLP research papers to analyze their contributions, aiding in the understanding of the nature of NLP research. Our findings reveal a rising involvement of machine learning in NLP since the early nineties, alongside a declining focus on adding knowledge about language or people; again, in post-2020, there has been a resurgence of focus on language and people. We hope this work will spark discussions on our community norms and inspire efforts to consciously shape the future.
Cited by:2
Resources:[PDF] from arxiv.org
:view less
Transforming Scholarly Landscapes: Influence of Large Language Models on Academic Fields beyond Computer Science
2024
Aniket Pramanick, Yufang Hou, Saif M Mohammad, Iryna Gurevych
arXiv preprint arXiv:2409.19508
:mehr lesen
Authors:Aniket Pramanick, Yufang Hou, Saif M Mohammad, Iryna Gurevych
Publication date:09/29/2024
Journal:arXiv preprint arXiv:2409.19508
Description:Large Language Models (LLMs) have ushered in a transformative era in Natural Language Processing (NLP), reshaping research and extending NLP's influence to other fields of study. However, there is little to no work examining the degree to which LLMs influence other research fields. This work empirically and systematically examines the influence and use of LLMs in fields beyond NLP. We curate LLMs and analyze papers citing LLMs to quantify their influence and reveal trends in their usage patterns. Our analysis reveals not only the increasing prevalence of LLMs in non-CS fields but also the disparities in their usage, with some fields utilizing them more frequently than others since 2018, notably Linguistics and Engineering together accounting for of LLM citations. Our findings further indicate that most of these fields predominantly employ task-agnostic LLMs, proficient in zero or few-shot learning without requiring further fine-tuning, to address their domain-specific problems. This study sheds light on the cross-disciplinary impact of NLP through LLMs, providing a better understanding of the opportunities and challenges.
Cited by:1
Resources:[PDF] from arxiv.org
:view less
SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinement
2024
Ishani Mondal, Zongxia Li, Yufang Hou, Anandhavelu Natarajan, Aparna Garimella, Jordan …
EMNLP Findings 2024
:mehr lesen
Authors:Ishani Mondal, Zongxia Li, Yufang Hou, Anandhavelu Natarajan, Aparna Garimella, Jordan Boyd-Graber
Publication date:09/28/2024
Journal:EMNLP Findings 2024
Description:Automating the creation of scientific diagrams from academic papers can significantly streamline the development of tutorials, presentations, and posters, thereby saving time and accelerating the process. Current text-to-image models struggle with generating accurate and visually appealing diagrams from long-context inputs. We propose SciDoc2Diagram, a task that extracts relevant information from scientific papers and generates diagrams, along with a benchmarking dataset, SciDoc2DiagramBench. We develop a multi-step pipeline SciDoc2Diagrammer that generates diagrams based on user intentions using intermediate code generation. We observed that initial diagram drafts were often incomplete or unfaithful to the source, leading us to develop SciDoc2Diagrammer-Multi-Aspect-Feedback (MAF), a refinement strategy that significantly enhances factual correctness and visual appeal and outperforms existing models on both automatic and human judgement.
Cited by:3
Resources:[PDF] from arxiv.org
:view less
Efficient Performance Tracking: Leveraging Large Language Models for Automated Construction of Scientific Leaderboards
2024
Furkan Şahinuç, Thy Thy Tran, Yulia Grishina, Yufang Hou, Bei Chen, Iryna Gurevych
EMNLP 2024
:mehr lesen
Authors:Furkan Şahinuç, Thy Thy Tran, Yulia Grishina, Yufang Hou, Bei Chen, Iryna Gurevych
Publication date:09/19/2024
Journal:EMNLP 2024
Description:Scientific leaderboards are standardized ranking systems that facilitate evaluating and comparing competitive methods. Typically, a leaderboard is defined by a task, dataset, and evaluation metric (TDM) triple, allowing objective performance assessment and fostering innovation through benchmarking. However, the exponential increase in publications has made it infeasible to construct and maintain these leaderboards manually. Automatic leaderboard construction has emerged as a solution to reduce manual labor. Existing datasets for this task are based on the community-contributed leaderboards without additional curation. Our analysis shows that a large portion of these leaderboards are incomplete, and some of them contain incorrect information. In this work, we present SciLead, a manually-curated Scientific Leaderboard dataset that overcomes the aforementioned problems. Building on this dataset, we propose three experimental settings that simulate real-world scenarios where TDM triples are fully defined, partially defined, or undefined during leaderboard construction. While previous research has only explored the first setting, the latter two are more representative of real-world applications. To address these diverse settings, we develop a comprehensive LLM-based framework for constructing leaderboards. Our experiments and analysis reveal that various LLMs often correctly identify TDM triples while struggling to extract result values from publications. We make our code and data publicly available.
Cited by:2
Resources:[PDF] from arxiv.org
:view less
SciLead Dataset (Efficient Performance Tracking: Leveraging Large Language Models for Automated Construction of Scientific Leaderboards)
2024
Furkan Sahinuc, Thy Thy Tran, Yulia Grishina, Yufang Hou, Bei Chen, Iryna Gurevych
:mehr lesen
Authors:Furkan Sahinuc, Thy Thy Tran, Yulia Grishina, Yufang Hou, Bei Chen, Iryna Gurevych
Publication date:09/12/2024
Publisher:Technical University of Darmstadt
:view less
The 2024 GEM shared task on multilingual data-to-text generation and summarization: Overview and preliminary results
2024
Simon Mille, João Sedoc, Yixin Liu, Elizabeth Clark, Agnes Johanna Axelsson, Miruna Adr…
Proceedings of the 17th International Natural Language Generation Conference: Generatio…
:mehr lesen
Authors:Simon Mille, João Sedoc, Yixin Liu, Elizabeth Clark, Agnes Johanna Axelsson, Miruna Adriana Clinciu, Yufang Hou, Saad Mahamood, Ishmael Nyunya Obonyo, Lining Zhang
Publication date:09/01/2024
Journal:Proceedings of the 17th International Natural Language Generation Conference: Generation Challenges
Pages:17-38
Description:We present an overview of the GEM 2024 shared task, which comprised of both data-to-text generation and summarization. New datasets were compiled specifically for the task to reduce data contamination in the large language models, which the participants were likely to use. The paper describes the tasks, the datasets, the participating systems, the evaluation methods, and some preliminary results. The full results will be presented at INLG ‘24.
Cited by:5
Resources:[PDF] from aclanthology.org
:view less
Grounding Fallacies Misrepresenting Scientific Publications in Evidence
2024
Max Glockner, Yufang Hou, Preslav Nakov, Iryna Gurevych
arXiv preprint arXiv:2408.12812
:mehr lesen
Authors:Max Glockner, Yufang Hou, Preslav Nakov, Iryna Gurevych
Publication date:08/23/2024
Journal:arXiv preprint arXiv:2408.12812
Description:Health-related misinformation claims often falsely cite a credible biomedical publication as evidence, which superficially appears to support the false claim. The publication does not really support the claim, but a reader could believe it thanks to the use of logical fallacies. Here, we aim to detect and to highlight such fallacies, which requires carefully assessing the exact content of the misrepresented publications. To achieve this, we introduce MissciPlus, an extension of the fallacy detection dataset Missci. MissciPlus builds on Missci by grounding the applied fallacies in real-world passages from misrepresented studies. This creates a realistic test-bed for detecting and verbalizing these fallacies under real-world input conditions, and enables novel passage-retrieval tasks. MissciPlus is the first logical fallacy dataset which pairs the real-world misrepresented evidence with incorrect claims, identical to the input to evidence-based fact-checking models. With MissciPlus, we i) benchmark retrieval models in identifying passages that support claims only when fallacies are applied, ii) evaluate how well LLMs articulate fallacious reasoning from misrepresented scientific passages, and iii) assess the effectiveness of fact-checking models in refuting claims that misrepresent biomedical research. Our findings show that current fact-checking models struggle to use relevant passages from misrepresented publications to refute misinformation. Moreover, these passages can mislead LLMs into accepting false claims as true.
Cited by:3
Resources:[PDF] from arxiv.org
:view less
Reconstructing Fallacies in Misrepresented Science and Argument Mining in the Wild
2024
Yufang Hou
:mehr lesen
Authors:Yufang Hou
Publication date:08/11/2024
Description:In this talk, I will discuss our recent work on applying and investigating language model (LM)-based argument mining technologies in real-world scenarios, including fact-checking misinformation that misrepresents scientific publications and tackling traditional argument mining tasks in various out-of-distribution (OOD) scenarios. First, I will discuss our work on reconstructing and grounding fallacies in misrepresented science, in which health-related misinformation claims often falsely cite a credible biomedical publication as evidence. I will present a new argumentation theoretical model for fallacious reasoning, together with a new dataset for real-world misinformation detection that misrepresents biomedical publications. In the second part of the talk, I will discuss our findings on LMs' capabilities for three OOD scenarios (topic shift, domain shift, and language shift) across eleven argument mining tasks.
:view less
System Architecture
2024
Ailbhe Finnerty Mutlu, Candice Moore, Yufang Hou, Francesca Bonin, Robert West, Susan M…
:mehr lesen
Authors:Ailbhe Finnerty Mutlu, Candice Moore, Yufang Hou, Francesca Bonin, Robert West, Susan Michie, John Shawe-Taylor, James Thomas, Marie Johnston, Pol Mac Aonghusa, Alison O'Mara-Eves, Gillian Stokes, Emma Norris, Marta Marques, Michael P Kelly, Debasis Ganguly, Lea Deleris, Kirsty Atha, Alison Wright, Clement Veall, Emily Hayes, Lisa Zhang, Janna Hastings, Paulina Schenk
Publication date:08/09/2024
Publisher:OSF
Description:The System Architecture team is working to develop user and machine interfaces for the ‘Knowledge System’. This will facilitate a flow of information from ‘data sources’(eg online publications databases such as PubMed, Mendeley, CrossRef etc) through ‘backend processes’, including the information extraction (eg, the extraction of structured data from PDFs) and inference processes, ie, prediction and recommendation algorithms, developed by the computer science team, through to ‘services’, which include a website allowing human users to query the system.
:view less
Beyond Abstracts: A New Dataset, Prompt Design Strategy and Method for Biomedical Synthesis Generation
2024
James O’Doherty, Cian Nolan, Yufang Hou, Anja Belz
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics…
:mehr lesen
Authors:James O’Doherty, Cian Nolan, Yufang Hou, Anja Belz
Publication date:08/01/2024
Journal:Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)
Pages:499-518
Description:The biomedical field relies on cost and time intensive systematic reviews of papers to enable practitioners to keep up to date with research. Impressive recent advances in large language models (LLMs) have made the task of automating at least part of the systematic review process feasible, but progress is slow. This paper identifies some factors that may have been holding research back, and proposes a new, enhanced dataset and prompting-based method for automatic synthesis generation, the most challenging step for automation. We test different models and types of information from and about biomedical studies for their usefulness in obtaining high-quality results. We find that, surprisingly, inclusion of paper abstracts can worsens results. Instead, study summary information, and system instructions informed by domain knowledge, are key to producing high-quality syntheses.
Resources:[PDF] from aclanthology.org
:view less
How to Handle Different Types of Out-of-Distribution Scenarios in Computational Argumentation? A Comprehensive and Fine-Grained Field Study
2024
Andreas Waldis, Yufang Hou, Iryna Gurevych
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics…
:mehr lesen
Authors:Andreas Waldis, Yufang Hou, Iryna Gurevych
Publication date:08/01/2024
Journal:Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Pages:14878-14898
Description:The advent of pre-trained Language Models (LMs) has markedly advanced natural language processing, but their efficacy in out-of-distribution (OOD) scenarios remains a significant challenge. Computational argumentation (CA), modeling human argumentation processes, is a field notably impacted by these challenges because complex annotation schemes and high annotation costs naturally lead to resources barely covering the multiplicity of available text sources and topics. Due to this data scarcity, generalization to data from uncovered covariant distributions is a common challenge for CA tasks like stance detection or argument classification. This work systematically assesses LMs' capabilities for such OOD scenarios. While previous work targets specific OOD types like topic shifts or OOD uniformly, we address three prevalent OOD scenarios in CA: topic shift, domain shift, and language shift. Our findings challenge the previously asserted general superiority of in-context learning (ICL) for OOD. We find that the efficacy of such learning paradigms varies with the type of OOD. Specifically, while ICL excels for domain shifts, prompt-based fine-tuning surpasses for topic shifts. To sum up, we navigate the heterogeneity of OOD scenarios in CA and empirically underscore the potential of base-sized LMs in overcoming these challenges.
Cited by:3
Resources:[PDF] from arxiv.org
:view less
A Course Shared Task on Evaluating LLM Output for Clinical Questions
2024
Yufang Hou, Thy Thy Tran, Doan Nam Long Vu, Yiwen Cao, Kai Li, Lukas Rohde, Iryna Gurevych
the sixth Workshop on Teaching NLP (co-located with ACL 2024)
:mehr lesen
Authors:Yufang Hou, Thy Thy Tran, Doan Nam Long Vu, Yiwen Cao, Kai Li, Lukas Rohde, Iryna Gurevych
Publication date:07/31/2024
Journal:the sixth Workshop on Teaching NLP (co-located with ACL 2024)
Description:This paper presents a shared task that we organized at the Foundations of Language Technology (FoLT) course in 2023/2024 at the Technical University of Darmstadt, which focuses on evaluating the output of Large Language Models (LLMs) in generating harmful answers to health-related clinical questions. We describe the task design considerations and report the feedback we received from the students. We expect the task and the findings reported in this paper to be relevant for instructors teaching natural language processing (NLP) and designing course assignments.
Resources:[PDF] from arxiv.org
:view less
WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia
2024
Yufang Hou, Alessandra Pascale, Javier Carnerero-Cano, Tigran Tchrakian, Radu Marinescu…
NeurIPS R&D Track, 2024
:mehr lesen
Authors:Yufang Hou, Alessandra Pascale, Javier Carnerero-Cano, Tigran Tchrakian, Radu Marinescu, Elizabeth Daly, Inkit Padhi, Prasanna Sattigeri
Publication date:06/19/2024
Journal:NeurIPS R&D Track, 2024
Description:Retrieval-augmented generation (RAG) has emerged as a promising solution to mitigate the limitations of large language models (LLMs), such as hallucinations and outdated information. However, it remains unclear how LLMs handle knowledge conflicts arising from different augmented retrieved passages, especially when these passages originate from the same source and have equal trustworthiness. In this work, we conduct a comprehensive evaluation of LLM-generated answers to questions that have varying answers based on contradictory passages from Wikipedia, a dataset widely regarded as a high-quality pre-training resource for most LLMs. Specifically, we introduce WikiContradict, a benchmark consisting of 253 high-quality, human-annotated instances designed to assess the performance of LLMs in providing a complete perspective on conflicts from the retrieved documents, rather than choosing one answer over another, when augmented with retrieved passages containing real-world knowledge conflicts. We benchmark a diverse range of both closed and open-source LLMs under different QA scenarios, including RAG with a single passage, and RAG with 2 contradictory passages. Through rigorous human evaluations on a subset of WikiContradict instances involving 5 LLMs and over 3,500 judgements, we shed light on the behaviour and limitations of these models. For instance, when provided with two passages containing contradictory facts, all models struggle to generate answers that accurately reflect the conflicting nature of the context, especially for implicit conflicts requiring reasoning. Since human evaluation is costly …
Cited by:6
Resources:[PDF] from neurips.cc
:view less
MISSCI: Reconstructing Fallacies in Misrepresented Science
2024
Max Glockner, Yufang Hou, Preslav Nakov, Iryna Gurevych
ACL 2024
:mehr lesen
Authors:Max Glockner, Yufang Hou, Preslav Nakov, Iryna Gurevych
Publication date:06/01/2024
Journal:ACL 2024
Description:Health-related misinformation on social networks can lead to poor decision-making and real-world dangers. Such misinformation often misrepresents scientific publications and cites them as "proof" to gain perceived credibility. To effectively counter such claims automatically, a system must explain how the claim was falsely derived from the cited publication. Current methods for automated fact-checking or fallacy detection neglect to assess the (mis)used evidence in relation to misinformation claims, which is required to detect the mismatch between them. To address this gap, we introduce Missci, a novel argumentation theoretical model for fallacious reasoning together with a new dataset for real-world misinformation detection that misrepresents biomedical publications. Unlike previous fallacy detection datasets, Missci (i) focuses on implicit fallacies between the relevant content of the cited publication and the inaccurate claim, and (ii) requires models to verbalize the fallacious reasoning in addition to classifying it. We present Missci as a dataset to test the critical reasoning abilities of large language models (LLMs), that are required to reconstruct real-world fallacious arguments, in a zero-shot setting. We evaluate two representative LLMs and the impact of different levels of detail about the fallacy classes provided to the LLM via prompts. Our experiments and human evaluation show promising results for GPT 4, while also demonstrating the difficulty of this task.
Cited by:11
Resources:[PDF] from arxiv.org
:view less
Exploring Task Definitions with LLMs: A Study in Citation Text Generation
2024
Furkan Sahinuc, Ilia Kuznetsov, Yufang Hou, Iryna Gurevych
:mehr lesen
Authors:Furkan Sahinuc, Ilia Kuznetsov, Yufang Hou, Iryna Gurevych
Publication date:05/27/2024
Cited by:2
:view less
On the Role of Summary Content Units in Text Summarization Evaluation
2024
Marcel Nawrath, Agnieszka Nowak, Tristan Ratz, Danilo C Walenta, Juri Opitz, Leonardo F…
NAACL 2024
:mehr lesen
Authors:Marcel Nawrath, Agnieszka Nowak, Tristan Ratz, Danilo C Walenta, Juri Opitz, Leonardo FR Ribeiro, João Sedoc, Daniel Deutsch, Simon Mille, Yixin Liu, Lining Zhang, Sebastian Gehrmann, Saad Mahamood, Miruna Clinciu, Khyathi Chandu, Yufang Hou
Publication date:04/02/2024
Journal:NAACL 2024
Description:At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs). These SCUs are concise sentences that decompose a summary into small facts. Such SCUs can be used to judge the quality of a candidate summary, possibly partially automated via natural language inference (NLI) systems. Interestingly, with the aim to fully automate the Pyramid evaluation, Zhang and Bansal (2021) show that SCUs can be approximated by automatically generated semantic role triplets (STUs). However, several questions currently lack answers, in particular: i) Are there other ways of approximating SCUs that can offer advantages? ii) Under which conditions are SCUs (or their approximations) offering the most value? In this work, we examine two novel strategies to approximate SCUs: generating SCU approximations from AMR meaning representations (SMUs) and from large language models (SGUs), respectively. We find that while STUs and SMUs are competitive, the best approximation quality is achieved by SGUs. We also show through a simple sentence-decomposition baseline (SSUs) that SCUs (and their approximations) offer the most value when ranking short summaries, but may not help as much when ranking systems or longer summaries.
Cited by:6
Resources:[PDF] from arxiv.org
:view less
Dive into the Chasm: Probing the Gap between In-and Cross-Topic Generalization
2024
Andreas Waldis, Yufang Hou, Iryna Gurevych
EACL Findings 2024
:mehr lesen
Authors:Andreas Waldis, Yufang Hou, Iryna Gurevych
Publication date:02/02/2024
Journal:EACL Findings 2024
Description:Pre-trained language models (LMs) perform well in In-Topic setups, where training and testing data come from the same topics. However, they face challenges in Cross-Topic scenarios where testing data is derived from distinct topics -- such as Gun Control. This study analyzes various LMs with three probing-based experiments to shed light on the reasons behind the In- vs. Cross-Topic generalization gap. Thereby, we demonstrate, for the first time, that generalization gaps and the robustness of the embedding space vary significantly across LMs. Additionally, we assess larger LMs and underscore the relevance of our analysis for recent models. Overall, diverse pre-training objectives, architectural regularization, or data deduplication contribute to more robust LMs and diminish generalization gaps. Our research contributes to a deeper understanding and comparison of language models across different generalization scenarios.
Cited by:5
Resources:[PDF] from arxiv.org
:view less
A Diachronic Analysis of Paradigm Shifts in NLP Research: When, How, and Why?
2023
Aniket Pramanick, Yufang Hou, Saif Mohammad, Iryna Gurevych
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,…
:mehr lesen
Authors:Aniket Pramanick, Yufang Hou, Saif Mohammad, Iryna Gurevych
Publication date:12/01/2023
Journal:Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Pages:2312-2326
Description:Understanding the fundamental concepts and trends in a scientific field is crucial for keeping abreast of its continuous advancement. In this study, we propose a systematic framework for analyzing the evolution of research topics in a scientific field using causal discovery and inference techniques. We define three variables to encompass diverse facets of the evolution of research topics within NLP and utilize a causal discovery algorithm to unveil the causal connections among these variables using observational data. Subsequently, we leverage this structure to measure the intensity of these relationships. By conducting extensive experiments on the ACL Anthology corpus, we demonstrate that our framework effectively uncovers evolutionary trends and the underlying causes for a wide range of NLP research topics. Specifically, we show that tasks and methods are primary drivers of research in NLP, with datasets following, while metrics have minimal impact.
Cited by:6
Resources:[PDF] from arxiv.org
:view less
‘Don’t Get Too Technical with Me’: A Discourse Structure-Based Framework for Automatic Science Journalism
2023
Ronald Cardenas, Bingsheng Yao, Dakuo Wang, Yufang Hou
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,…
:mehr lesen
Authors:Ronald Cardenas, Bingsheng Yao, Dakuo Wang, Yufang Hou
Publication date:12/01/2023
Journal:Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Pages:1186-1202
Description:Science journalism refers to the task of reporting technical findings of a scientific paper as a less technical news article to the general public audience. We aim to design an automated system to support this real-world task (ie, automatic science journalism) by 1) introducing a newly-constructed and real-world dataset (SciTechNews), with tuples of a publicly-available scientific paper, its corresponding news article, and an expert-written short summary snippet; 2) proposing a novel technical framework that integrates a paper’s discourse structure with its metadata to guide generation; and, 3) demonstrating with extensive automatic and human experiments that our model outperforms other baseline methods (eg Alpaca and ChatGPT) in elaborating a content plan meaningful for the target audience, simplify the information selected, and produce a coherent final report in a layman’s style.
Cited by:8
Resources:[PDF] from aclanthology.org
:view less
LOWRECORP: the Low-Resource NLG Corpus Building Challenge
2023
Khyathi Raghavi Chandu, David M Howcroft, Dimitra Gkatzia, Yi-Ling Chung, Yufang Hou, C…
Proceedings of the 16th International Natural Language Generation Conference: Generatio…
:mehr lesen
Authors:Khyathi Raghavi Chandu, David M Howcroft, Dimitra Gkatzia, Yi-Ling Chung, Yufang Hou, Chris Chinenye Emezue, Pawan Rajpoot, Tosin Adewumi
Publication date:09/01/2023
Journal:Proceedings of the 16th International Natural Language Generation Conference: Generation Challenges
Pages:1-9
Description:Most languages in the world do not have sufficient data available to develop neural-network-based natural language generation (NLG) systems. To alleviate this resource scarcity, we propose a novel challenge for the NLG community: low-resource language corpus development (LOWRECORP). We present an innovative framework to collect a single dataset with dual tasks to maximize the efficiency of data collection efforts and respect language consultant time. Specifically, we focus on a text-chat-based interface for two generation tasks–conversational response generation grounded in a source document and/or image and dialogue summarization (from the former task). The goal of this shared task is to collectively develop grounded datasets for local and low-resourced languages. To enable data collection, we make available web-based software that can be used to collect these grounded conversations and summaries. Submissions will be assessed for the size, complexity, and diversity of the corpora to ensure quality control of the datasets as well as any enhancements to the interface or novel approaches to grounding conversations.
Cited by:1
Resources:[PDF] from aclanthology.org
:view less
Bridging Resolution: A Journey Towards Modelling Referential Discourse Entities
2023
Yufang Hou
:mehr lesen
Authors:Yufang Hou
Publication date:07/09/2023
Description:Information status and anaphora play an important role in text cohesion and language understanding. In this talk, I will first give an overview of my decade-long work on bridging resolution and information status classification. Next, I will discuss our research findings on probing LLMs for bridging inference. Finally, I will outline several challenging research questions on modelling referential discourse entities.
:view less
SustaiNLP 2023: Fourth Workshop on Simple and Efficient Natural Language Processing
2023
Nafise Sadat Moosavi, Iryna Gurevych, Yufang Hou, Gyuwan Kim, Young Jin Kim, Tal Schust…
:mehr lesen
Authors:Nafise Sadat Moosavi, Iryna Gurevych, Yufang Hou, Gyuwan Kim, Young Jin Kim, Tal Schuster, Ameeta Agrawal
Publication date:07/09/2023
Description:The Natural Language Processing (NLP) community has, in recent years, demonstrated a notable focus on improving higher scores on standard benchmarks and taking the lead on community-wide leaderboards (eg, GLUE, SentEval). While this aspiration has led to improvements in benchmark performance of (predominantly neural) models, it has also came at a cost, ie, increased model complexity and the ever-growing amount of computational resources required for training and using the current state-of-the-art models. Moreover, the recent research efforts have, for the most part, failed to identify sources of empirical gains in models, often failing to empirically justify the model complexity beyond benchmark performance Because of these easily observable trends, we have proposed the SustaiNLP workshop with the goal of promoting more sustainable NLP research and practices, with two main objectives:(1 …
:view less
Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP)
2023
Nafise Sadat Moosavi, Iryna Gurevych, Yufang Hou, Gyuwan Kim, Young Jin Kim, Tal Schust…
Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing …
:mehr lesen
Authors:Nafise Sadat Moosavi, Iryna Gurevych, Yufang Hou, Gyuwan Kim, Young Jin Kim, Tal Schuster, Ameeta Agrawal
Publication date:07/01/2023
Journal:Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP)
Description:It is our great pleasure to welcome you to the fourth edition of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing. The Natural Language Processing community has, in recent years, demonstrated a notable focus on improving higher scores on standard benchmarks and taking the lead on community-wide leaderboards (eg, GLUE, SentEval). While this aspiration has led to improvements in benchmark performance of (predominantly neural) models, it has also came at a cost, ie, increased model complexity and the evergrowing amount of computational resources required for training and using the current state-of-the-art models. Moreover, the recent research efforts have, for the most part, failed to identify sources of empirical gains in models, often failing to empirically justify the model complexity beyond benchmark performance.
Resources:[PDF] from aclanthology.org
:view less
A Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization
2023
Lining Zhang, Simon Mille, Yufang Hou, Daniel Deutsch, Elizabeth Clark, Yixin Liu, Saad…
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics…
:mehr lesen
Authors:Lining Zhang, Simon Mille, Yufang Hou, Daniel Deutsch, Elizabeth Clark, Yixin Liu, Saad Mahamood, Sebastian Gehrmann, Miruna Clinciu, Khyathi Raghavi Chandu, João Sedoc
Publication date:07/01/2023
Journal:Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Pages:14944-14982
Description:To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization. Thus, we investigate the recruitment of high-quality Amazon Mechanical Turk workers via a two-step pipeline. We show that we can successfully filter out subpar workers before they carry out the evaluations and obtain high-agreement annotations with similar constraints on resources. Although our workers demonstrate a strong consensus among themselves and CloudResearch workers, their alignment with expert judgments on a subset of the data is not as expected and needs further training in correctness. This paper still serves as a best practice for the recruitment of qualified annotators in other challenging annotation tasks.
Cited by:14
Resources:[PDF] from arxiv.org
:view less
PairSpanBERT: An Enhanced Language Model for Bridging Resolution
2023
Hideo Kobayashi, Yufang Hou, Vincent Ng
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics…
:mehr lesen
Authors:Hideo Kobayashi, Yufang Hou, Vincent Ng
Publication date:07/01/2023
Journal:Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Pages:6931-6946
Description:We present PairSpanBERT, a SpanBERT-based pre-trained model specialized for bridging resolution. To this end, we design a novel pre-training objective that aims to learn the contexts in which two mentions are implicitly linked to each other from a large amount of data automatically generated either heuristically or via distance supervision with a knowledge graph. Despite the noise inherent in the automatically generated data, we achieve the best results reported to date on three evaluation datasets for bridging resolution when replacing SpanBERT with PairSpanBERT in a state-of-the-art resolver that jointly performs entity coreference resolution and bridging resolution.
Cited by:1
Resources:[PDF] from aclanthology.org
:view less
Matching Pairs: Attributing Fine-Tuned Models to their Pre-Trained Large Language Models
2023
Myles Foley, Ambrish Rawat, Taesung Lee, Yufang Hou, Gabriele Picco, Giulio Zizzo
ACL 2023
:mehr lesen
Authors:Myles Foley, Ambrish Rawat, Taesung Lee, Yufang Hou, Gabriele Picco, Giulio Zizzo
Publication date:06/15/2023
Journal:ACL 2023
Description:The wide applicability and adaptability of generative large language models (LLMs) has enabled their rapid adoption. While the pre-trained models can perform many tasks, such models are often fine-tuned to improve their performance on various downstream applications. However, this leads to issues over violation of model licenses, model theft, and copyright infringement. Moreover, recent advances show that generative technology is capable of producing harmful content which exacerbates the problems of accountability within model supply chains. Thus, we need a method to investigate how a model was trained or a piece of text was generated and what their pre-trained base model was. In this paper we take the first step to address this open problem by tracing back the origin of a given fine-tuned LLM to its corresponding pre-trained base model. We consider different knowledge levels and attribution strategies, and find that we can correctly trace back 8 out of the 10 fine tuned models with our best method.
Cited by:9
Resources:[PDF] from arxiv.org
:view less
Are Fairy Tales Fair? Analyzing Gender Bias in Temporal Narrative Event Chains of Children’s Fairy Tales
2023
Paulina Toro Isaza, Guangxuan Xu, Akintoye Oloko, Yufang Hou, Nanyun Peng, Dakuo Wang
ACL 2023
:mehr lesen
Authors:Paulina Toro Isaza, Guangxuan Xu, Akintoye Oloko, Yufang Hou, Nanyun Peng, Dakuo Wang
Publication date:05/26/2023
Journal:ACL 2023
Description:Social biases and stereotypes are embedded in our culture in part through their presence in our stories, as evidenced by the rich history of humanities and social science literature analyzing such biases in children stories. Because these analyses are often conducted manually and at a small scale, such investigations can benefit from the use of more recent natural language processing methods that examine social bias in models and data corpora. Our work joins this interdisciplinary effort and makes a unique contribution by taking into account the event narrative structures when analyzing the social bias of stories. We propose a computational pipeline that automatically extracts a story's temporal narrative verb-based event chain for each of its characters as well as character attributes such as gender. We also present a verb-based event annotation scheme that can facilitate bias analysis by including categories such as those that align with traditional stereotypes. Through a case study analyzing gender bias in fairy tales, we demonstrate that our framework can reveal bias in not only the unigram verb-based events in which female and male characters participate but also in the temporal narrative order of such event participation.
Cited by:12
Resources:[PDF] from arxiv.org
:view less
CiteBench: A benchmark for Scientific Citation Text Generation
2022
Martin Funkquist, Ilia Kuznetsov, Yufang Hou, Iryna Gurevych
EMNLP 2023
:mehr lesen
Authors:Martin Funkquist, Ilia Kuznetsov, Yufang Hou, Iryna Gurevych
Publication date:12/19/2022
Journal:EMNLP 2023
Description:Science progresses by building upon the prior body of knowledge documented in scientific publications. The acceleration of research makes it hard to stay up-to-date with the recent developments and to summarize the ever-growing body of prior work. To address this, the task of citation text generation aims to produce accurate textual summaries given a set of papers-to-cite and the citing paper context. Due to otherwise rare explicit anchoring of cited documents in the citing paper, citation text generation provides an excellent opportunity to study how humans aggregate and synthesize textual knowledge from sources. Yet, existing studies are based upon widely diverging task definitions, which makes it hard to study this task systematically. To address this challenge, we propose CiteBench: a benchmark for citation text generation that unifies multiple diverse datasets and enables standardized evaluation of citation text generation models across task designs and domains. Using the new benchmark, we investigate the performance of multiple strong baselines, test their transferability between the datasets, and deliver new insights into the task definition and evaluation to guide future research in citation text generation. We make the code for CiteBench publicly available at https://github.com/UKPLab/citebench.
Cited by:10
Resources:[PDF] from arxiv.org
:view less
Proceedings of The Third Workshop on Simple and Efficient Natural Language Processing (SustaiNLP)
2022
Angela Fan, Iryna Gurevych, Yufang Hou, Zornitsa Kozareva, Sasha Luccioni, Nafise Sadat…
Proceedings of The Third Workshop on Simple and Efficient Natural Language Processing (…
:mehr lesen
Authors:Angela Fan, Iryna Gurevych, Yufang Hou, Zornitsa Kozareva, Sasha Luccioni, Nafise Sadat Moosavi, Sujith Ravi, Gyuwan Kim, Roy Schwartz, Andreas Rücklé
Publication date:12/01/2022
Journal:Proceedings of The Third Workshop on Simple and Efficient Natural Language Processing (SustaiNLP)
Description:It is our great pleasure to welcome you to the third edition of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing. The Natural Language Processing community has, in recent years, demonstrated a notable focus on improving higher scores on standard benchmarks and taking the lead on community-wide leaderboards (eg, GLUE, SentEval). While this aspiration has led to improvements in benchmark performance of (predominantly neural) models, it has also came at a cost, ie, increased model complexity and the evergrowing amount of computational resources required for training and using the current state-of-the-art models. Moreover, the recent research efforts have, for the most part, failed to identify sources of empirical gains in models, often failing to empirically justify the model complexity beyond benchmark performance.
Cited by:1
Resources:[PDF] from aclanthology.org
:view less
Envisioning a Human-AI collaborative system to transform policies into decision models
2022
Vanessa Lopez, Gabriele Picco, Inge Vejsbjerg, Thanh Lam Hoang, Yufang Hou, Marco Luca …
arXiv preprint arXiv:2212.06882
:mehr lesen
Authors:Vanessa Lopez, Gabriele Picco, Inge Vejsbjerg, Thanh Lam Hoang, Yufang Hou, Marco Luca Sbodio, John Segrave-Daly, Denisa Moga, Sean Swords, Miao Wei, Eoin Carroll
Publication date:11/01/2022
Journal:arXiv preprint arXiv:2212.06882
Description:Regulations govern many aspects of citizens' daily lives. Governments and businesses routinely automate these in the form of coded rules (e.g., to check a citizen's eligibility for specific benefits). However, the path to automation is long and challenging. To address this, recent global initiatives for digital government, proposing to simultaneously express policy in natural language for human consumption as well as computationally amenable rules or code, are gathering broad public-sector interest. We introduce the problem of semi-automatically building decision models from eligibility policies for social services, and present an initial emerging approach to shorten the route from policy documents to executable, interpretable and standardised decision models using AI, NLP and Knowledge Graphs. Despite the many open domain challenges, in this position paper we explore the enormous potential of AI to assist government agencies and policy experts in scaling the production of both human-readable and machine executable policy rules, while improving transparency, interpretability, traceability and accountability of the decision making.
Resources:[PDF] from arxiv.org
:view less
Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation
2022
Max Glockner, Yufang Hou, Iryna Gurevych
EMNLP 2022
:mehr lesen
Authors:Max Glockner, Yufang Hou, Iryna Gurevych
Publication date:10/25/2022
Journal:EMNLP 2022
Description:Misinformation emerges in times of uncertainty when credible information is limited. This is challenging for NLP-based fact-checking as it relies on counter-evidence, which may not yet be available. Despite increasing interest in automatic fact-checking, it is still unclear if automated approaches can realistically refute harmful real-world misinformation. Here, we contrast and compare NLP fact-checking with how professional fact-checkers combat misinformation in the absence of counter-evidence. In our analysis, we show that, by design, existing NLP task definitions for fact-checking cannot refute misinformation as professional fact-checkers do for the majority of claims. We then define two requirements that the evidence in datasets must fulfill for realistic fact-checking: It must be (1) sufficient to refute the claim and (2) not leaked from existing fact-checking articles. We survey existing fact-checking datasets and find that all of them fail to satisfy both criteria. Finally, we perform experiments to demonstrate that models trained on a large-scale fact-checking dataset rely on leaked evidence, which makes them unsuitable in real-world scenarios. Taken together, we show that current NLP fact-checking cannot realistically combat real-world misinformation because it depends on unrealistic assumptions about counter-evidence in the data.
Cited by:45
Resources:[PDF] from arxiv.org
:view less
End-to-end neural bridging resolution
2022
Hideo Kobayashi, Yufang Hou, Vincent Ng
Proceedings of the 29th International Conference on Computational Linguistics, 766-778
:mehr lesen
Authors:Hideo Kobayashi, Yufang Hou, Vincent Ng
Publication date:10/01/2022
Journal:Proceedings of the 29th International Conference on Computational Linguistics
Pages:766-778
Description:The state of bridging resolution research is rather unsatisfactory: not only are state-of-the-art resolvers evaluated in unrealistic settings, but the neural models underlying these resolvers are weaker than those used for entity coreference resolution. In light of these problems, we evaluate bridging resolvers in an end-to-end setting, strengthen them with better encoders, and attempt to gain a better understanding of them via perturbation experiments and a manual analysis of their outputs.
Cited by:4
Resources:[PDF] from aclanthology.org
:view less
Nece: Narrative event chain extraction toolkit
2022
Guangxuan Xu, Paulina Toro Isaza, Moshi Li, Akintoye Oloko, Bingsheng Yao, Cassia Sanct…
arXiv preprint arXiv:2208.08063
:mehr lesen
Authors:Guangxuan Xu, Paulina Toro Isaza, Moshi Li, Akintoye Oloko, Bingsheng Yao, Cassia Sanctos, Aminat Adebiyi, Yufang Hou, Nanyun Peng, Dakuo Wang
Publication date:08/17/2022
Journal:arXiv preprint arXiv:2208.08063
Description:To understand a narrative, it is essential to comprehend the temporal event flows, especially those associated with main characters; however, this can be challenging with lengthy and unstructured narrative texts. To address this, we introduce NECE, an open-access, document-level toolkit that automatically extracts and aligns narrative events in the temporal order of their occurrence. Through extensive evaluations, we show the high quality of the NECE toolkit and demonstrates its downstream application in analyzing narrative bias regarding gender. We also openly discuss the shortcomings of the current approach, and potential of leveraging generative models in future works. Lastly the NECE toolkit includes both a Python library and a user-friendly web interface, which offer equal access to professionals and layman audience alike, to visualize event chain, obtain narrative flows, or study narrative bias.
Cited by:2
Resources:[PDF] from arxiv.org
:view less
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
2022
Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papa…
EMNLP 2022 (Demo)
:mehr lesen
Authors:Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina McMillan-Major, Anna Shvets, Ashish Upadhyay, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hendrik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou, Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Leonardo FR Ribeiro, Lewis Tunstall, Li Zhang, Mahima Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi, Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood, Salomey Osei
Publication date:06/22/2022
Journal:EMNLP 2022 (Demo)
Description:Evaluation in machine learning is usually informed by past choices, for example which datasets or metrics to use. This standardization enables the comparison on equal footing using leaderboards, but the evaluation choices become sub-optimal as better alternatives arise. This problem is especially pertinent in natural language generation which requires ever-improving suites of datasets, metrics, and human evaluation to make definitive claims. To make following best model evaluation practices easier, we introduce GEMv2. The new version of the Generation, Evaluation, and Metrics Benchmark introduces a modular infrastructure for dataset, model, and metric developers to benefit from each others work. GEMv2 supports 40 documented datasets in 51 languages. Models for all datasets can be evaluated online and our interactive data card creation and rendering tools make it easier to add new datasets to the living benchmark.
Cited by:17
Resources:[PDF] from arxiv.org
:view less
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
2022
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Ada…
Transactions on Machine Learning Research
:mehr lesen
Authors:Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Rahane, Anantharaman S Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein
Publication date:06/09/2022
Journal:Transactions on Machine Learning Research
Description:Language models demonstrate both quantitative improvement and new qualitative capabilities with increasing scale. Despite their potentially transformative impact, these new capabilities are as yet poorly characterized. In order to inform future research, prepare for disruptive new model capabilities, and ameliorate socially harmful effects, it is vital that we understand the present and near-future capabilities and limitations of language models. To address this challenge, we introduce the Beyond the Imitation Game benchmark (BIG-bench). BIG-bench currently consists of 204 tasks, contributed by 450 authors across 132 institutions. Task topics are diverse, drawing problems from linguistics, childhood development, math, common-sense reasoning, biology, physics, social bias, software development, and beyond. BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models. We evaluate the behavior of OpenAI's GPT models, Google-internal dense transformer architectures, and Switch-style sparse transformers on BIG-bench, across model sizes spanning millions to hundreds of billions of parameters. In addition, a team of human expert raters performed all tasks in order to provide a strong baseline. Findings include: model performance and calibration both improve with scale, but are poor in absolute terms (and when compared with rater performance); performance is remarkably similar across model classes, though with benefits from sparsity; tasks that improve gradually and predictably commonly involve a large knowledge or memorization component, whereas tasks that exhibit "breakthrough" behavior at …
Cited by:1513
Resources:[PDF] from arxiv.org
:view less
Finding sub-task structure with natural language instruction
2022
Ryokan Ri, Yufang Hou, Radu Marinescu, Akihiro Kishimoto
Proceedings of the First Workshop on Learning with Natural Language Supervision, 1-9
:mehr lesen
Authors:Ryokan Ri, Yufang Hou, Radu Marinescu, Akihiro Kishimoto
Publication date:05/01/2022
Journal:Proceedings of the First Workshop on Learning with Natural Language Supervision
Pages:1-9
Description:When mapping a natural language instruction to a sequence of actions, it is often useful toidentify sub-tasks in the instruction. Such sub-task segmentation, however, is not necessarily provided in the training data. We present the A2LCTC (Action-to-Language Connectionist Temporal Classification) algorithm to automatically discover a sub-task segmentation of an action sequence. A2LCTC does not require annotations of correct sub-task segments and learns to find them from pairs of instruction and action sequence in a weakly-supervised manner. We experiment with the ALFRED dataset and show that A2LCTC accurately finds the sub-task structures. With the discovered sub-tasks segments, we also train agents that work on the downstream task and empirically show that our algorithm improves the performance.
Cited by:2
Resources:[PDF] from aclanthology.org
:view less
Constrained Multi-Task Learning for Bridging Resolution
2022
Hideo Kobayashi, Yufang Hou, Vincent Ng
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics…
:mehr lesen
Authors:Hideo Kobayashi, Yufang Hou, Vincent Ng
Publication date:05/01/2022
Journal:Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Pages:759-770
Description:We examine the extent to which supervised bridging resolvers can be improved without employing additional labeled bridging data by proposing a novel constrained multi-task learning framework for bridging resolution, within which we (1) design cross-task consistency constraints to guide the learning process;(2) pre-train the entity coreference model in the multi-task framework on the large amount of publicly available coreference data; and (3) integrating prior knowledge encoded in rule-based resolvers. Our approach achieves state-of-the-art results on three standard evaluation corpora.
Cited by:8
Resources:[PDF] from aclanthology.org
:view less
Educational Question Generation of Children Storybooks via Question Type Distribution Learning and Event-Centric Summarization
2022
Zhenjie Zhao, Yufang Hou, Dakuo Wang, Mo Yu, Chengzhong Liu, Xiaojuan Ma
ACL 2022
:mehr lesen
Authors:Zhenjie Zhao, Yufang Hou, Dakuo Wang, Mo Yu, Chengzhong Liu, Xiaojuan Ma
Publication date:03/27/2022
Journal:ACL 2022
Description:Generating educational questions of fairytales or storybooks is vital for improving children's literacy ability. However, it is challenging to generate questions that capture the interesting aspects of a fairytale story with educational meaningfulness. In this paper, we propose a novel question generation method that first learns the question type distribution of an input story paragraph, and then summarizes salient events which can be used to generate high-cognitive-demand questions. To train the event-centric summarizer, we finetune a pre-trained transformer-based sequence-to-sequence model using silver samples composed by educational question-answer pairs. On a newly proposed educational question answering dataset FairytaleQA, we show good performance of our method on both automatic and human evaluation metrics. Our work indicates the necessity of decomposing question type distribution learning and event-centric summary generation for educational question generation.
Cited by:48
Resources:[PDF] from arxiv.org
:view less
Report on the 1st workshop on argumentation knowledge graphs (ArgKG 2021) at AKBC 2021
2022
Tirthankar Ghosal, Khalid Al-Khatib, Yufang Hou, Anita de Waard, Dayne Freitag
ACM SIGIR Forum 55 (2), 1-12
:mehr lesen
Authors:Tirthankar Ghosal, Khalid Al-Khatib, Yufang Hou, Anita de Waard, Dayne Freitag
Publication date:03/17/2022
Journal:ACM SIGIR Forum
Volume:55
Issue:2
Pages:1-12
Publisher:ACM
Description:The first workshop on Argumentation Knowledge Graphs (ArgKG) was held virtually at the Automated Knowledge Base Construction (AKBC 2021) conference on October 7, 2021. ArgKG @ AKBC 2021 brought together the Computational Argumentation and Knowledge Graphs communities, aiming to promote cross-pollination of ideas and encourage discussions and collaborations between the two communities. This paper describes the workshop and compiles several of its findings and insights. Date: 7 October, 2021. Website: https://argkg21.argmining.org.
Cited by:1
Resources:[PDF] from rug.nl
:view less
Privacy-aware supervised classification: An informative subspace based multi-objective approach
2022
Chandan Biswas, Debasis Ganguly, Partha Sarathi Mukherjee, Ujjwal Bhattacharya, Yufang Hou
Pattern Recognition 122, 108301
:mehr lesen
Authors:Chandan Biswas, Debasis Ganguly, Partha Sarathi Mukherjee, Ujjwal Bhattacharya, Yufang Hou
Publication date:02/01/2022
Journal:Pattern Recognition
Volume:122
Pages:108301
Publisher:Pergamon
Description:Sharing the raw or an abstract representation of a labelled dataset on cloud platforms can potentially expose sensitive information of the data to an adversary, e.g., in the case of an emotion classification task from text, an adversary-agnostic abstract representation of the text data may eventually lead an adversary to identify the demographics of the authors, such as their gender and age. In this paper, we propose a universal defense mechanism against such malicious attempts of stealing sensitive information from data shared on cloud platforms. More specifically, our proposed method employs an informative subspace based multi-objective approach to obtain a sensitive information aware encoding of the data representation. A number of experiments conducted on both standard text and image datasets demonstrate that our proposed approach is able to reduce the effectiveness of the adversarial task (i.e., in other …
Cited by:14
Resources:[PDF] from gla.ac.uk
:view less
Ensembling Graph Predictions for AMR Parsing
2021
Thanh Lam Hoang, Gabriele Picco, Yufang Hou, Young-Suk Lee, Lam Nguyen, Dzung Phan, Van…
Advances in Neural Information Processing Systems 34, 8495-8505
:mehr lesen
Authors:Thanh Lam Hoang, Gabriele Picco, Yufang Hou, Young-Suk Lee, Lam Nguyen, Dzung Phan, Vanessa López, Ramon Fernandez Astudillo
Publication date:12/06/2021
Journal:Advances in Neural Information Processing Systems
Volume:34
Pages:8495-8505
Description:In many machine learning tasks, models are trained to predict structure data such as graphs. For example, in natural language processing, it is very common to parse texts into dependency trees or abstract meaning representation (AMR) graphs. On the other hand, ensemble methods combine predictions from multiple models to create a new one that is more robust and accurate than individual predictions. In the literature, there are many ensembling techniques proposed for classification or regression problems, however, ensemble graph prediction has not been studied thoroughly. In this work, we formalize this problem as mining the largest graph that is the most supported by a collection of graph predictions. As the problem is NP-Hard, we propose an efficient heuristic algorithm to approximate the optimal solution. To validate our approach, we carried out experiments in AMR parsing problems. The experimental results demonstrate that the proposed approach can combine the strength of state-of-the-art AMR parsers to create new predictions that are more accurate than any individual models in five standard benchmark datasets.
Cited by:31
Resources:[PDF] from neurips.cc
:view less
Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing
2021
Nafise Sadat Moosavi, Iryna Gurevych, Angela Fan, Thomas Wolf, Yufang Hou, Ana Marasovi…
Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing
:mehr lesen
Authors:Nafise Sadat Moosavi, Iryna Gurevych, Angela Fan, Thomas Wolf, Yufang Hou, Ana Marasović, Sujith Ravi
Publication date:11/01/2021
Journal:Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing
Description:It is our great pleasure to welcome you to the second edition of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing. The Natural Language Processing community has, in recent years, demonstrated a notable focus on improving higher scores on standard benchmarks and taking the lead on community-wide leaderboards (eg, GLUE, SentEval). While this aspiration has led to improvements in benchmark performance of (predominantly neural) models, it has also came at a cost, ie, increased model complexity and the evergrowing amount of computational resources required for training and using the current state-of-theart models. Moreover, the recent research efforts have, for the most part, failed to identify sources of empirical gains in models, often failing to empirically justify the model complexity beyond benchmark performance.
Resources:[PDF] from aclanthology.org
:view less
Proceedings of the 8th Workshop on Argument Mining
2021
Khalid Al Khatib, Yufang Hou, Manfred Stede
Proceedings of the 8th Workshop on Argument Mining
:mehr lesen
Authors:Khalid Al Khatib, Yufang Hou, Manfred Stede
Publication date:11/01/2021
Journal:Proceedings of the 8th Workshop on Argument Mining
Description:Argument mining (also known as “argumentation mining”) is a young and gradually maturing research area within computational linguistics. At its heart, argument mining involves the automatic identification of argumentative structures in free text, such as the conclusions, premises, and inference schemes of arguments as well as their interrelations and counter-considerations. To date, researchers have investigated argument mining on genres such as legal documents, product reviews, news articles, online debates, user-generated web discourse, Wikipedia articles, academic literature, persuasive essays, tweets, and dialogues. Recently, also argument quality assessment and generation came into focus. In addition, argument mining is inherently tied to stance and sentiment analysis, since every argument carries a stance towards its topic, often expressed with sentiment. Argument mining gives rise to various practical applications of great importance. In particular, it provides methods that can find and visualize the main pro and con arguments in a text corpus—or even on in an argument search on the web—towards a topic or query of interest. In instructional contexts, written and diagrammed arguments represent educational data that can be mined for conveying and assessing students’ command of course material. In information retrieval, argument mining is expected to play a salient role in the emerging field of conversational search. And with the IBM Debater Project, technology based on argument mining recently received a lot of media attention.
Cited by:1
Resources:[PDF] from aclanthology.org
:view less
HAConvGNN: Hierarchical Attention Based Convolutional Graph Neural Network for Code Documentation Generation in Jupyter Notebooks
2021
Xuye Liu, Dakuo Wang, April Wang, Yufang Hou, Lingfei Wu
:mehr lesen
Authors:Xuye Liu, Dakuo Wang, April Wang, Yufang Hou, Lingfei Wu
Publication date:09/09/2021
Description:Jupyter notebook allows data scientists to write machine learning code together with its documentation in cells. In this paper, we propose a new task of code documentation generation (CDG) for computational notebooks. In contrast to the previous CDG tasks which focus on generating documentation for single code snippets, in a computational notebook, one documentation in a markdown cell often corresponds to multiple code cells, and these code cells have an inherent structure. We proposed a new model (HAConvGNN) that uses a hierarchical attention mechanism to consider the relevant code cells and the relevant code tokens information when generating the documentation. Tested on a new corpus constructed from well-documented Kaggle notebooks, we show that our model outperforms other baseline models.
Cited by:25
Resources:[PDF] from arxiv.org
:view less
End-to-end Neural Information Status Classification
2021
Yufang Hou
EMNLP Findings 2021
:mehr lesen
Authors:Yufang Hou
Publication date:09/06/2021
Journal:EMNLP Findings 2021
Description:Most previous studies on information status (IS) classification and bridging anaphora recognition assume that the gold mention or syntactic tree information is given (Hou et al., 2013; Roesiger et al., 2018; Hou, 2020; Yu and Poesio, 2020). In this paper, we propose an end-to-end neural approach for information status classification. Our approach consists of a mention extraction component and an information status assignment component. During the inference time, our system takes a raw text as the input and generates mentions together with their information status. On the ISNotes corpus (Markert et al., 2012), we show that our information status assignment component achieves new state-of-the-art results on fine-grained IS classification based on gold mentions. Furthermore, our system performs significantly better than other baselines for both mention extraction and fine-grained IS classification in the end-to-end setting. Finally, we apply our system on BASHI (Roesiger, 2018) and SciCorp (Roesiger, 2016) to recognize referential bridging anaphora. We find that our end-to-end system trained on ISNotes achieves competitive results on bridging anaphora recognition compared to the previous state-of-the-art system that relies on syntactic information and is trained on the in-domain datasets (Yu and Poesio, 2020).
Cited by:8
Resources:[PDF] from arxiv.org
:view less
End-to-End Construction of NLP Knowledge Graph
2021
Ishani Mondal, Yufang Hou, Charles Jochim
Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 1885-1895
:mehr lesen
Authors:Ishani Mondal, Yufang Hou, Charles Jochim
Publication date:08/01/2021
Journal:Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021
Pages:1885-1895
Description:This paper studies the end-to-end construction of an NLP Knowledge Graph (KG) from scientific papers. We focus on extracting four types of relations: evaluatedOn between tasks and datasets, evaluatedBy between tasks and evaluation metrics, as well as coreferent and related relations between the same type of entities. For instance,“F1 score” is coreferent with “F-measure”. We introduce novel methods for each of these relation types and apply our final framework (SciNLP-KG) to 30,000 NLP papers from ACL Anthology to build a large-scale KG, which can facilitate automatically constructing scientific leaderboards for the NLP community. The results of our experiments indicate that the resulting KG contains high-quality information.
Cited by:62
Resources:[PDF] from aclanthology.org
:view less
Argument Mining for Scholarly Document Processing: Taking Stock and Looking Ahead
2021
Khalid Al Khatib, Tirthankar Ghosal, Yufang Hou, Anita de Waard, Dayne Freitag
Proceedings of the Second Workshop on Scholarly Document Processing, 56-65
:mehr lesen
Authors:Khalid Al Khatib, Tirthankar Ghosal, Yufang Hou, Anita de Waard, Dayne Freitag
Publication date:06/01/2021
Journal:Proceedings of the Second Workshop on Scholarly Document Processing
Pages:56-65
Description:Argument mining targets structures in natural language related to interpretation and persuasion which are central to scientific communication. Most scholarly discourse involves interpreting experimental evidence and attempting to persuade other scientists to adopt the same conclusions. While various argument mining studies have addressed student essays and news articles, those that target scientific discourse are still scarce. This paper surveys existing work in argument mining of scholarly discourse, and provides an overview of current models, data, tasks, and applications. We identify a number of key challenges confronting argument mining in the scientific domain, and suggest some possible solutions and future directions.
Cited by:26
Resources:[PDF] from aclanthology.org
:view less
D2S: Document-to-Slide Generation Via Query-Based Text Summarization
2021
Edward Sun, Yufang Hou, Dakuo Wang, Yunfeng Zhang, Nancy XR Wang
:mehr lesen
Authors:Edward Sun, Yufang Hou, Dakuo Wang, Yunfeng Zhang, Nancy XR Wang
Publication date:05/08/2021
Description:Presentations are critical for communication in all areas of our lives, yet the creation of slide decks is often tedious and time-consuming. There has been limited research aiming to automate the document-to-slides generation process and all face a critical challenge: no publicly available dataset for training and benchmarking. In this work, we first contribute a new dataset, SciDuet, consisting of pairs of papers and their corresponding slides decks from recent years' NLP and ML conferences (e.g., ACL). Secondly, we present D2S, a novel system that tackles the document-to-slides task with a two-step approach: 1) Use slide titles to retrieve relevant and engaging text, figures, and tables; 2) Summarize the retrieved context into bullet points with long-form question answering. Our evaluation suggests that long-form QA outperforms state-of-the-art summarization baselines on both automated ROUGE metrics and qualitative human evaluation.
Cited by:45
Resources:[PDF] from arxiv.org
:view less
Probing for Bridging Inference in Transformer Language Models
2021
Onkar Pandit, Yufang Hou
:mehr lesen
Authors:Onkar Pandit, Yufang Hou
Publication date:04/19/2021
Cited by:22
:view less
An autonomous debating system
2021
Noam Slonim, Yonatan Bilu, Carlos Alzate, Roy Bar-Haim, Ben Bogin, Francesca Bonin, Les…
Nature 591 (7850), 379-384
:mehr lesen
Authors:Noam Slonim, Yonatan Bilu, Carlos Alzate, Roy Bar-Haim, Ben Bogin, Francesca Bonin, Leshem Choshen, Edo Cohen-Karlik, Lena Dankin, Lilach Edelstein, Liat Ein-Dor, Roni Friedman-Melamed, Assaf Gavron, Ariel Gera, Martin Gleize, Shai Gretz, Dan Gutfreund, Alon Halfon, Daniel Hershcovich, Ron Hoory, Yufang Hou, Shay Hummel, Michal Jacovi, Charles Jochim, Yoav Kantor, Yoav Katz, David Konopnicki, Zvi Kons, Lili Kotlerman, Dalia Krieger, Dan Lahav, Tamar Lavee, Ran Levy, Naftali Liberman, Yosi Mass, Amir Menczel, Shachar Mirkin, Guy Moshkowich, Shila Ofek-Koifman, Matan Orbach, Ella Rabinovich, Ruty Rinott, Slava Shechtman, Dafna Sheinwald, Eyal Shnarch, Ilya Shnayderman, Aya Soffer, Artem Spector, Benjamin Sznajder, Assaf Toledo, Orith Toledo-Ronen, Elad Venezian, Ranit Aharonov
Publication date:03/01/2021
Journal:Nature
Volume:591
Issue:7850
Pages:379-384
Publisher:Nature Publishing Group
Description:Artificial intelligence (AI) is defined as the ability of machines to perform tasks that are usually associated with intelligent beings. Argument and debate are fundamental capabilities of human intelligence, essential for a wide range of human activities, and common to all human societies. The development of computational argumentation technologies is therefore an important emerging discipline in AI research. Here we present Project Debater, an autonomous debating system that can engage in a competitive debate with humans. We provide a complete description of the system’s architecture, a thorough and systematic evaluation of its operation across a wide range of debate topics, and a detailed account of the system’s performance in its public debut against three expert human debaters. We also highlight the fundamental differences between debating with humans as opposed to challenging humans in game …
Cited by:251
:view less
The gem benchmark: Natural language generation, its evaluation and metrics
2021
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu…
Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics…
:mehr lesen
Authors:Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Rubungo Andre Niyongabo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez, Sashank Santhanam, João Sedoc, Thibault Sellam, Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Yerukola
Publication date:02/02/2021
Journal:Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021)
Description:We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for which we are organizing a shared task at our ACL 2021 Workshop and to which we invite the entire NLG community to participate.
Cited by:178
Resources:[PDF] from arxiv.org
:view less
TDMSci: A Specialized Corpus for Scientific Literature Entity Tagging of Tasks Datasets and Metrics
2021
Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, Debasis Ganguly
:mehr lesen
Authors:Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, Debasis Ganguly
Publication date:01/25/2021
Description:Tasks, Datasets and Evaluation Metrics are important concepts for understanding experimental scientific papers. However, most previous work on information extraction for scientific literature mainly focuses on the abstracts only, and does not treat datasets as a separate type of entity (Zadeh and Schumann, 2016; Luan et al., 2018). In this paper, we present a new corpus that contains domain expert annotations for Task (T), Dataset (D), Metric (M) entities on 2,000 sentences extracted from NLP papers. We report experiment results on TDM extraction using a simple data augmentation strategy and apply our tagger to around 30,000 NLP papers from the ACL Anthology. The corpus is made publicly available to the community for fostering research on scientific publication summarization (Erera et al., 2019) and knowledge discovery.
Cited by:46
Resources:[PDF] from arxiv.org
:view less
Fine-grained Information Status Classification Using Discourse Context-Aware BERT
2020
Yufang Hou
COLING 2020
:mehr lesen
Authors:Yufang Hou
Publication date:10/26/2020
Journal:COLING 2020
Description:Previous work on bridging anaphora recognition (Hou et al., 2013a) casts the problem as a subtask of learning fine-grained information status (IS). However, these systems heavily depend on many hand-crafted linguistic features. In this paper, we propose a simple discourse context-aware BERT model for fine-grained IS classification. On the ISNotes corpus (Markert et al., 2012), our model achieves new state-of-the-art performance on fine-grained IS classification, obtaining a 4.8 absolute overall accuracy improvement compared to Hou et al. (2013a). More importantly, we also show an improvement of 10.5 F1 points for bridging anaphora recognition without using any complex hand-crafted semantic features designed for capturing the bridging phenomenon. We further analyze the trained model and find that the most attended signals for each IS category correspond well to linguistic notions of information status.
Cited by:8
Resources:[PDF] from arxiv.org
:view less
Behavioural Science
2020
Candice Moore, Yufang Hou, Francesca Bonin, Robert West, Susan Michie, John Shawe-Taylo…
:mehr lesen
Authors:Candice Moore, Yufang Hou, Francesca Bonin, Robert West, Susan Michie, John Shawe-Taylor, James Thomas, Ailbhe Finnerty Mutlu, Marie Johnston, Pol Mac Aonghusa, Alison O'Mara-Eves, Gillian Stokes, Emma Norris, Marta Marques, Michael P Kelly, Debasis Ganguly, Lea Deleris, Kirsty Atha, Alison Wright, Clement Veall
Publication date:05/15/2020
Publisher:OSF
:view less
Ontology Development Methods
2020
Emma Norris, Yufang Hou, Candice Moore, Francesca Bonin, Robert West, Susan Michie, Joh…
:mehr lesen
Authors:Emma Norris, Yufang Hou, Candice Moore, Francesca Bonin, Robert West, Susan Michie, John Shawe-Taylor, James Thomas, Ailbhe Finnerty Mutlu, Marie Johnston, Pol Mac Aonghusa, Alison O'Mara-Eves, Gillian Stokes, Marta Marques, Michael P Kelly, Debasis Ganguly, Lea Deleris, Kirsty Atha, Alison Wright, Clement Veall
Publication date:05/15/2020
Publisher:OSF
:view less
Human Behaviour-Change Project
2020
Robert West, Susan Michie, John Shawe-Taylor, James Thomas, Ailbhe Finnerty, Marie John…
:mehr lesen
Authors:Robert West, Susan Michie, John Shawe-Taylor, James Thomas, Ailbhe Finnerty, Marie Johnston, Pol Mac Aonghusa, Alison O'Mara-Eves, Gillian Stokes, Emma Norris, Marta Marques, Michael P Kelly, Debasis Ganguly, Candice Moore, Yufang Hou, Francesca Bonin, Alison Wright, Clement Veall, Silje Zink, Paulina Schenk
Publication date:05/15/2020
Publisher:OSF
Description:The Human Behaviour-Change Project (HBCP) is creating an online Knowledge System that uses Artificial Intelligence, in particular Natural Language Processingand Machine Learning, to extract information from behaviour change intervention evaluation reports to answer key questions about the evidence. It is a collaboration between behavioural scientists, computer scientists and system architects. The Knowledge System is being curated with annotations of published reports according to the Behaviour Change Intervention Ontology also being developed as part of the project. The ontology is used to organise knowledge contained in behaviour change intervention evaluation reports. The Knowledge System will continually search publication databases to find behaviour change intervention evaluation reports, extract and synthesise the findings, provide up-to-date answers to questions, and draw inferences about behaviour change. Practitioners, policy makers and researchers will be able to query the system to obtain answers to variants of the key question:‘What intervention (s) work, compared with what, how well, with what exposure, with what behaviours, for how long, for whom, in what settings and why?
Cited by:10
:view less
HBCP corpus: a new resource for the analysis of behavioural change intervention reports
2020
Francesca Bonin, Martin Gleize, Ailbhe Finnerty, Candice Moore, Charles Jochim, Emma No…
Proceedings of the Twelfth Language Resources and Evaluation Conference, 1967-1975
:mehr lesen
Authors:Francesca Bonin, Martin Gleize, Ailbhe Finnerty, Candice Moore, Charles Jochim, Emma Norris, Yufang Hou, Alison J Wright, Debasis Ganguly, Emily Hayes, Silje Zink, Alessandra Pascale, Pol Mac Aonghusa, Susan Michie
Publication date:05/01/2020
Journal:Proceedings of the Twelfth Language Resources and Evaluation Conference
Pages:1967-1975
Description:Due to the fast pace at which research reports in behaviour change are published, researchers, consultants and policymakers would benefit from more automatic ways to process these reports. Automatic extraction of the reports’ intervention content, population, settings and their results etc. are essential in synthesising and summarising the literature. However, to the best of our knowledge, no unique resource exists at the moment to facilitate this synthesis. In this paper, we describe the construction of a corpus of published behaviour change intervention evaluation reports aimed at smoking cessation. We also describe and release the annotation of 57 entities, that can be used as an off-the-shelf data resource for tasks such as entity recognition, etc. Both the corpus and the annotation dataset are being made available to the community.
Cited by:16
Resources:[PDF] from aclanthology.org
:view less
Bridging Anaphora Resolution as Question Answering
2020
Yufang Hou
ACL 2020
:mehr lesen
Authors:Yufang Hou
Publication date:04/16/2020
Journal:ACL 2020
Description:Most previous studies on bridging anaphora resolution (Poesio et al., 2004; Hou et al., 2013b; Hou, 2018a) use the pairwise model to tackle the problem and assume that the gold mention information is given. In this paper, we cast bridging anaphora resolution as question answering based on context. This allows us to find the antecedent for a given anaphor without knowing any gold mention information (except the anaphor itself). We present a question answering framework (BARQA) for this task, which leverages the power of transfer learning. Furthermore, we propose a novel method to generate a large amount of "quasi-bridging" training data. We show that our model pre-trained on this dataset and fine-tuned on a small amount of in-domain dataset achieves new state-of-the-art results for bridging anaphora resolution on two bridging corpora (ISNotes (Markert et al., 2012) and BASHI (Roesiger, 2018)).
Cited by:51
Resources:[PDF] from arxiv.org
:view less
End-to-end argumentation knowledge graph construction
2020
Khalid Al-Khatib, Yufang Hou, Henning Wachsmuth, Charles Jochim, Francesca Bonin, Benno…
Proceedings of the AAAI Conference on Artificial Intelligence 34 (05), 7367-7374
:mehr lesen
Authors:Khalid Al-Khatib, Yufang Hou, Henning Wachsmuth, Charles Jochim, Francesca Bonin, Benno Stein
Publication date:04/03/2020
Journal:Proceedings of the AAAI Conference on Artificial Intelligence
Volume:34
Issue:05
Pages:7367-7374
Description:This paper studies the end-to-end construction of an argumentation knowledge graph that is intended to support argument synthesis, argumentative question answering, or fake news detection, among others. The study is motivated by the proven effectiveness of knowledge graphs for interpretable and controllable text generation and exploratory search. Original in our work is that we propose a model of the knowledge encapsulated in arguments. Based on this model, we build a new corpus that comprises about 16k manual annotations of 4740 claims with instances of the model's elements, and we develop an end-to-end framework that automatically identifies all modeled types of instances. The results of experiments show the potential of the framework for building a web-based argumentation graph that is of high quality and large scale.
Cited by:61
Resources:[PDF] from aaai.org
:view less
Corpus Wide Argument Mining—A Working Solution
2020
Liat Ein-Dor, Eyal Shnarch, Lena Dankin, Alon Halfon, Benjamin Sznajder, Ariel Gera, Ca…
Proceedings of the AAAI Conference on Artificial Intelligence 34 (05), 7683-7691
:mehr lesen
Authors:Liat Ein-Dor, Eyal Shnarch, Lena Dankin, Alon Halfon, Benjamin Sznajder, Ariel Gera, Carlos Alzate, Martin Gleize, Leshem Choshen, Yufang Hou, Yonatan Bilu, Ranit Aharonov, Noam Slonim
Publication date:04/03/2020
Journal:Proceedings of the AAAI Conference on Artificial Intelligence
Volume:34
Issue:05
Pages:7683-7691
Description:One of the main tasks in argument mining is the retrieval of argumentative content pertaining to a given topic. Most previous work addressed this task by retrieving a relatively small number of relevant documents as the initial source for such content. This line of research yielded moderate success, which is of limited use in a real-world system. Furthermore, for such a system to yield a comprehensive set of relevant arguments, over a wide range of topics, it requires leveraging a large and diverse corpus in an appropriate manner. Here we present a first end-to-end high-precision, corpus-wide argument mining system. This is made possible by combining sentence-level queries over an appropriate indexing of a very large corpus of newspaper articles, with an iterative annotation scheme. This scheme addresses the inherent label bias in the data and pinpoints the regions of the sample space whose manual labeling is required to obtain high-precision among top-ranked candidates.
Cited by:81
Resources:[PDF] from aaai.org
:view less
A Summarization System for Scientific Documents
2019
Shai Erera, Michal Shmueli-Scheuer, Guy Feigenblat, Ora Peled Nakash, Odellia Boni, Hag…
EMNLP 2019 (Demo)
:mehr lesen
Authors:Shai Erera, Michal Shmueli-Scheuer, Guy Feigenblat, Ora Peled Nakash, Odellia Boni, Haggai Roitman, Doron Cohen, Bar Weiner, Yosi Mass, Or Rivlin, Guy Lev, Achiya Jerbi, Jonathan Herzig, Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, David Konopnicki
Publication date:08/29/2019
Journal:EMNLP 2019 (Demo)
Description:We present a novel system providing summaries for Computer Science publications. Through a qualitative user study, we identified the most valuable scenarios for discovery, exploration and understanding of scientific documents. Based on these findings, we built a system that retrieves and summarizes scientific documents for a given information need, either in form of a free-text query or by choosing categorized values such as scientific tasks, datasets and more. Our system ingested 270,000 papers, and its summarization module aims to generate concise yet detailed summaries. We validated our approach with human experts.
Cited by:71
Resources:[PDF] from arxiv.org
:view less
Fine-grained Information Status Classification Using Discourse Context-Aware Self-Attention
2019
Yufang Hou
arXiv preprint arXiv:1908.04755
:mehr lesen
Authors:Yufang Hou
Publication date:08/13/2019
Journal:arXiv preprint arXiv:1908.04755
Description:Previous work on bridging anaphora recognition (Hou et al., 2013a) casts the problem as a subtask of learning fine-grained information status (IS). However, these systems heavily depend on many hand-crafted linguistic features. In this paper, we propose a discourse context-aware self-attention neural network model for fine-grained IS classification. On the ISNotes corpus (Markert et al., 2012), our model with the contextually-encoded word representations (BERT) (Devlin et al., 2018) achieves new state-of-the-art performances on fine-grained IS classification, obtaining a 4.1% absolute overall accuracy improvement compared to Hou et al. (2013a). More importantly, we also show an improvement of 3.9% F1 for bridging anaphora recognition without using any complex hand-crafted semantic features designed for capturing the bridging phenomenon.
Resources:[PDF] from arxiv.org
:view less
Extracting Factual Min/Max Age Information from Clinical Trial Studies
2019
Yufang Hou, Debasis Ganguly, Lea A Deleris, Francesca Bonin
ClinicalNLP 2019 (Co-located at NAACL 2019)
:mehr lesen
Authors:Yufang Hou, Debasis Ganguly, Lea A Deleris, Francesca Bonin
Publication date:04/01/2019
Journal:ClinicalNLP 2019 (Co-located at NAACL 2019)
Description:Population age information is an essential characteristic of clinical trials. In this paper, we focus on extracting minimum and maximum (min/max) age values for the study samples from clinical research articles. Specifically, we investigate the use of a neural network model for question answering to address this information extraction task. The min/max age QA model is trained on the massive structured clinical study records from ClinicalTrials.gov. For each article, based on multiple min and max age values extracted from the QA model, we predict both actual min/max age values for the study samples and filter out non-factual age expressions. Our system improves the results over (i) a passage retrieval based IE system and (ii) a CRF-based system by a large margin when evaluated on an annotated dataset consisting of 50 research papers on smoking cessation.
Resources:[PDF] from arxiv.org
:view less
Will it Blend? Blending Weak and Strong Labeled Data in a Neural Network for Argumentation Mining
2018
Eyal Shnarch, Carlos Alzate, Lena Dankin, Martin Gleize, Yufang Hou, Leshem Choshen, Ra…
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics…
:mehr lesen
Authors:Eyal Shnarch, Carlos Alzate, Lena Dankin, Martin Gleize, Yufang Hou, Leshem Choshen, Ranit Aharonov, Noam Slonim
Publication date:07/01/2018
Journal:Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Pages:599-605
Description:The process of obtaining high quality labeled data for natural language understanding tasks is often slow, error-prone, complicated and expensive. With the vast usage of neural networks, this issue becomes more notorious since these networks require a large amount of labeled data to produce satisfactory results. We propose a methodology to blend high quality but scarce strong labeled data with noisy but abundant weak labeled data during the training of neural networks. Experiments in the context of topic-dependent evidence detection with two forms of weak labeled data show the advantages of the blending scheme. In addition, we provide a manually annotated data set for the task of topic-dependent evidence detection. We believe that blending weak and strong labeled data is a general notion that may be applicable to many language understanding tasks, and can especially assist researchers who wish to train a network but have a small amount of high quality labeled data for their task of interest.
Cited by:83
Resources:[PDF] from aclanthology.org
:view less
Unrestricted bridging resolution
2018
Yufang Hou, Katja Markert, Michael Strube
Computational Linguistics 44 (2), 237-284
:mehr lesen
Authors:Yufang Hou, Katja Markert, Michael Strube
Publication date:06/01/2018
Journal:Computational Linguistics
Volume:44
Issue:2
Pages:237-284
Publisher:MIT Press
Description:In contrast to identity anaphors, which indicate coreference between a noun phrase and its antecedent, bridging anaphors link to their antecedent(s) via lexico-semantic, frame, or encyclopedic relations. Bridging resolution involves recognizing bridging anaphors and finding links to antecedents. In contrast to most prior work, we tackle both problems. Our work also follows a more wide-ranging definition of bridging than most previous work and does not impose any restrictions on the type of bridging anaphora or relations between anaphor and antecedent. We create a corpus (ISNotes) annotated for information status (IS), bridging being one of the IS subcategories. The annotations reach high reliability for all categories and marginal reliability for the bridging subcategory. We use a two-stage statistical global inference method for bridging resolution. Given all mentions in a document, the first stage, bridging …
Cited by:51
Resources:[PDF] from mit.edu
:view less
Enhanced Word Representations for Bridging Anaphora Resolution
2018
Yufang Hou
NAACL 2018
:mehr lesen
Authors:Yufang Hou
Publication date:03/13/2018
Journal:NAACL 2018
Description:Most current models of word representations(e.g.,GloVe) have successfully captured fine-grained semantics. However, semantic similarity exhibited in these word embeddings is not suitable for resolving bridging anaphora, which requires the knowledge of associative similarity (i.e., relatedness) instead of semantic similarity information between synonyms or hypernyms. We create word embeddings (embeddings_PP) to capture such relatedness by exploring the syntactic structure of noun phrases. We demonstrate that using embeddings_PP alone achieves around 30% of accuracy for bridging anaphora resolution on the ISNotes corpus. Furthermore, we achieve a substantial gain over the state-of-the-art system (Hou et al., 2013) for bridging antecedent selection.
Cited by:25
Resources:[PDF] from arxiv.org
:view less
Argument Relation Classification Using a Joint Inference Model
2017
Yufang Hou, Charles Jochim
the 4th argMining workshop@EMNLP 2017
:mehr lesen
Authors:Yufang Hou, Charles Jochim
Publication date:09/08/2017
Journal:the 4th argMining workshop@EMNLP 2017
Description:In this paper, we address the problem of argument relation classification where argument units are from different texts. We design a joint inference method for the task by modeling argument relation classification and stance classification jointly. We show that our joint model improves the results over several strong baselines.
Cited by:36
Resources:[PDF] from aclanthology.org
:view less
Collective classification for fine-grained information status
2012
Katja Markert, Yufang Hou, Michael Strube
Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics…
:mehr lesen
Authors:Katja Markert, Yufang Hou, Michael Strube
Publication date:07/01/2012
Journal:Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Pages:795-804
Description:Previous work on classifying information status (Nissim, 2006; Rahman and Ng, 2011) is restricted to coarse-grained classification and focuses on conversational dialogue. We here introduce the task of classifying finegrained information status and work on written text. We add a fine-grained information status layer to the Wall Street Journal portion of the OntoNotes corpus. We claim that the information status of a mention depends not only on the mention itself but also on other mentions in the vicinity and solve the task by collectively classifying the information status of all mentions. Our approach strongly outperforms reimplementations of previous work.
Cited by:79
Resources:[PDF] from aclanthology.org
:view less
Weakly supervised graph-based coreference resolution for clinical data
2011
Jie Cai, Eva Mújdricza-Maydt, Yufang Hou, Michael Strube
Proceedings of the 5th i2b2 Shared Tasks and Workshop on Challenges in Natural Language…
:mehr lesen
Authors:Jie Cai, Eva Mújdricza-Maydt, Yufang Hou, Michael Strube
Publication date:10/01/2011
Journal:Proceedings of the 5th i2b2 Shared Tasks and Workshop on Challenges in Natural Language Processing for Clinical Data, Washington, DC
Pages:20-21
Cited by:2
:view less
Publishing distributed files as Linked Data
2011
Zhihong Shen, Yufang Hou, Jianhui Li
3, 1694-1698
:mehr lesen
Authors:Zhihong Shen, Yufang Hou, Jianhui Li
Publication date:07/26/2011
Volume:3
Pages:1694-1698
Publisher:IEEE
Description:File systems act as backbones of many information systems, however, it is still hard to search and integrate information about files which are distributed in different locations over different protocols. Traditional researches about file systems focus on file sharing and transfer, paying little attention to semantic annotation and interlinking of files in distributed environment. In this paper, we study how to publish and consume files as Linked Data. Some principles are discussed in detail, including choosing appropriate HTTP URIs, providing metadata and content of files, generating external links between files and other resources, connecting real repositories located in different places. Then we present VDB-FilePub, a component of VisualDB for publishing the files stored in different repositories as Linked Data. After introducing the architecture of VDB-FilePub, we give some scenarios of how other applications consume …
Cited by:10
Resources:[PDF] from github.io
:view less
:zurück zur übersicht