Unsupervised Contrast-Consistent Ranking with Language Models
We analyze the ranking capabilities of language models by comparing pairwise, pointwise and listwise prompting techniques. We then propose an unsupervised probing method termed Contrast-Consistent Ranking (CCR). CCR relates multiple language model queries through a surrogate model that projects the language model's outputs to a shared ranking scale.
Niklas Stoehr 11
Pengxiang Cheng 22
Jing Wang 22
Daniel Preotiuc-Pietro 22
Rajarshi Bhowmik 22
EACL 2024
Estimating conflict losses and reporting biases
We develop a statistical model to better estimate losses reported in multiple, possibly biased news reports. We publish a dataset of 4,609 reports of military and civilian losses caused by the Russian invasion in the Ukraine.
Benjamin Radford 19
Yaoyao Dai 19
Niklas Stoehr 11
Aaron Schein 17
Mya Fernandez 19
Hanif Sajid 19
PNAS Journal
Generalizing Backpropagation for Gradient-Based Interpretability
In this paper, we observe that the gradient computation of a model is a special case of a more general formulation using semirings. This observation allows us to generalize the backpropagation algorithm to efficiently compute interpretable statistics about the gradient graph of a neural network, such as the highest-weighted path and entropy.
Kevin Du 11
Lucas Torroba Hennigen 13
Niklas Stoehr 11
Alex Warstadt 11
Ryan Cotterell 11
Outstanding Paper Award at ACL 2023
Sentiment as an Ordinal Latent Variable
We propose a Bayesian generative model that learns a composite sentiment dictionary as an interpolation between six existing sentiment dictionaries with different discrete and continuous scales. We argue that sentiment is a latent concept with intrinsically ranking-based characteristics.
Niklas Stoehr 11
Ryan Cotterell 11
Aaron Schein 17
EACL 2023
Extracting Victim Counts from Text
We cast victim count extraction as a question answering (QA) task with a regression or classification objective. We compare regex, dependency parsing, semantic role labeling-based approaches, and pretrained text-to-text models.
Mian Zhong 11
Shehzaad Dhuliawala 11
Niklas Stoehr 11
EACL 2023
World Models for Math Story Problems
In this paper, we consolidate previous work on categorizing and representing math story problems and develop MathWorld, a graph-based semantic formalism specific for the domain of math story problems.
Andreas Opedal 11,20
Niklas Stoehr 11
Abu Saparov 21
Mrinmaya Sachan 11
Findings of ACL 2023
The Ordered Matrix Dirichlet for State-Space Models
This paper introduces the Ordered Matrix Dirichlet (OMD) as a prior distribution over ordered stochastic matrices wherein the discrete distribution in the kth row is stochastically dominated by the (k+1)th, such that probability mass is shifted to the right when moving down rows.
Niklas Stoehr 11
Benjamin Radford 19
Ryan Cotterell 11
Aaron Schein 17
AISTATS 2023
The Architectural Bottleneck Principle
In this paper, we seek to measure how much information a component in a neural network could extract from the representations fed into it. Our work stands in contrast to prior probing work, most of which investigates how much information a model’s representations contain.
Tiago Pimentel 12
Josef Valvoda 12
Niklas Stoehr 11
Ryan Cotterell 11
An Ordinal Latent Variable Model of Conflict Intensity
For the quantitative monitoring of international relations, political events are extracted from the news and parsed into "who-did-what-to-whom" patterns. This has resulted in large data collections which require aggregate statistics for analysis. The Goldstein Scale is an expert-based measure that ranks individual events on a one-dimensional scale from conflictual to cooperative.
Niklas Stoehr 11
Lucas Torroba Hennigen 13
Josef Valvoda 12
Robert West 15
Ryan Cotterell 11
Aaron Schein 17
ACL 2023
Rethinking the Event Coding Pipeline with Prompt Entailment
In this work, we propose PR-ENT, a new event coding approach that is more flexible and resource-efficient, while maintaining competitive accuracy.
Clement Lefebvre 18
Niklas Stoehr 11
FEVER at EACL 2023
SeismographAPI: Visualising Temporal-Spatial Crisis Data
We present SeismographAPI, an open-source library for visualising temporal-spatial crisis data on the country- and sub-country level.
Raphael Lepuschitz 16
Niklas Stoehr 11
Classifying Dyads for Militarized Conflict Analysis
Understanding the origins of militarized conflict is a complex, yet important undertaking. Existing research seeks to build this understanding by considering bi-lateral relationships between entity pairs (dyadic causes) and multi-lateral relationships among multiple entities (systemic causes).
Niklas Stoehr 11
Lucas Torroba Hennigen 13
Samin Ahbab
Robert West 15
Ryan Cotterell 11, 12
What About the Precedent: An Information-Theoretic Analysis of Common Law
We are the first to approach this question computationally by comparing two longstanding jurisprudential views; Halsbury's, who believes that the arguments of the precedent are the main determinant of the outcome, and Goodhart's, who believes that what matters most is the precedent's facts. We base our study on the corpus of legal cases from the European Court of Human Rights (ECtHR).
Josef Valvoda 12
Tiago Pimentel 12
Niklas Stoehr 11
Ryan Cotterell 11,12
Simone Teufel 12
Sentiment Political Compass: A Data-driven Analysis of Online Newspapers regarding Political Orientation
This article introduces the Sentiment Political Compass (SPC), a data‐driven framework for analyzing political bias of newspapers toward political parties.
Fabian Falck 4
Julian Marstaller 7
Niklas Stoehr 5
Sören Maucher 7
Jeana Ren 7
Andreas Thalhammer 7
Achim Rettinger 7
Rudi Studer 7
Policy & Internet, Wiley Online Library, 2019
Fear-anger contests: Governmental and populist politics of emotion
This article explores how political actors use the emotions of fear and anger in what we call fear-anger contests. Our theory distinguishes between governmental and populist actors and posits that, in a contest for media attention and the hearts and minds of citizens, populists pursue a politics of anger whereas governmental actors pursue a politics of fear.
Joerg Friedrichs 5
Niklas Stoehr 11
Giuliano Formisano 5
Journal of Online Social Networks and Media
The CoRisk-Index: A data-mining approach to identify industry-specific risk assessments related to COVID-19 in real-time
The CoRisk-Index extends the spectre of existing, but less frequently updated economic indicators by tapping into an unexploited and highly abundant data source. The new indicator can be considered business-internally since corporations provide information themselves and data fidelity is self-incentivized due to law enforcement.
Fabian Stephany 5
Niklas Stoehr 9
Philipp Darius 10
Leonie Neuhäuser 10
Ole Teutloff 10
Fabian Braesemann 5
Nature Humanities and Social Sciences Communications
Disentangling Interpretable Generative Parameters of Random and Real-World Graphs
This work approaches graph generation (decoding) as the inverse of graph compression (encoding). We show that in a disentanglement-focused deep autoencoding framework, specifically Beta-Variational Autoencoders (Beta-VAE), choices of generative procedures and their parameters arise naturally in the latent space.
Niklas Stoehr 3
Emine Yilmaz 3
Marc Brockschmidt 6
Jan Stuehmer 6
Workshop on Graph Representation Learning, 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver
Mining the Automotive Industry: A Network Analysis of Corporate Positioning and Technological Trends
Following the need for monitoring shifting industries, we present a network-centered analysis of car manufacturer web pages. Solely exploiting publicly-available information, we construct large networks from web pages and hyperlinks. We tag web pages concerned with topics like e-mobility and environment or autonomous driving, and investigate their relevance in the network.
Niklas Stoehr 9
Fabian Braesemann 5
Michael Frommelt 9
Shi Zhou 3
CompleNet 2020, published in Springer Nature
Global networks in collaborative programming
To understand the dynamics of the digital knowledge economy, it is crucial to reveal the geography of global flows of knowledge on digital platforms. This article visualizes a key form of knowledge production in the digital economy: mapping the joint collaborations of users from different cities on Stack Overflow, the world’s most popular question-and-answer website for programming questions.
Fabian Braesemann 5
Niklas Stoehr 3
Mark Graham 5
Regional Studies, Regional Science, Volume 6, 2019
1 TU Berlin 2 University of California, Irvine 3 University College London 4 Imperial College London 5 University of Oxford 6 Microsoft Research 7 KIT 8 Carnegie Mellon University 9 IBM 10 Hertie School of Governance 11 ETH Zurich 12 University of Cambridge 13 MIT 14 European Commission 15 EPFL 16 University of Innsbruck 17 University of Chicago 18 Swiss Data Science Center 19 UNC Charlotte 20 MPI Tübingen 21 NYU 22 Bloomberg
Unsupervised Contrast-Consistent Ranking with Language Models
We analyze the ranking capabilities of language models by comparing pairwise, pointwise and listwise prompting techniques. We then propose an unsupervised probing method termed Contrast-Consistent Ranking (CCR). CCR relates multiple language model queries through a surrogate model that projects the language model's outputs to a shared ranking scale.
Niklas Stoehr 11
Pengxiang Cheng 22
Jing Wang 22
Daniel Preotiuc-Pietro 22
Rajarshi Bhowmik 22
EACL 2024
Estimating conflict losses and reporting biases
We develop a statistical model to better estimate losses reported in multiple, possibly biased news reports. We publish a dataset of 4,609 reports of military and civilian losses caused by the Russian invasion in the Ukraine.
Benjamin Radford 19
Yaoyao Dai 19
Niklas Stoehr 11
Aaron Schein 17
Mya Fernandez 19
Hanif Sajid 19
PNAS Journal
Generalizing Backpropagation for Gradient-Based Interpretability
In this paper, we observe that the gradient computation of a model is a special case of a more general formulation using semirings. This observation allows us to generalize the backpropagation algorithm to efficiently compute interpretable statistics about the gradient graph of a neural network, such as the highest-weighted path and entropy.
Kevin Du 11
Lucas Torroba Hennigen 13
Niklas Stoehr 11
Alex Warstadt 11
Ryan Cotterell 11
Outstanding Paper Award at ACL 2023
Sentiment as an Ordinal Latent Variable
We propose a Bayesian generative model that learns a composite sentiment dictionary as an interpolation between six existing sentiment dictionaries with different discrete and continuous scales. We argue that sentiment is a latent concept with intrinsically ranking-based characteristics.
Niklas Stoehr 11
Ryan Cotterell 11
Aaron Schein 17
EACL 2023
Extracting Victim Counts from Text
We cast victim count extraction as a question answering (QA) task with a regression or classification objective. We compare regex, dependency parsing, semantic role labeling-based approaches, and pretrained text-to-text models.
Mian Zhong 11
Shehzaad Dhuliawala 11
Niklas Stoehr 11
EACL 2023
World Models for Math Story Problems
In this paper, we consolidate previous work on categorizing and representing math story problems and develop MathWorld, a graph-based semantic formalism specific for the domain of math story problems.
Andreas Opedal 11,20
Niklas Stoehr 11
Abu Saparov 21
Mrinmaya Sachan 11
Findings of ACL 2023
The Ordered Matrix Dirichlet for State-Space Models
This paper introduces the Ordered Matrix Dirichlet (OMD) as a prior distribution over ordered stochastic matrices wherein the discrete distribution in the kth row is stochastically dominated by the (k+1)th, such that probability mass is shifted to the right when moving down rows.
Niklas Stoehr 11
Benjamin Radford 19
Ryan Cotterell 11
Aaron Schein 17
AISTATS 2023
The Architectural Bottleneck Principle
In this paper, we seek to measure how much information a component in a neural network could extract from the representations fed into it. Our work stands in contrast to prior probing work, most of which investigates how much information a model’s representations contain.
Tiago Pimentel 12
Josef Valvoda 12
Niklas Stoehr 11
Ryan Cotterell 11
An Ordinal Latent Variable Model of Conflict Intensity
For the quantitative monitoring of international relations, political events are extracted from the news and parsed into "who-did-what-to-whom" patterns. This has resulted in large data collections which require aggregate statistics for analysis. The Goldstein Scale is an expert-based measure that ranks individual events on a one-dimensional scale from conflictual to cooperative.
Niklas Stoehr 11
Lucas Torroba Hennigen 13
Josef Valvoda 12
Robert West 15
Ryan Cotterell 11
Aaron Schein 17
ACL 2023
Rethinking the Event Coding Pipeline with Prompt Entailment
In this work, we propose PR-ENT, a new event coding approach that is more flexible and resource-efficient, while maintaining competitive accuracy.
Clement Lefebvre 18
Niklas Stoehr 11
FEVER at EACL 2023
SeismographAPI: Visualising Temporal-Spatial Crisis Data
We present SeismographAPI, an open-source library for visualising temporal-spatial crisis data on the country- and sub-country level.
Raphael Lepuschitz 16
Niklas Stoehr 11
Classifying Dyads for Militarized Conflict Analysis
Understanding the origins of militarized conflict is a complex, yet important undertaking. Existing research seeks to build this understanding by considering bi-lateral relationships between entity pairs (dyadic causes) and multi-lateral relationships among multiple entities (systemic causes).
Niklas Stoehr 11
Lucas Torroba Hennigen 13
Samin Ahbab
Robert West 15
Ryan Cotterell 11, 12
What About the Precedent: An Information-Theoretic Analysis of Common Law
We are the first to approach this question computationally by comparing two longstanding jurisprudential views; Halsbury's, who believes that the arguments of the precedent are the main determinant of the outcome, and Goodhart's, who believes that what matters most is the precedent's facts. We base our study on the corpus of legal cases from the European Court of Human Rights (ECtHR).
Josef Valvoda 12
Tiago Pimentel 12
Niklas Stoehr 11
Ryan Cotterell 11,12
Simone Teufel 12
Sentiment Political Compass: A Data-driven Analysis of Online Newspapers regarding Political Orientation
This article introduces the Sentiment Political Compass (SPC), a data‐driven framework for analyzing political bias of newspapers toward political parties.
Fabian Falck 4
Julian Marstaller 7
Niklas Stoehr 5
Sören Maucher 7
Jeana Ren 7
Andreas Thalhammer 7
Achim Rettinger 7
Rudi Studer 7
Policy & Internet, Wiley Online Library, 2019
Fear-anger contests: Governmental and populist politics of emotion
This article explores how political actors use the emotions of fear and anger in what we call fear-anger contests. Our theory distinguishes between governmental and populist actors and posits that, in a contest for media attention and the hearts and minds of citizens, populists pursue a politics of anger whereas governmental actors pursue a politics of fear.
Joerg Friedrichs 5
Niklas Stoehr 11
Giuliano Formisano 5
Journal of Online Social Networks and Media
The CoRisk-Index: A data-mining approach to identify industry-specific risk assessments related to COVID-19 in real-time
The CoRisk-Index extends the spectre of existing, but less frequently updated economic indicators by tapping into an unexploited and highly abundant data source. The new indicator can be considered business-internally since corporations provide information themselves and data fidelity is self-incentivized due to law enforcement.
Fabian Stephany 5
Niklas Stoehr 9
Philipp Darius 10
Leonie Neuhäuser 10
Ole Teutloff 10
Fabian Braesemann 5
Nature Humanities and Social Sciences Communications
Disentangling Interpretable Generative Parameters of Random and Real-World Graphs
This work approaches graph generation (decoding) as the inverse of graph compression (encoding). We show that in a disentanglement-focused deep autoencoding framework, specifically Beta-Variational Autoencoders (Beta-VAE), choices of generative procedures and their parameters arise naturally in the latent space.
Niklas Stoehr 3
Emine Yilmaz 3
Marc Brockschmidt 6
Jan Stuehmer 6
Workshop on Graph Representation Learning, 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver
Mining the Automotive Industry: A Network Analysis of Corporate Positioning and Technological Trends
Following the need for monitoring shifting industries, we present a network-centered analysis of car manufacturer web pages. Solely exploiting publicly-available information, we construct large networks from web pages and hyperlinks. We tag web pages concerned with topics like e-mobility and environment or autonomous driving, and investigate their relevance in the network.
Niklas Stoehr 9
Fabian Braesemann 5
Michael Frommelt 9
Shi Zhou 3
CompleNet 2020, published in Springer Nature
Global networks in collaborative programming
To understand the dynamics of the digital knowledge economy, it is crucial to reveal the geography of global flows of knowledge on digital platforms. This article visualizes a key form of knowledge production in the digital economy: mapping the joint collaborations of users from different cities on Stack Overflow, the world’s most popular question-and-answer website for programming questions.
Fabian Braesemann 5
Niklas Stoehr 3
Mark Graham 5
Regional Studies, Regional Science, Volume 6, 2019
1 TU Berlin 2 University of California, Irvine 3 University College London 4 Imperial College London 5 University of Oxford 6 Microsoft Research 7 KIT 8 Carnegie Mellon University 9 IBM 10 Hertie School of Governance 11 ETH Zurich 12 University of Cambridge 13 MIT 14 European Commission 15 EPFL 16 University of Innsbruck 17 University of Chicago 18 Swiss Data Science Center 19 UNC Charlotte 20 MPI Tübingen 21 NYU 22 Bloomberg