Publications

2022

1. Preprint

   Exclusive Supermask Subnetwork Training for Continual Learning

   Yadav, Prateek, and Bansal, Mohit

   In 2022

   Abs arXiv Bib Code

   Continual Learning (CL) methods mainly focus on avoiding catastrophic forgetting by employing techniques like experience replay, regularization, and parameter isolation. Recently, Wortsman et al. proposed SupSup which exploits the power of a supermask, which is a sparse binary mask that selects parameters inside a randomly initialized model resulting in a subnetwork. For each task, they learn a supermask and across different tasks these supermasks can have intersections resulting in overlapping model parameters. They prevent forgetting because the model weights are never trained and hence, there can be no conflicting weight updates by different tasks. However, there are a few limitations of using supermasks: (1) for a single task its performance is worse than a fully trained model; (2) when learning multiple tasks sequentially, naive extensions of SupSup does not bridge this performance gap and training tasks’ subnetwork weights will cause conflicting updates to the overlapping parameters across tasks resulting in forgetting. Hence, we propose ExSSNeT (Exclusive Supermask SubNetwork Training), which performs exclusive and non-overlapping subnetwork weight training to continually learn tasks. For a single task, ExSSNeT bridges the performance gap between SupSup and a fully trained model. For multiple tasks, our method avoids conflicting updates to the overlapping weights by subsequent tasks for improving individual task performance while avoiding forgetting. Furthermore, we propose a KNN-based method which can be added to ExSSNeT to dynamically initialize a new task’s mask based on the previous tasks for improved knowledge transfer. We demonstrate the effectiveness of our proposed method by comparing it with strong baselines on several popular vision and text classification tasks. Our method is particularly advantageous for sparse masks with 2−10% of the model parameters, resulting in an average improvement of 8.3% over SupSup. Finally, ExSSNeT scales to a large number of tasks (100) and our KNN-based mask initialization leads to further improvements over ExSSNeT.

   @inproceedings{yadav2022exssnet,
     abbr = {Preprint},
     bibtex_show = {true},
     selected = {true},
     code = {https://github.com/prateeky2806/exessnet},
     author = {Yadav, Prateek and Bansal, Mohit},
     title = {Exclusive Supermask Subnetwork Training for Continual Learning},
     year = {2022},
     arxiv = {2210.10209}
   }

2. ACL

   Explanation Graph Generation via Pre-trained Language Models: An Empirical Study with Contrastive Learning

   Saha, Swarnadeep, Yadav, Prateek, and Bansal, Mohit

   In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) 2022

   Abs Bib Code Poster

   Pre-trained sequence-to-sequence language models have led to widespread success in many natural language generation tasks. However, there has been relatively less work on analyzing their ability to generate structured outputs such as graphs. Unlike natural language, graphs have distinct structural and semantic properties in the context of a downstream NLP task, e.g., generating a graph that is connected and acyclic can be attributed to its structural constraints, while the semantics of a graph can refer to how meaningfully an edge represents the relation between two node concepts. In this work, we study pre-trained language models that generate explanation graphs in an end-to-end manner and analyze their ability to learn the structural constraints and semantics of such graphs. We first show that with limited supervision, pre-trained language models often generate graphs that either violate these constraints or are semantically incoherent. Since curating large amount of human-annotated graphs is expensive and tedious, we propose simple yet effective ways of graph perturbations via node and edge edit operations that lead to structurally and semantically positive and negative graphs. Next, we leverage these graphs in different contrastive learning models with Max-Margin and InfoNCE losses. Our methods lead to significant improvements in both structural and semantic accuracy of explanation graphs and also generalize to other similar graph generation tasks. Lastly, we show that human errors are the best negatives for contrastive learning and also that automatically generating more such human-like negative graphs can lead to further improvements.

   @inproceedings{saha2022explanation,
     abbr = {ACL},
     bibtex_show = {true},
     selected = {true},
     code = {https://github.com/swarnaHub/ExplagraphGen},
     poster = {https://drive.google.com/file/d/13U1XX99efxzkQzmBGQv_P3nN79DnKukc/view?usp=sharing},
     title = {Explanation Graph Generation via Pre-trained Language Models: An Empirical Study with Contrastive Learning},
     author = {Saha, Swarnadeep and Yadav, Prateek and Bansal, Mohit},
     booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
     month = nov,
     year = {2022},
     publisher = {Association for Computational Linguistics}
   }

3. Preprint

   Low-Cost Algorithmic Recourse for Users With Uncertain Cost Functions

   Yadav, Prateek, Hase, Peter, and Bansal, Mohit

   2022

   arXiv Bib Code

   @misc{yadav2021lowcost,
     abbr = {Preprint},
     bibtex_show = {true},
     selected = {true},
     code = {https://github.com/prateeky2806/EMC-COLS-recourse},
     title = {Low-Cost Algorithmic Recourse for Users With Uncertain Cost Functions},
     author = {Yadav, Prateek and Hase, Peter and Bansal, Mohit},
     year = {2022},
     arxiv = {2111.01235},
     archiveprefix = {arXiv},
     primaryclass = {cs.LG}
   }

2021

1. EMNLP [ORAL]

   ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning

   Saha, Swarnadeep, Yadav, Prateek, Bauer, Lisa, and Bansal, Mohit

   In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) 2021

   Abs arXiv Bib Code Poster

   Despite the promising results of current cross-lingual models for spoken language understanding systems, they still suffer from imperfect cross-lingual representation alignments between the source and target languages, which makes the performance sub-optimal. To cope with this issue, we propose a regularization approach to further align word-level and sentence-level representations across languages without any external resource. First, we regularize the representation of user utterances based on their corresponding labels. Second, we regularize the latent variable model (Liu et al., 2019) by leveraging adversarial training to disentangle the latent variables. Experiments on the cross-lingual spoken language understanding task show that our model outperforms current state-of-the-art methods in both few-shot and zero-shot scenarios, and our model, trained on a few-shot setting with only 3% of the target language training data, achieves comparable performance to the supervised training with all the training data.

   @inproceedings{saha2021explagraphs,
     abbr = {EMNLP [ORAL]},
     bibtex_show = {true},
     selected = {true},
     code = {https://github.com/swarnaHub/ExplaGraphs},
     poster = {https://drive.google.com/file/d/1EzhGbP3rJlH6X49xAYsApiM_jYfc1V8k/view?usp=sharing},
     arxiv = {2104.07644},
     title = {ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning},
     author = {Saha, Swarnadeep and Yadav, Prateek and Bauer, Lisa and Bansal, Mohit},
     booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
     month = nov,
     year = {2021},
     publisher = {Association for Computational Linguistics}
   }

2. NAACL [ORAL]

   multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning

   Saha, Swarnadeep, Yadav, Prateek, and Bansal, Mohit

   In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 2021

   Abs arXiv Bib Code Poster

   We focus on a type of linguistic formal reasoning where the goal is to reason over explicit knowledge in the form of natural language facts and rules (Clark et al., 2020). A recent work, named PRover (Saha et al., 2020), performs such reasoning by answering a question and also generating a proof graph that explains the answer. However, compositional reasoning is not always unique and there may be multiple ways of reaching the correct answer. Thus, in our work, we address a new and challenging problem of generating multiple proof graphs for reasoning over natural language rule-bases. Each proof provides a different rationale for the answer, thereby improving the interpretability of such reasoning systems. In order to jointly learn from all proof graphs and exploit the correlations between multiple proofs for a question, we pose this task as a set generation problem over structured output spaces where each proof is represented as a directed graph. We propose two variants of a proof-set generation model, multiPRover. Our first model, Multilabel-multiPRover, generates a set of proofs via multi-label classification and implicit conditioning between the proofs; while the second model, Iterative-multiPRover, generates proofs iteratively by explicitly conditioning on the previously generated proofs. Experiments on multiple synthetic, zero-shot, and human-paraphrased datasets reveal that both multiPRover models significantly outperform PRover on datasets containing multiple gold proofs. Iterative-multiPRover obtains state-of-the-art proof F1 in zero-shot scenarios where all examples have single correct proofs. It also generalizes better to questions requiring higher depths of reasoning where multiple proofs are more frequent.

   @inproceedings{saha2021multiprover,
     abbr = {NAACL [ORAL]},
     bibtex_show = {true},
     selected = {true},
     code = {https://github.com/swarnaHub/multiPRover},
     poster = {https://drive.google.com/file/d/1_TUoWE8xd2w6T4zvAsdHatJW764Z-61w/view?usp=sharing},
     arxiv = {2106.01354},
     title = {multi{PR}over: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning},
     author = {Saha, Swarnadeep and Yadav, Prateek and Bansal, Mohit},
     booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
     month = jun,
     year = {2021},
     address = {Online},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/2021.naacl-main.287},
     doi = {10.18653/v1/2021.naacl-main.287},
     pages = {3662--3677}
   }

3. Preprint

   Discrete Time Latent Hawkes Processes for Modeling Multidimensional Temporal Event Streams

   Yadav, Prateek, Sankaran, Raman, Dutta, Partha, and Bhatt, Rushi

   Preprint 2021

   Abs Bib PDF

   Multidimensional event streams are common in many applications such as content on social platforms. Existing models using Hawkes processes and its variants often ignore important information about the causal parents of the events, which typically is readily available in social media applications. These models also ignore the disproportionate response created by some of the rare events, such as the impact of a “like” on a content by an influencer. Addressing these limitations, this paper proposes a novel Bayesian dIscRete Time Hawkes (BIRTH) model, a Bayesian generative model for multidimensional event streams data with causal information. Through its latent variables, BIRTH is flexible enough to capture contrasting responses invoked by multiple events of the same type. Moreover, being a discrete-time model, the latent parameters scale as O(#Time bins) in BIRTH as compared to O(#events) for continuous-time processes, thus scaling better in the settings when the number of events is huge. For inference, we propose a Gibbs sampling based inference procedure for BIRTH, which is suitable when the whole data can be processed together. While a full variational inference procedure is difficult to arrive at due to non-conjugate factors in the posterior, we propose a Stochastic hybrid Gibbs-Variational Inference (SVI) algorithm, which is beneficial in the settings where Gibbs might be expensive in terms of memory requirements. SVI has per-iteration memory complexity proportional to the chosen minibatch size, and also extends easily for online streaming settings of the data. We thoroughly evaluate BIRTH’s abilities over both synthetic and real-world social network event streams. Specifically, on synthetic datasets we demonstrate model fitting, recovery of planted structure and identification of the rare events. For a social network dataset we show significantly higher likelihoods along with rare event identification.

   @article{virality,
     abbr = {Preprint},
     bibtex_show = {true},
     selected = {false},
     title = {Discrete Time Latent Hawkes Processes for Modeling Multidimensional Temporal Event Streams},
     author = {{Yadav}, Prateek and {Sankaran}, Raman and {Dutta}, Partha and {Bhatt}, Rushi},
     journal = {Preprint},
     year = {2021},
     pdf = {birth.pdf}
   }

4. GCLR AAAI

   Rank Refinement: An Algorithmic framework with applications to diversity aware influence maximization

   Yadav, Prateek, and Rajkumar, Arun

   GCLR AAAI 2021

   Abs Bib PDF

   Several real-world problems including network influence maximization, rank aggregation, etc. can be posed as problems that output a good ranking of a set of items. We introduce a unifying algorithmic framework based on a novel notion called rank refinement of monotonic set functions to tackle such problems. We prove that under very mild monotonicity assumptions the proposed algorithm converges to a stable ranking. We show that IMRank, a highly scalable influence maximization algorithm can be derived as a special case of our framework. By careful choice of the monotonic set functions, we derive novel generalizations of IMRank that balances the influence and diversity of the top-ranked nodes without compromising on scalability. We provide extensive experimental analysis on both synthetic data-sets based on stochastic block models and large scale real-world data-sets to demonstrate the efficacy of the proposed framework.

   @article{RR,
     abbr = {GCLR AAAI},
     bibtex_show = {true},
     selected = {false},
     title = {Rank Refinement: An Algorithmic framework with applications to diversity aware influence maximization},
     author = {{Yadav}, Prateek and {Rajkumar}, Arun},
     journal = {GCLR AAAI},
     year = {2021},
     pdf = {rr.pdf}
   }

2020

1. CIKM

   NHP: Neural Hypergraph Link Prediction

   Yadati, Naganand, Nitin, Vikram, Nimishakavi, Madhav, Yadav, Prateek, Louis, Anand, and Talukdar, Partha

   In Proceedings of the 29th ACM International Conference on Information & Knowledge Management 2020

   Abs Bib PDF

   Link prediction insimple graphs is a fundamental problem in which new links between vertices are predicted based on the observed structure of the graph. However, in many real-world applications, there is a need to model relationships among vertices that go beyond pairwise associations. For example, in a chemical reaction, relationship among the reactants and products is inherently higher-order. Additionally, there is a need to represent the direction from reactants to products. Hypergraphs provide a natural way to represent such complex higher-order relationships. Graph Convolutional Network (GCN) has recently emerged as a powerful deep learning-based approach for link prediction over simple graphs. However, their suitability for link prediction in hypergraphs is underexplored – we fill this gap in this paper and propose Neural Hyperlink Predictor (NHP). NHP adapts GCNs for link prediction in hypergraphs. We propose two variants of NHP – NHP-U and NHP-D – for link prediction over undirected and directed hypergraphs, respectively. To the best of our knowledge, NHP-D is the first-ever method for link prediction over directed hypergraphs. An important feature of NHP is that it can also be used for hyperlinks in which dissimilar vertices interact (e.g. acids reacting with bases). Another attractive feature of NHP is that it can be used to predict unseen hyperlinks at test time (inductive hyperlink prediction). Through extensive experiments on multiple real-world datasets, we show NHP’s effectiveness.

   @inproceedings{10.1145/3340531.3411870,
     abbr = {CIKM},
     bibtex_show = {true},
     selected = {false},
     author = {Yadati, Naganand and Nitin, Vikram and Nimishakavi, Madhav and Yadav, Prateek and Louis, Anand and Talukdar, Partha},
     title = {NHP: Neural Hypergraph Link Prediction},
     year = {2020},
     isbn = {9781450368599},
     publisher = {Association for Computing Machinery},
     address = {New York, NY, USA},
     pdf = {https://doi.org/10.1145/3340531.3411870},
     doi = {10.1145/3340531.3411870},
     booktitle = {Proceedings of the 29th ACM International Conference on Information & Knowledge Management},
     pages = {1705–1714},
     numpages = {10},
     keywords = {directed hypergraph, graph neural network, knowledge graph canonicalisation, link prediction},
     location = {Virtual Event, Ireland},
     series = {CIKM '20}
   }

2019

1. NeurIPS

   HyperGCN: A New Method For Training Graph Convolutional Networks on Hypergraphs

   Yadati, Naganand, Nimishakavi, Madhav, Yadav, Prateek, Nitin, Vikram, Louis, Anand, and Talukdar, Partha

   In Advances in Neural Information Processing Systems 2019

   Abs arXiv Bib Supp Code

   In many real-world network datasets such as co-authorship, co-citation, email communication, etc., relationships are complex and go beyond pairwise. Hypergraphs provide a flexible and natural modeling tool to model such complex relationships. The obvious existence of such complex relationships in many real-world networks naturaly motivates the problem of learning with hypergraphs. A popular learning paradigm is hypergraph-based semi-supervised learning (SSL) where the goal is to assign labels to initially unlabeled vertices in a hypergraph. Motivated by the fact that a graph convolutional network (GCN) has been effective for graph-based SSL, we propose HyperGCN, a novel GCN for SSL on attributed hypergraphs. Additionally, we show how HyperGCN can be used as a learning-based approach for combinatorial optimisation on NP-hard hypergraph problems. We demonstrate HyperGCN’s effectiveness through detailed experimentation on real-world hypergraphs. We have made HyperGCN’s source code available to foster reproducible research.

   @inproceedings{NEURIPS2019_1efa39bc,
     abbr = {NeurIPS},
     bibtex_show = {true},
     selected = {true},
     author = {Yadati, Naganand and Nimishakavi, Madhav and Yadav, Prateek and Nitin, Vikram and Louis, Anand and Talukdar, Partha},
     booktitle = {Advances in Neural Information Processing Systems},
     editor = {Wallach, H. and Larochelle, H. and Beygelzimer, A. and d\textquotesingle Alch\'{e}-Buc, F. and Fox, E. and Garnett, R.},
     pages = {},
     publisher = {Curran Associates, Inc.},
     title = {HyperGCN: A New Method For Training Graph Convolutional Networks on Hypergraphs},
     url = {https://proceedings.neurips.cc/paper/2019/file/1efa39bcaec6f3900149160693694536-Paper.pdf},
     volume = {32},
     year = {2019},
     arxiv = {1809.02589},
     code = {https://github.com/malllabiisc/HyperGCN},
     supp = {https://papers.nips.cc/paper/2019/hash/1efa39bcaec6f3900149160693694536-Abstract.html}
   }

2. AISTATS

   Lovasz Convolutional Networks

   Yadav, Prateek, Nimishakavi, Madhav, Yadati, Naganand, Vashishth, Shikhar, Rajkumar, Arun, and Talukdar, Partha

   In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics 2019

   Abs arXiv Bib PDF Code Poster

   Semi-supervised learning on graph structured data has received significant attention with the recent introduction of Graph Convolution Networks (GCN). While traditional methods have focused on optimizing a loss augmented with Laplacian regularization framework, GCNs perform an implicit Laplacian type regularization to capture local graph structure. In this work, we propose Lovasz Convolutional Network (LCNs) which are capable of incorporating global graph properties. LCNs achieve this by utilizing Lovasz’s orthonormal embeddings of the nodes. We analyse local and global properties of graphs and demonstrate settings where LCNs tend to work better than GCNs. We validate the proposed method on standard random graph models such as stochastic block models (SBM) and certain community structure based graphs where LCNs outperform GCNs and learn more intuitive embeddings. We also perform extensive binary and multi-class classification experiments on real world datasets to demonstrate LCN’s effectiveness. In addition to simple graphs, we also demonstrate the use of LCNs on hyper-graphs by identifying settings where they are expected to work better than GCNs.

   @inproceedings{pmlr-v89-yadav19a,
     abbr = {AISTATS},
     bibtex_show = {true},
     selected = {true},
     code = {https://github.com/malllabiisc/lcn},
     poster = {https://drive.google.com/file/d/1j78XF6FpDp11Ubo1YI-wFIBV-52d2HZ1/view?usp=sharing},
     arxiv = {1805.11365},
     title = {Lovasz Convolutional Networks},
     author = {Yadav, Prateek and Nimishakavi, Madhav and Yadati, Naganand and Vashishth, Shikhar and Rajkumar, Arun and Talukdar, Partha},
     booktitle = {Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics},
     pages = {1978--1987},
     year = {2019},
     editor = {Chaudhuri, Kamalika and Sugiyama, Masashi},
     volume = {89},
     series = {Proceedings of Machine Learning Research},
     month = {16--18 Apr},
     publisher = {PMLR},
     pdf = {http://proceedings.mlr.press/v89/yadav19a/yadav19a.pdf},
     url = {https://proceedings.mlr.press/v89/yadav19a.html}
   }

3. ACL

   Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks

   Vashishth, Shikhar, Yadav, Prateek*, Bhandari, Manik*, Rai, Piyush, Bhattacharyya, Chiranjib, and Talukdar, Partha

   In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics 2019

   Abs arXiv Bib Supp Code Poster

   Word embeddings have been widely adopted across several NLP applications. Most existing word embedding methods utilize sequential context of a word to learn its embedding. While there have been some attempts at utilizing syntactic context of a word, such methods result in an explosion of the vocabulary size. In this paper, we overcome this problem by proposing SynGCN, a flexible Graph Convolution based method for learning word embeddings. SynGCN utilizes the dependency context of a word without increasing the vocabulary size. Word embeddings learned by SynGCN outperform existing methods on various intrinsic and extrinsic tasks and provide an advantage when used with ELMo. We also propose SemGCN, an effective framework for incorporating diverse semantic knowledge for further enhancing learned word representations. We make the source code of both models available to encourage reproducible research.

   @inproceedings{vashishth-etal-2019-incorporating,
     abbr = {ACL},
     bibtex_show = {true},
     selected = {true},
     arxiv = {1809.04283},
     title = {Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks},
     author = {Vashishth, Shikhar and Yadav, Prateek* and Bhandari, Manik* and Rai, Piyush and Bhattacharyya, Chiranjib and Talukdar, Partha},
     booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
     month = jul,
     year = {2019},
     address = {Florence, Italy},
     publisher = {Association for Computational Linguistics},
     url = {https://aclanthology.org/P19-1320},
     doi = {10.18653/v1/P19-1320},
     pages = {3308--3318},
     code = {https://github.com/malllabiisc/WordGCN},
     supp = {https://drive.google.com/file/d/18ZUlrAF1kvvG7LaZcet20tSZZwlLVLds/view?usp=sharing},
     poster = {https://drive.google.com/file/d/14DMn5d2r06lvruJkpEo2TsVgwyqxQBLz/view?usp=sharing}
   }

4. AISTATS

   Confidence-based Graph Convolutional Networks for Semi-Supervised Learning

   Vashishth, Shikhar*, Yadav, Prateek*, Bhandari, Manik, and Talukdar, Partha

   In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics 2019

   Abs arXiv Bib PDF Code

   Predicting properties of nodes in a graph is an important problem with applications in a variety of domains. Graph-based Semi Supervised Learning (SSL) methods aim to address this problem by labeling a small subset of the nodes as seeds, and then utilizing the graph structure to predict label scores for the rest of the nodes in the graph. Recently, Graph Convolutional Networks (GCNs) have achieved impressive performance on the graph-based SSL task. In addition to label scores, it is also desirable to have confidence scores associated with them. Unfortunately, confidence estimation in the context of GCN has not been previously explored. We fill this important gap in this paper and propose ConfGCN, which estimates labels scores along with their confidences jointly in GCN-based setting. ConfGCN uses these estimated confidences to determine the influence of one node on another during neighborhood aggregation, thereby acquiring anisotropic capabilities. Through extensive analysis and experiments on standard benchmarks, we find that ConfGCN is able to outperform state-of-the-art baselines. We have made ConfGCN’s source code available to encourage reproducible research.

   @inproceedings{pmlr-v89-vashishth19a,
     abbr = {AISTATS},
     bibtex_show = {false},
     selected = {false},
     arxiv = {1901.08255},
     code = {https://github.com/malllabiisc/ConfGCN},
     title = {Confidence-based Graph Convolutional Networks for Semi-Supervised Learning},
     author = {Vashishth, Shikhar* and Yadav, Prateek* and Bhandari, Manik and Talukdar, Partha},
     booktitle = {Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics},
     pages = {1792--1801},
     year = {2019},
     editor = {Chaudhuri, Kamalika and Sugiyama, Masashi},
     volume = {89},
     series = {Proceedings of Machine Learning Research},
     month = {16--18 Apr},
     publisher = {PMLR},
     pdf = {http://proceedings.mlr.press/v89/vashishth19a/vashishth19a.pdf},
     url = {https://proceedings.mlr.press/v89/vashishth19a.html}
   }