Li "Harry" Zhang 张力
About Me
I am a third-year PhD student researcher focusing on Natural Language Processing, mentored by Prof. Chris Callison-Burch at the University of Pennsylvania. I was an undergraduate at the University of Michigan in 2018, previously mentored by Prof. Rada Mihalcea and Prof. Dragomir Radev. I'm also a drummer, musician, producer, and video creator.
My area of focus is on procedures, like recipes or instructions. I teach machines to understand and reason about them.
zharry@seas.upenn.edu
Work Experience
Research Intern
2019, 2021
Teaching Assistant
2016 - 2020
Summer Analyst in Technology
2017
Education
University of Pennsylvania
Ph.D.; GPA: 3.92/4.00; In progress
University of Michigan
B.S.E.; Class of 2018; GPA: 3.82/4.00
Shenzhen Middle School
High School Diploma; Class of 2015; GPA: 4.23/4.30
Service
Reviewer of COLING 2022
Reviewer of ARR since October 2021
Reviewer of LREC 2022
Session chair of AACL-IJCNLP 2020
Co-organizer of CLUNCH in 2020, Penn's NLP seminar series
Reviewer of COLING 2020
Reviewer of Computer Speech and Language 2018
Research
Reasoning about Procedures with Natural Language Processing
A procedure consists of a goal and some steps. To "record music" you need to "buy suitable equipments"; such is a goal-step relation. But before you "buy suitable equipments", you should "plan out your budget"; such is a step-step temporal relation. We use wikiHow to build datasets to train and test models that can reason about such relations. Our models show strong zero- and few-shot results on other tasks, especially on intent detection in dialogs. We also extend this task to multimodal settings.
[6] Reasoning about Goals, Steps, and Temporal Ordering with WikiHow; Li Zhang*Equal contribution, Qing Lyu*Equal contribution and Chris Callison-Burch; in EMNLP 2020; presented at Workshop on Enormous Language Models at ICLR 2021.Paper BibTeX Repo
@inproceedings{zhang-etal-2020-reasoning,
title = "Reasoning about Goals, Steps, and Temporal Ordering with {W}iki{H}ow",
author = "Zhang, Li and
Lyu, Qing and
Callison-Burch, Chris",
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
month = nov,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.emnlp-main.374",
pages = "4630--4639",
}
[16] Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models; (441 authors in total) ... Li Zhang*Equal contribution, Qing Lyu*Equal contribution and Chris Callison-Burch; in arXiv.Paper
[7] Intent Detection with WikiHow; Li Zhang, Qing Lyu and Chris Callison-Burch; in AACL-IJCNLP 2020.Paper BibTeX Repo
@inproceedings{zhang-etal-2020-intent,
title = "Intent Detection with {W}iki{H}ow",
author = "Zhang, Li and
Lyu, Qing and
Callison-Burch, Chris",
booktitle = "Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing",
month = dec,
year = "2020",
address = "Suzhou, China",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.aacl-main.35",
pages = "328--333",
abstract = "Modern task-oriented dialog systems need to reliably understand users{'} intents. Intent detection is even more challenging when moving to new domains or new languages, since there is little annotated data. To address this challenge, we present a suite of pretrained intent detection models which can predict a broad range of intended goals from many actions because they are trained on wikiHow, a comprehensive instructional website. Our models achieve state-of-the-art results on the Snips dataset, the Schema-Guided Dialogue dataset, and all 3 languages of the Facebook multilingual dialog datasets. Our models also demonstrate strong zero- and few-shot performance, reaching over 75{\%} accuracy using only 100 training examples in all datasets.",
}
[9] Visual Goal-Step Inference using wikiHow; Yue Yang, Artemis Panagopoulou, Qing Lyu, Li Zhang, Mark Yatskar and Chris Callison-Burch; In EMNLP 2021; presented at the 2nd Workshop on Advances in Language and Vision Research at NAACL 2021.Paper BibTeX
@inproceedings{yang-etal-2021-visual,
title = "Visual Goal-Step Inference using wiki{H}ow",
author = "Yang, Yue and
Panagopoulou, Artemis and
Lyu, Qing and
Zhang, Li and
Yatskar, Mark and
Callison-Burch, Chris",
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2021",
address = "Online and Punta Cana, Dominican Republic",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.emnlp-main.165",
pages = "2167--2179",
abstract = "Understanding what sequence of steps are needed to complete a goal can help artificial intelligence systems reason about human activities. Past work in NLP has examined the task of goal-step inference for text. We introduce the visual analogue. We propose the Visual Goal-Step Inference (VGSI) task, where a model is given a textual goal and must choose which of four images represents a plausible step towards that goal. With a new dataset harvested from wikiHow consisting of 772,277 images representing human actions, we show that our task is challenging for state-of-the-art multimodal models. Moreover, the multimodal representation learned from our data can be effectively transferred to other datasets like HowTo100m, increasing the VGSI accuracy by 15 - 20{\%}. Our task will facilitate multimodal reasoning about procedural events.",
}
One of the holy-grails of procedural knowledge is for models to construct complete, high-quality procedures from scratch. We push the limit of our models and explore this difficult task in 18 languages, using techniques in both information retrieval and generation by language models. The task turns out extremely challenging, but not impossible.
[8] Goal-Oriented Script Construction; Qing Lyu*Equal contribution, Li Zhang*Equal contribution and Chris Callison-Burch; in INLG 2021.Paper BibTeX Repo
@inproceedings{lyu-etal-2021-goal,
title = "Goal-Oriented Script Construction",
author = "Lyu, Qing and
Zhang, Li and
Callison-Burch, Chris",
booktitle = "Proceedings of the 14th International Conference on Natural Language Generation",
month = aug,
year = "2021",
address = "Aberdeen, Scotland, UK",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.inlg-1.19",
pages = "184--200",
abstract = "The knowledge of scripts, common chains of events in stereotypical scenarios, is a valuable asset for task-oriented natural language understanding systems. We propose the Goal-Oriented Script Construction task, where a model produces a sequence of steps to accomplish a given goal. We pilot our task on the first multilingual script learning dataset supporting 18 languages collected from wikiHow, a website containing half a million how-to articles. For baselines, we consider both a generation-based approach using a language model and a retrieval-based approach by first retrieving the relevant steps from a large candidate pool and then ordering them. We show that our task is practical, feasible but challenging for state-of-the-art Transformer models, and that our methods can be readily deployed for various other datasets and domains with decent zero-shot performance.",
}
While all our previous work has treated procedures as a flat structure, procedural events are inherently hierarchical. To "become a great musician", you need to "practice", which in turn requires "making effective plans". We constructed an event hierarchy from wikiHow and showed that it efficiently aided users perform daily tasks and improved performance of downstream tasks such as video retrieval.
[10] Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data; Shuyan Zhou*Equal contribution, Li Zhang*Equal contribution, Yue Yang, Qing Lyu, Pengcheng Yin, Chris Callison-Burch and Graham Neubig; in ACL 2022.Paper BibTeX Demo
@inproceedings{zhou-etal-2022-show,
title = "Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Data",
author = "Zhou, Shuyan and
Zhang, Li and
Yang, Yue and
Lyu, Qing and
Yin, Pengcheng and
Callison-Burch, Chris and
Neubig, Graham",
booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
month = may,
year = "2022",
address = "Dublin, Ireland",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.acl-long.214",
pages = "2998--3012",
abstract = "Procedures are inherently hierarchical. To {``}make videos{''}, one may need to {``}purchase a camera{''}, which in turn may require one to {``}set a budget{''}. While such hierarchical knowledge is critical for reasoning about complex procedures, most existing work has treated procedures as shallow structures without modeling the parent-child relation. In this work, we attempt to construct an open-domain hierarchical knowledge-base (KB) of procedures based on wikiHow, a website containing more than 110k instructional articles, each documenting the steps to carry out a complex procedure. To this end, we develop a simple and efficient method that links steps (e.g., {``}purchase a camera{''}) in an article to other articles with similar goals (e.g., {``}how to choose a camera{''}), recursively constructing the KB. Our method significantly outperforms several strong baselines according to automatic evaluation, human judgment, and application to downstream tasks such as instructional video retrieval.",
}
To see how our research on procedures can be applied, I co-led University of Pennsylvania's effort to participate in the Alexa Prize TaskBot challenge, where we build a dialog system to help users do household tasks.
[14] QuakerBot: A Household Dialog System Powered by Large Language Models; Artemis Panagopoulou, Manni Arora, Li Zhang, (9 more), Chris Callison-Burch, Mark Yatskar; To be published.Paper BibTeX
@TODO
I wrote a tutorial on reasoning about procedures.
[15] Reasoning about Procedures with Natural Language Processing: A Tutorial; Li Zhang; in arXiv.Paper BibTeX
@misc{https://doi.org/10.48550/arxiv.2205.07455,
doi = {10.48550/ARXIV.2205.07455},
url = {https://arxiv.org/abs/2205.07455},
author = {Zhang, Li},
keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {Reasoning about Procedures with Natural Language Processing: A Tutorial},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution Non Commercial Share Alike 4.0 International}
}
Label-Aware Semantic Role Labeling
Semantic role labeling answers the question of "who did what to whom, when and how", extracting important information about a predicate. While previous work has treated the semantic role labels as symbolic, we explicitly use their definitions and advance state-of-the-art with some limitations.
[11] Label Definitions Improve Semantic Role Labeling; Li Zhang, Ishan Jindal, Yunyao Li; in NAACL 2022.BibTeX Repo
@inproceedings{zhang-etal-2022-label,
title = "Label Definitions Improve Semantic Role Labeling",
author = "Zhang, Li and
Jindal, Ishan and
Li, Yunyao",
booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
month = july,
year = "2022",
address = "Seattle, USA",
publisher = "Association for Computational Linguistics"
}
Probing the Understanding of Recursive Noun Phrases
Do large language models know that a "favorite new movie" is not necessarily a "new favorite movie"?
[12] Is "my favorite new movie" my favorite movie? Probing the Understanding of Recursive Noun Phrases; Qing Lyu, Hua Zheng, Daoxin Li, Li Zhang, Marianna Apidianaki and Chris Callison-Burch; in NAACL 2022.Paper BibTeX Repo
@inproceedings{lyu-etal-2022-favorite,
title = "Is 'my favorite new movie' my favorite movie? Probing the Understanding of Recursive Noun Phrases",
author = "Lyu, Qing and
Zheng, Hua and
Li, Daoxin and
Zhang, Li and
Apidianaki, Marianna and
Callison-Burch, Chris",
booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
month = july,
year = "2022",
address = "Seattle, USA",
publisher = "Association for Computational Linguistics"
}
Text Simplification: Split and Rephrase
Split and Rephrase is a text simplification task to rewrite a complex sentence into several simpler ones. We show that the existing benchmark is too simplistic, developing a rule-based model using no training data which performs on par with the current state-of-the-art neural model. We then propose two new crowdsourced benchmarks with improved quality. We also provide a study on the flaws of BLEU score, and the cost-efficiency of using crowd workers to evaluate models.
[5] Small but Mighty: New Benchmarks for Split and Rephrase; Li Zhang, Huaiyu Zhu, Siddhartha Brahma and Yunyao Li; in EMNLP 2020; a part of the GEM Benchmark.Paper BibTeX Repo
@inproceedings{zhang-etal-2020-small,
title = "Small but Mighty: New Benchmarks for Split and Rephrase",
author = "Zhang, Li and
Zhu, Huaiyu and
Brahma, Siddhartha and
Li, Yunyao",
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
month = nov,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.emnlp-main.91",
pages = "1198--1205",
}
[17] GEMv2: Multilingual NLG Benchmarking in a Single Line of Code; (55 authors in total) ... Li Zhang; in arXiv.Paper
Improving Semantic Similarity with Sentence Embeddings
Recent advancement on neural sentence embeddings show highly competitive performance on semantic similarity tasks. However, the embeddings don't usually just work off-the-shelf, as we show that the transfer learning methodology is crucial to performance. We propose a fine-tuning approach and a multi-label approach which outperforms most alternative transfer learning approaches on semantic similarity tasks, achieving state-of-the-art performance on multiple datasets.
[4] Multi-Label Transfer Learning for Multi-Relational Semantic Similarity; Li Zhang, Steven R. Wilson and Rada Mihalcea; In *SEM 2019. Paper BibTeX Slides
@inproceedings{zhang-etal-2019-multi,
title = "Multi-Label Transfer Learning for Multi-Relational Semantic Similarity",
author = "Zhang, Li and
Wilson, Steven and
Mihalcea, Rada",
booktitle = "Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*{SEM} 2019)",
month = jun,
year = "2019",
address = "Minneapolis, Minnesota",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/S19-1005",
pages = "44--50",
abstract = "Multi-relational semantic similarity datasets define the semantic relations between two short texts in multiple ways, e.g., similarity, relatedness, and so on. Yet, all the systems to date designed to capture such relations target one relation at a time. We propose a multi-label transfer learning approach based on LSTM to make predictions for several relations simultaneously and aggregate the losses to update the parameters. This multi-label regression approach jointly learns the information provided by the multiple relations, rather than treating them as separate tasks. Not only does this approach outperform the single-task approach and the traditional multi-task learning approach, but it also achieves state-of-the-art performance on all but one relation of the Human Activity Phrase dataset.",
}
[3] Direct Network Transfer: Transfer Learning of Sentence Embeddings for Semantic Similarity; Li Zhang, Steven R. Wilson and Rada Mihalcea; in arXiv pre-print; presented at IC2S2 2018.Paper BibTeX Poster
@misc{zhang2018direct,
title={Direct Network Transfer: Transfer Learning of Sentence Embeddings for Semantic Similarity},
author={Li Zhang and Steven R. Wilson and Rada Mihalcea},
year={2018},
eprint={1804.07835},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
Crosslingual Information Retrieval
Sept 2019 - April 2021
This work is a part of the IARPA BETTER project. Given training data of event and entity extraction in one language, the goal is to to transfer to another language without training data. We consider zero-shot transfer and some translation-based baselines.Text-to-SQL Evaluation
This work is a part of the UM-IBM Sapphire project. The goal is to build a dialog system able to answer questions about university course information. While tackling the task of translating natural language to SQL, we identified flaws in the current text-to-SQL evaluation scheme and proposed alternatives. I contributed to building the a text-to-SQL dataset and implementing named entitiy recognition as a preprocessing step.
[1] Improving Text-to-SQL Evaluation Methodology; Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan Dhanalakshmi Ramanathan, Sesh Sadasivam, Rui Zhang and Dragomir Radev; in ACL 2018.Paper BibTeX Repo Poster
@InProceedings{acl18sql,
author = {Catherine Finegan-Dollak\* and Jonathan K. Kummerfeld\* and Li Zhang and Karthik Ramanathan and Sesh Sadasivam and Rui Zhang and Dragomir Radev},
title = {Improving Text-to-SQL Evaluation Methodology},
booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
shortvenue = {ACL},
month = {July},
year = {2018},
address = {Melbourne, Victoria, Australia},
pages = {351--360},
abstract = {To be informative, an evaluation must measure how well systems generalize to realistic unseen data. We identify limitations of and propose improvements to current evaluations of text-to-SQL systems. First, we compare human-generated and automatically generated questions, characterizing properties of queries necessary for real-world applications. To facilitate evaluation on multiple datasets, we release standardized and improved versions of seven existing datasets and one new text-to-SQL dataset. Second, we show that the current division of data into training and test sets measures robustness to variations in the way questions are asked, but only partially tests how well systems generalize to new queries; therefore, we propose a complementary dataset split for evaluation of future work. Finally, we demonstrate how the common practice of anonymizing variables during evaluation removes an important challenge of the task. Our observations highlight key difficulties, and our methodology enables effective measurement of future development.},
url = {http://aclweb.org/anthology/P18-1033},
software = {https://github.com/jkkummerfeld/text2sql-data},
data = {https://github.com/jkkummerfeld/text2sql-data},
}
Entity and Event Extraction from Scratch Using Minimal Training Data
This work is a part of the DARPA AIDA project. From the texts, audios and videos recounting the Russia-Ukraine conflict in 2014, the goal is to extract knowledge elements and generate hypotheses about real-life events. I used named entity recognition, keyword extraction and word embeddings to extract textual entities from the data and assign them with categories from the given ontology.
[2] Entity and Event Extraction from Scratch Using Minimal Training Data; Laura Burdick, Steven R. Wilson, Oana Ignat, Charles F. Welch, Li Zhang, Mingzhe Wang, Jia Deng and Rada Mihalcea; in TAC 2018.Paper BibTeX Poster
@article{Burdick2018EntityAE,
title={Entity and Event Extraction from Scratch Using Minimal Training Data},
author={Laura Burdick and Steven R. Wilson and Oana Ignat and Charles F Welch and Li Zhang and Mingzhe Wang and Jia Deng and Rada Mihalcea},
journal={Theory and Applications of Categories},
year={2018}
}
Cartoon Caption Clustering
Jan 2016 - Apr 2016
In each volume of the New Yorker magazine, there is a comic section where thousands of readers submit funny captions. The goal is to automatically divide them into clusters based on their theme of humor (what they are joking about) using unsupervised learning. Work had been done years ago but the codes were scattered and underdocumented. I as a freshman was in charge of this project, to bring the existing system up to date and to make optimization.
ACL Anthology Network
Sept 2016 - Dec 2016
AAN encompases our corpus of resources on NLP and related fields and the research projects which build upon this corpus. We have collected around 6,500 surveys, tutorials and other resources and created a search engine which allows users to easily browse these resources. I helped build and maintain this power anthology with information regarding numerous papers included in top NLP venues. It features paper citation, author citation, and author collaboration, etc.
Music
I play, record and produce music regularly. I enjoy genres ranging from progressive metal to pop to jazz. I'm passionate about rearranging and recreating soundtracks from video games and animes (ACG). I take pride in my 10k-subscriber channel on Bilibili. My music can be found on all major streaming platforms and YouTube.
To view in HD, hover on a video and click on 去bilibili观看.