Li "Harry" Zhang 张力
About Me
I am a third-year PhD student researcher focusing on Natural Language Processing, mentored by Prof. Chris Callison-Burch at the University of Pennsylvania. I graduated from the University of Michigan in 2018, previously mentored by Prof. Rada Mihalcea and Prof. Dragomir Radev. I'm also a multi-instrumental musician, music producer, and video creator.
CV
University of Pennsylvania
Shenzhen, China
zharry@seas.upenn.edu
Music
I play, record and produce music regularly. I enjoy genres ranging from progressive metal to pop to jazz. I'm passionate about rearranging and recreating soundtracks from video games and animes (ACG). My music can be found on all major streaming platforms and my video channels on Bilibili and YouTube.
ACG music rearranged and remade as metal or rock
ACG music rearranged and remade as jazz, fusion or funk
Band cover of metal songs
Drum covers
Guitar and bass covers
Pool
I used to play pool competitively. I was in the university team and played in intercollegiate tournaments regularly.
Research Highlights
Learning Procedural Knowledge with 
Nov 2019 - Present Semantics Dialog Commonsense reasoning
Events are at the core of natural language, capturing "what is happening" instead of just "what is written down". Procedural events, specifically, are human activities that can be seen as a series of steps to accomplish a certain goal. For example, recipes, manuals to assemble furniture, how-to guides are all about procedrual events. These human-centered events involve idiosyncratic challenges and fascinating benefits.
Our first effort is to extract the procedural events. For example, to "record music" you need to "buy suitable equipments"; such is a goal-step relation. But before you "buy suitable equipments", you should "plan out your budget"; such is a step-step temporal relation. We use wikiHow to build datasets to train and test models that can reason about such relations. Our models show strong zero- and few-shot results on other tasks, especially on intent detection in dialogs.[6][7]
The inference of goal-step relations does not have to be confined to texts. Images and videos can also represent procedural events, with potentially richer information. We extend our task to a multimodal setting, showing the potential to generalize from the visuals in wikiHow to other domains.[9]
One of the holy-grails of procedural knowledge is for models to construct complete, high-quality procedures from scratch. We push the limit of our models and explore this difficult task in 18 languages, using techniques in both information retrieval and generation by language models. The task turns out extremely challenging, but not impossible.[8]
While all our previous work has treated procedures as a flat structure, procedural events are inherently hierarchical. To "become a great musician", you need to "practice", which in turn requires "making effective plans". We constructed an event hierarchy from wikiHow and showed that it efficiently aided users perform daily tasks and improved performance of downstream tasks such as video retrieval.
To show the power of our datasets and models, I co-lead University of Pennsylvania's effort to participate in the Alexa Prize TaskBot challenge, where we build a dialog system to help users do household tasks.
Label-Aware Semantic Role Labeling
May 2021 - Aug 2021 Sentence Semantics
(work done as intern at IBM Research) Semantic role labeling answers the question of "who did what to whom, when and how", extracting important information about a predicate. While previous work has treated the semantic role labels as symbolic, we explicitly use their definitions and advance state-of-the-art with some limitations.
Split and Rephrase: Evaluation Benchmarks and Metrics [5]
Apr 2019 - Jun 2019 Text Simplification Resources and Evaluation
(work done as intern at IBM Research) Split and Rephrase is a text simplification task to rewrite a complex sentence into several simpler ones. We show that the existing benchmark is too simplistic, developing a rule-based model using no training data which performs on par with the current state-of-the-art neural model. We then propose two new crowdsourced benchmarks with improved quality. We also provide a study on the flaws of BLEU score, and the cost-efficiency of using crowd workers to evaluate models.
Sentence Embeddings, Transfer Learning and Semantic Similarity [3][4]
Oct 2017 - Sept 2018 Sentence Semantics
Recent advancement on neural sentence embeddings show highly competitive performance on semantic similarity tasks. However, the embeddings don't usually just work off-the-shelf, as we show that the transfer learning methodology is crucial to performance. We propose a fine-tuning approach and a multi-label approach which outperforms most alternative transfer learning approaches on semantic similarity tasks, achieving state-of-the-art performance on multiple datasets.
Crosslingual Information Retrieval
Sept 2019 - Present Information Extraction
This work is a part of the IARPA BETTER project. Given training data of event and entity extraction in one language, the goal is to to transfer to another language without training data. We consider zero-shot transfer and some translation-based baselines.Academic Advising Dialogue System and Text-to-SQL Generation [1]
Sep 2015 - Apr 2017 Dialogue Resources and Evaluation
This work is a part of the UM-IBM Sapphire project. The goal is to build a dialog system able to answer questions about university course information. While tackling the task of translating natural language to SQL, we identified flaws in the current text-to-SQL evaluation scheme and proposed alternatives. I contributed to building the a text-to-SQL dataset and implementing named entitiy recognition as a preprocessing step.
Active Interpretation of Disparate Alternatives[2]
Jan 2018 - Feb 2019
This work is a part of the DARPA AIDA project. From the texts, audios and videos recounting the Russia-Ukraine conflict in 2014, the goal is to extract knowledge elements and generate hypotheses about real-life events. I used named entity recognition, keyword extraction and word embeddings to extract textual entities from the data and assign them with categories from the given ontology.
Cartoon Caption Clustering
Jan 2016 - Apr 2016
In each volume of the New Yorker magazine, there is a comic section where thousands of readers submit funny captions. The goal is to automatically divide them into clusters based on their theme of humor (what they are joking about) using unsupervised learning. Work had been done years ago but the codes were scattered and underdocumented. I as a freshman was in charge of this project, to bring the existing system up to date and to make optimization.
ACL Anthology Network
Sept 2016 - Dec 2016
AAN encompases our corpus of resources on NLP and related fields and the research projects which build upon this corpus. We have collected around 6,500 surveys, tutorials and other resources and created a search engine which allows users to easily browse these resources. I helped build and maintain this power anthology with information regarding numerous papers included in top NLP venues. It features paper citation, author citation, and author collaboration, etc.
Publications
[9] Visual Goal-Step Inference using wikiHow
Yue Yang, Artemis Panagopoulou, Qing Lyu, Li Zhang, Mark Yatskar and Chris Callison-Burch
Paper BibTeX In EMNLP 2021; presented at the 2nd Workshop on Advances in Language and Vision Research at NAACL 2021.
@inproceedings{yang-etal-2021-visual,
title = "Visual Goal-Step Inference using wikiHow",
author = "Yang, Yue and
Panagopoulou, Artemis and
Lyu, Qing and
Zhang, Li and
Yatskar, Mark and
Callison-Burch, Chris",
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2021",
address = "Punta Cana, Dominican Republic",
publisher = "Association for Computational Linguistics",
abstract = "Procedural events can often be thought of as a high level goal composed of a sequence of steps. Inferring the sub-sequence of steps of a goal can help artificial intelligence systems reason about human activities. Past work in NLP has examined the task of goal-step inference for text. We introduce the visual analogue. We propose the Visual Goal-Step Inference (VGSI) task where a model is given a textual goal and must choose a plausible step towards that goal from among four candidate images. Our task is challenging for state-of-the-art muitimodal models. We introduce a novel dataset harvested from wikiHow that consists of 772,294 images representing human actions. We show that the knowledge learned from our data can effectively transfer to other datasets like HowTo100M, increasing the multiple-choice accuracy by 15\% to 20\%. Our task will facilitate multi-modal reasoning about procedural events.",
}
[8] Goal-Oriented Script Construction
Qing Lyu*Equal contribution, Li Zhang*Equal contribution and Chris Callison-Burch
Paper BibTeX Repo In INLG 2021.
@inproceedings{lyu-etal-2021-goal,
title = "Goal-Oriented Script Construction",
author = "Lyu, Qing and
Zhang, Li and
Callison-Burch, Chris",
booktitle = "Proceedings of the 14th International Conference on Natural Language Generation",
month = aug,
year = "2021",
address = "Aberdeen, Scotland, UK",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.inlg-1.19",
pages = "184--200",
abstract = "The knowledge of scripts, common chains of events in stereotypical scenarios, is a valuable asset for task-oriented natural language understanding systems. We propose the Goal-Oriented Script Construction task, where a model produces a sequence of steps to accomplish a given goal. We pilot our task on the first multilingual script learning dataset supporting 18 languages collected from wikiHow, a website containing half a million how-to articles. For baselines, we consider both a generation-based approach using a language model and a retrieval-based approach by first retrieving the relevant steps from a large candidate pool and then ordering them. We show that our task is practical, feasible but challenging for state-of-the-art Transformer models, and that our methods can be readily deployed for various other datasets and domains with decent zero-shot performance.",
}
[7] Intent Detection with WikiHow
Li Zhang, Qing Lyu and Chris Callison-Burch
Paper BibTeX Repo In AACL-IJCNLP 2020.
@inproceedings{zhang-etal-2020-intent,
title = "Intent Detection with {W}iki{H}ow",
author = "Zhang, Li and
Lyu, Qing and
Callison-Burch, Chris",
booktitle = "Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing",
month = dec,
year = "2020",
address = "Suzhou, China",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.aacl-main.35",
pages = "328--333",
abstract = "Modern task-oriented dialog systems need to reliably understand users{'} intents. Intent detection is even more challenging when moving to new domains or new languages, since there is little annotated data. To address this challenge, we present a suite of pretrained intent detection models which can predict a broad range of intended goals from many actions because they are trained on wikiHow, a comprehensive instructional website. Our models achieve state-of-the-art results on the Snips dataset, the Schema-Guided Dialogue dataset, and all 3 languages of the Facebook multilingual dialog datasets. Our models also demonstrate strong zero- and few-shot performance, reaching over 75{\%} accuracy using only 100 training examples in all datasets.",
}
[6] Reasoning about Goals, Steps, and Temporal Ordering with WikiHow
Li Zhang*Equal contribution, Qing Lyu*Equal contribution and Chris Callison-Burch
Paper BibTeX Repo In EMNLP 2020; presented at Workshop on Enormous Language Models at ICLR 2021.
@inproceedings{zhang-etal-2020-reasoning,
title = "Reasoning about Goals, Steps, and Temporal Ordering with {W}iki{H}ow",
author = "Zhang, Li and
Lyu, Qing and
Callison-Burch, Chris",
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
month = nov,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.emnlp-main.374",
pages = "4630--4639",
}
[5] Small but Mighty: New Benchmarks for Split and Rephrase
Li Zhang, Huaiyu Zhu, Siddhartha Brahma and Yunyao Li
Paper BibTeX Repo In EMNLP 2020.
@inproceedings{zhang-etal-2020-small,
title = "Small but Mighty: New Benchmarks for Split and Rephrase",
author = "Zhang, Li and
Zhu, Huaiyu and
Brahma, Siddhartha and
Li, Yunyao",
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
month = nov,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.emnlp-main.91",
pages = "1198--1205",
}
[4] Multi-Label Transfer Learning for Multi-Relational Semantic Similarity
Li Zhang, Steven R. Wilson and Rada Mihalcea
Paper BibTeX Slides In *SEM 2019.
@inproceedings{zhang-etal-2019-multi,
title = "Multi-Label Transfer Learning for Multi-Relational Semantic Similarity",
author = "Zhang, Li and
Wilson, Steven and
Mihalcea, Rada",
booktitle = "Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*{SEM} 2019)",
month = jun,
year = "2019",
address = "Minneapolis, Minnesota",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/S19-1005",
pages = "44--50",
abstract = "Multi-relational semantic similarity datasets define the semantic relations between two short texts in multiple ways, e.g., similarity, relatedness, and so on. Yet, all the systems to date designed to capture such relations target one relation at a time. We propose a multi-label transfer learning approach based on LSTM to make predictions for several relations simultaneously and aggregate the losses to update the parameters. This multi-label regression approach jointly learns the information provided by the multiple relations, rather than treating them as separate tasks. Not only does this approach outperform the single-task approach and the traditional multi-task learning approach, but it also achieves state-of-the-art performance on all but one relation of the Human Activity Phrase dataset.",
}
[3] Direct Network Transfer: Transfer Learning of Sentence Embeddings for Semantic Similarity
Li Zhang, Steven R. Wilson and Rada Mihalcea
Paper BibTeX Poster In arXiv pre-print; presented at IC2S2 2018.
@misc{zhang2018direct,
title={Direct Network Transfer: Transfer Learning of Sentence Embeddings for Semantic Similarity},
author={Li Zhang and Steven R. Wilson and Rada Mihalcea},
year={2018},
eprint={1804.07835},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
[2] Entity and Event Extraction from Scratch Using Minimal Training Data
Laura Burdick, Steven R. Wilson, Oana Ignat, Charles F. Welch, Li Zhang, Mingzhe Wang, Jia Deng and Rada Mihalcea
Paper BibTeX Poster In TAC 2018.
@article{Burdick2018EntityAE,
title={Entity and Event Extraction from Scratch Using Minimal Training Data},
author={Laura Burdick and Steven R. Wilson and Oana Ignat and Charles F Welch and Li Zhang and Mingzhe Wang and Jia Deng and Rada Mihalcea},
journal={Theory and Applications of Categories},
year={2018}
}
[1] Improving Text-to-SQL Evaluation Methodology
Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan Dhanalakshmi Ramanathan, Sesh Sadasivam, Rui Zhang and Dragomir Radev
Paper BibTeX Repo Poster In ACL 2018.
@InProceedings{acl18sql,
author = {Catherine Finegan-Dollak\* and Jonathan K. Kummerfeld\* and Li Zhang and Karthik Ramanathan and Sesh Sadasivam and Rui Zhang and Dragomir Radev},
title = {Improving Text-to-SQL Evaluation Methodology},
booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
shortvenue = {ACL},
month = {July},
year = {2018},
address = {Melbourne, Victoria, Australia},
pages = {351--360},
abstract = {To be informative, an evaluation must measure how well systems generalize to realistic unseen data. We identify limitations of and propose improvements to current evaluations of text-to-SQL systems. First, we compare human-generated and automatically generated questions, characterizing properties of queries necessary for real-world applications. To facilitate evaluation on multiple datasets, we release standardized and improved versions of seven existing datasets and one new text-to-SQL dataset. Second, we show that the current division of data into training and test sets measures robustness to variations in the way questions are asked, but only partially tests how well systems generalize to new queries; therefore, we propose a complementary dataset split for evaluation of future work. Finally, we demonstrate how the common practice of anonymizing variables during evaluation removes an important challenge of the task. Our observations highlight key difficulties, and our methodology enables effective measurement of future development.},
url = {http://aclweb.org/anthology/P18-1033},
software = {https://github.com/jkkummerfeld/text2sql-data},
data = {https://github.com/jkkummerfeld/text2sql-data},
}
Work Experience
Research Intern @ IBM Research
2019, 2021
I did NLP research and software development on semantic role labeling, and previously, text simplification.
Teaching Assistant
2016 - 2020
At Penn, I instructed CIS 530: Computational Linguistics (Winter, Fall 2020). At Michigan, I instructed EECS 595: Natural Language Processing (Fall 2018) and EECS 280: Programming and Introductory Data Structures (Winter, Fall 2016).
Summer Analyst in Technology @ Goldman Sachs
2017
I performed software engineering, data analytics and machine learning.
Education
University of Pennsylvania
Philadephia, PA, USA
Ph.D. Computer and Information Science; In progress
GPA: 3.92/4.00
University of Michigan
Ann Arbor, MI, USA
B.S.E. Computer Science; Class of 2018
GPA: 3.82/4.00 summa cum laude
Shenzhen Middle School
Shenzhen, China
High School Diploma; Class of 2015
GPA: 4.23/4.30