JDS Journal of Data Science 1683-86021680-743X1680-743X School of Statistics, Renmin University of China JDS1012 10.6339/21-JDS1012 Data Science in Action Augmented Abstractive Summarization with Document-Level Semantic Graph BiQiwei1 LiHaoyuan2 LuKun3 YangHanfanghyang@ruc.edu.cn14∗ School of Statistics, Renmin University of China, Beijing, China T.H. Chan School of Public Health, Harvard University, Boston, MA, USA Department of ORFE, Princeton University, Princeton, New Jersey, USA Center for Applied Statistics, School of Statistics, Renmin University of China, Beijing, China Corresponding author. Email: hyang@ruc.edu.cn. 2021452021193450464 Supplementary Material

Supplementary material online include: data link, python code and an instruction file needed to reproduce the results; an appendix containing additional structures and experiments we have tried. The web link is https://github.com/martin6336/DSGSum.

71220201042021 2021 The Author(s). Published by the School of Statistics and the Center for Applied Statistics, Renmin University of China.2021 Open access article under the CC BY license.

Previous abstractive methods apply sequence-to-sequence structures to generate summary without a module to assist the system to detect vital mentions and relationships within a document. To address this problem, we utilize semantic graph to boost the generation performance. Firstly, we extract important entities from each document and then establish a graph inspired by the idea of distant supervision (Mintz et al., 2009). Then, we combine a Bi-LSTM with a graph encoder to obtain the representation of each graph node. A novel neural decoder is presented to leverage the information of such entity graphs. Automatic and human evaluations show the effectiveness of our technique.

 distant supervise entity extraction graph attention neural network information extraction References Bahdanau D, Cho K, Bengio Y (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint: https://arxiv.org/abs/1409.0473. Banarescu L, Bonial C, Cai S, Georgescu M, Griffitt K, Hermjakob U, et al. (2013). Abstract Meaning Representation for sembanking. In: Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse (S Dipper, M Liakata, A Pareja-Lora, eds.), 178–186. Association for Computational Linguistics, Sofia, Bulgaria. Berg-Kirkpatrick T, Burkett D, Klein D (2012). An empirical investigation of statistical significance in NLP. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (M Pasca, J Henderson, J Tsujii, eds.), 995–1005. Association for Computational Linguistics, Jeju Island, Korea. Celikyilmaz A, Bosselut A, He X, Choi Y (2018). Deep communicating agents for abstractive summarization. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) (A Stent, H Ji, MA Walker, eds.), 1662–1675. Association for Computational Linguistics, New Orleans, Louisiana. Chen YC, Bansal M (2018). Fast abstractive summarization with reinforce-selected sentence rewriting. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Y Miyao, I Gurevych, eds.), 675–686. Association for Computational Linguistics, Melbourne, Australia. Damonte M, Cohen SB (2019). Structural neural encoders for AMR-to-text generation. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (T Solorio, C Doran, J Burstein, eds.), 3649–3658. Association for Computational Linguistics, Minneapolis, Minnesota. Durrett G, Berg-Kirkpatrick T, Klein D (2016). Learning-based single-document summarization with compression and anaphoricity constraints. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1998–2008. Association for Computational Linguistics, Berlin, Germany. Fan L, Yu D, Wang L (2018). Robust neural abstractive summarization systems and evaluation against adversarial information. arXiv preprint: https://arxiv.org/abs/1810.06065 Fernandes P, Allamanis M, Brockschmidt M (2019). Structured neural summarization. Gehrmann S, Deng Y, Rush A (2018). Bottom-up abstractive summarization. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (J Tsujii, J Hockenmaier, D Chiang, E Riloff, eds.), 4098–4109. Association for Computational Linguistics, Brussels, Belgium. Guu K, Lee K, Tung Z, Pasupat P, Chang MW (2020). Realm: Retrieval-augmented language model pre-training. arXiv preprint: https://arxiv.org/abs/2002.08909 Hochreiter S, Schmidhuber J (1997). Long short-term memory. Neural Comput., 9(8): 1735–1780. Honnibal M, Montani I, Van Landeghem S, Boyd A (2020). spaCy: Industrial-strength Natural Language Processing in Python. Zenodo, https://doi.org/10.5281/zenodo.1212303 Koncel-Kedziorski R, Bekal D, Luan Y, Lapata M, Hajishirzi H (2019). Text generation from knowledge graphs with graph transformers. arXiv preprint: https://arxiv.org/abs/1904.02342 Lee K, He L, Lewis M, Zettlemoyer L (2017). End-to-end neural coreference resolution. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (M Palmer, R Hwa, S Riedel, eds.), 188–197. Association for Computational Linguistics, Copenhagen, Denmark. Liu Y, Lapata M (2019). Text summarization with pretrained encoders. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (K Inui, J Jiang, V Ng, X Wan, eds.), 3730–3740. Association for Computational Linguistics, Hong Kong, China. Logan R, Liu NF, Peters ME, Gardner M, Singh S (2019). Barack’s wife Hillary: Using knowledge graphs for fact-aware language modeling. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (A Korhonen, DR Traum, L Màrquez, eds.), 5962–5971. Association for Computational Linguistics, Florence, Italy. Luan Y, He L, Ostendorf M, Hajishirzi H (2018). Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (E Riloff, D Chiang, J Hockenmaier, J Tsujii, eds.), 3219–3232. Association for Computational Linguistics, Brussels, Belgium. Manning C, Surdeanu M, Bauer J, Finkel J, Bethard S, McClosky D (2014). The Stanford CoreNLP natural language processing toolkit. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 55–60. Association for Computational Linguistics, Baltimore, Maryland. Mintz M, Bills S, Snow R, Jurafsky D (2009). Distant supervision for relation extraction without labeled data. In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP (J Wiebe, J Su, K-Y Su, eds.), 1003–1011. Association for Computational Linguistics, Suntec, Singapore. Nallapati R, Zhou B, dos Santos C, Gulçehre Ç, Xiang B (2016). Abstractive text summarization using sequence-to-sequence RNNs and beyond. In: Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning (S Riezler, Y Goldberg, eds.), 280–290. Association for Computational Linguistics, Berlin, Germany. Paulus R, Xiong C, Socher R (2018). A deep reinforced model for abstractive summarization. In: International Conference on Learning Representations. Rush AM, Chopra S, Weston J (2015). A neural attention model for abstractive sentence summarization. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (L Màrquez, C Callison-Burch, J Su, D Pighin, Y Marton, eds.), 379–389. Association for Computational Linguistics, Lisbon, Portugal. See A, Liu PJ, Manning CD (2017). Get to the point: Summarization with pointer-generator networks. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (R Barzilay, M-Y Kan, eds.), 1073–1083. Association for Computational Linguistics, Vancouver, Canada. Seo M, Kembhavi A, Farhadi A, Hajishirzi H (2017). Bidirectional attention flow for machine comprehension. arXiv preprint: https://arxiv.org/abs/1611.01603 Sharma E, Huang L, Hu Z, Wang L (2019). An entity-driven framework for abstractive summarization. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (K Inui, J Jiang, V Ng, X Wan, eds.), 3280–3291. Association for Computational Linguistics, Hong Kong, China. Speer R, Havasi C (2012). Representing general relational knowledge in ConceptNet 5. In: Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12) (N Calzolari, K Choukri, Declerck T, MU Dogan, B Maegaard, J Mariani, J Odijk, S Piperidis,eds.), 3679–3686. European Language Resources Association (ELRA), Istanbul, Turkey. Tay Y, Bahri D, Metzler D, Juan DC, Zhao Z, Zheng C (2020). Synthesizer: Rethinking self-attention in transformer models. arXiv preprint: https://arxiv.org/abs/2005.00743 Trisedya BD, Weikum G, Qi J, Zhang R (2019). Neural relation extraction for knowledge base enrichment. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (A Korhonen, DR Traum, L Màrquez, eds.), 229–240. Association for Computational Linguistics, Florence, Italy. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. (2017). Attention is all you need. In: Advances in Neural Information Processing Systems (I Guyon, UV Luxburg, S Bengio, H Wallach, R Fergus, S Vishwanathan, R Garnett, eds.), volume 30. Curran Associates, Inc. Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, et al. (2019). Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint: https://arxiv.org/abs/1910.03771 Zhang Y, Qi P, Manning CD (2018). Graph convolution over pruned dependency trees improves relation extraction. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (E Riloff, D Chiang, J Hockenmaier, J Tsujii, eds.), 2205–2215. Association for Computational Linguistics, Brussels, Belgium.