Augmented Abstractive Summarization with Document-Level Semantic Graph
Volume 19, Issue 3 (2021), pp. 450–464
Pub. online: 4 May 2021
Type: Data Science In Action
Received
7 December 2020
7 December 2020
Accepted
10 April 2021
10 April 2021
Published
4 May 2021
4 May 2021
Abstract
Previous abstractive methods apply sequence-to-sequence structures to generate summary without a module to assist the system to detect vital mentions and relationships within a document. To address this problem, we utilize semantic graph to boost the generation performance. Firstly, we extract important entities from each document and then establish a graph inspired by the idea of distant supervision (Mintz et al., 2009). Then, we combine a Bi-LSTM with a graph encoder to obtain the representation of each graph node. A novel neural decoder is presented to leverage the information of such entity graphs. Automatic and human evaluations show the effectiveness of our technique.
Supplementary material
Supplementary MaterialSupplementary material online include: data link, python code and an instruction file needed to reproduce the results; an appendix containing additional structures and experiments we have tried. The web link is https://github.com/martin6336/DSGSum.
References
Bahdanau D, Cho K, Bengio Y (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint: https://arxiv.org/abs/1409.0473.
Banarescu L, Bonial C, Cai S, Georgescu M, Griffitt K, Hermjakob U, et al. (2013). Abstract Meaning Representation for sembanking. In: Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse (S Dipper, M Liakata, A Pareja-Lora, eds.), 178–186. Association for Computational Linguistics, Sofia, Bulgaria.
Berg-Kirkpatrick T, Burkett D, Klein D (2012). An empirical investigation of statistical significance in NLP. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (M Pasca, J Henderson, J Tsujii, eds.), 995–1005. Association for Computational Linguistics, Jeju Island, Korea.
Celikyilmaz A, Bosselut A, He X, Choi Y (2018). Deep communicating agents for abstractive summarization. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) (A Stent, H Ji, MA Walker, eds.), 1662–1675. Association for Computational Linguistics, New Orleans, Louisiana.
Chen YC, Bansal M (2018). Fast abstractive summarization with reinforce-selected sentence rewriting. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (Y Miyao, I Gurevych, eds.), 675–686. Association for Computational Linguistics, Melbourne, Australia.
Damonte M, Cohen SB (2019). Structural neural encoders for AMR-to-text generation. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (T Solorio, C Doran, J Burstein, eds.), 3649–3658. Association for Computational Linguistics, Minneapolis, Minnesota.
Durrett G, Berg-Kirkpatrick T, Klein D (2016). Learning-based single-document summarization with compression and anaphoricity constraints. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1998–2008. Association for Computational Linguistics, Berlin, Germany.
Fan L, Yu D, Wang L (2018). Robust neural abstractive summarization systems and evaluation against adversarial information. arXiv preprint: https://arxiv.org/abs/1810.06065
Guu K, Lee K, Tung Z, Pasupat P, Chang MW (2020). Realm: Retrieval-augmented language model pre-training. arXiv preprint: https://arxiv.org/abs/2002.08909
Honnibal M, Montani I, Van Landeghem S, Boyd A (2020). spaCy: Industrial-strength Natural Language Processing in Python. Zenodo, https://doi.org/10.5281/zenodo.1212303
Koncel-Kedziorski R, Bekal D, Luan Y, Lapata M, Hajishirzi H (2019). Text generation from knowledge graphs with graph transformers. arXiv preprint: https://arxiv.org/abs/1904.02342
Liu Y, Lapata M (2019). Text summarization with pretrained encoders. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (K Inui, J Jiang, V Ng, X Wan, eds.), 3730–3740. Association for Computational Linguistics, Hong Kong, China.
Logan R, Liu NF, Peters ME, Gardner M, Singh S (2019). Barack’s wife Hillary: Using knowledge graphs for fact-aware language modeling. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (A Korhonen, DR Traum, L Màrquez, eds.), 5962–5971. Association for Computational Linguistics, Florence, Italy.
Luan Y, He L, Ostendorf M, Hajishirzi H (2018). Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (E Riloff, D Chiang, J Hockenmaier, J Tsujii, eds.), 3219–3232. Association for Computational Linguistics, Brussels, Belgium.
Manning C, Surdeanu M, Bauer J, Finkel J, Bethard S, McClosky D (2014). The Stanford CoreNLP natural language processing toolkit. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 55–60. Association for Computational Linguistics, Baltimore, Maryland.
Mintz M, Bills S, Snow R, Jurafsky D (2009). Distant supervision for relation extraction without labeled data. In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP (J Wiebe, J Su, K-Y Su, eds.), 1003–1011. Association for Computational Linguistics, Suntec, Singapore.
Nallapati R, Zhou B, dos Santos C, Gulçehre Ç, Xiang B (2016). Abstractive text summarization using sequence-to-sequence RNNs and beyond. In: Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning (S Riezler, Y Goldberg, eds.), 280–290. Association for Computational Linguistics, Berlin, Germany.
Rush AM, Chopra S, Weston J (2015). A neural attention model for abstractive sentence summarization. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (L Màrquez, C Callison-Burch, J Su, D Pighin, Y Marton, eds.), 379–389. Association for Computational Linguistics, Lisbon, Portugal.
See A, Liu PJ, Manning CD (2017). Get to the point: Summarization with pointer-generator networks. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (R Barzilay, M-Y Kan, eds.), 1073–1083. Association for Computational Linguistics, Vancouver, Canada.
Seo M, Kembhavi A, Farhadi A, Hajishirzi H (2017). Bidirectional attention flow for machine comprehension. arXiv preprint: https://arxiv.org/abs/1611.01603
Sharma E, Huang L, Hu Z, Wang L (2019). An entity-driven framework for abstractive summarization. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (K Inui, J Jiang, V Ng, X Wan, eds.), 3280–3291. Association for Computational Linguistics, Hong Kong, China.
Speer R, Havasi C (2012). Representing general relational knowledge in ConceptNet 5. In: Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12) (N Calzolari, K Choukri, Declerck T, MU Dogan, B Maegaard, J Mariani, J Odijk, S Piperidis,eds.), 3679–3686. European Language Resources Association (ELRA), Istanbul, Turkey.
Tay Y, Bahri D, Metzler D, Juan DC, Zhao Z, Zheng C (2020). Synthesizer: Rethinking self-attention in transformer models. arXiv preprint: https://arxiv.org/abs/2005.00743
Trisedya BD, Weikum G, Qi J, Zhang R (2019). Neural relation extraction for knowledge base enrichment. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (A Korhonen, DR Traum, L Màrquez, eds.), 229–240. Association for Computational Linguistics, Florence, Italy.
Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, et al. (2019). Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint: https://arxiv.org/abs/1910.03771
Zhang Y, Qi P, Manning CD (2018). Graph convolution over pruned dependency trees improves relation extraction. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (E Riloff, D Chiang, J Hockenmaier, J Tsujii, eds.), 2205–2215. Association for Computational Linguistics, Brussels, Belgium.