Journal of Data Science logo


Login Register

  1. Home
  2. Issues
  3. Volume 21, Issue 2 (2023): Special Issue: Symposium Data Science and Statistics 2022
  4. Identifying Drone Web Sites in Multiple ...

Journal of Data Science

Submit your article Information
  • Article info
  • More
    Article info

Identifying Drone Web Sites in Multiple Countries and Languages with a Single Model
Volume 21, Issue 2 (2023): Special Issue: Symposium Data Science and Statistics 2022, pp. 225–238
Piet Daas † ORCID icon link to view author Piet Daas details   Blanca de Miguel ORCID icon link to view author Blanca de Miguel details   Maria de Miguel ORCID icon link to view author Maria de Miguel details  

Authors

 
Placeholder
https://doi.org/10.6339/23-JDS1087
Pub. online: 26 January 2023      Type: Data Science In Action      Open accessOpen Access

† The views expressed in this paper are those of the authors and do not necessarily reflect the policies of Statistics Netherlands.

Received
25 July 2022
Accepted
17 January 2023
Published
26 January 2023

Abstract

A text-based, bag-of-words, model was developed to identify drone company websites for multiple European countries in different languages. A collection of Spanish drone and non-drone websites was used for initial model development. Various classification methods were compared. Supervised logistic regression (L2-norm) performed best with an accuracy of 87% on the unseen test set. The accuracy of the later model improved to 88% when it was trained on texts in which all Spanish words were translated into English. Retraining the model on texts in which all typical Spanish words, such as names of cities and regions, and words indicative for specific periods in time, such as the months of the year and days of the week, were removed did not affect the overall performance of the model and made it more generally applicable. Applying the cleaned, completely English word-based, model to a collection of Irish and Italian drone and non-drone websites revealed, after manual inspection, that it was able to detect drone websites in those countries with an accuracy of 82 and 86%, respectively. The classification of Italian texts required the creation of a translation list in which all 1560 English word-based features in the model were translated to their Italian analogs. Because the model had a very high recall, 93, 100, and 97% on Spanish, Irish and Italian drone websites respectively, it was particularly well suited to select potential drone websites in large collections of websites.

References

 
Aggarwal C (2016). Data Mining: The Textbook. Springer, New York.
 
Almeida F, Xexéo G (2019). Word embeddings: A survey. CoRR, arXiv preprint: https://arxiv.org/abs/1901.09069
 
Antonacopoulos A, Hu J (2003). Web document analysis: Challenges and opportunities. World Scientific Publishing Co. Pte. Ltd., Singapore.
 
Apertium (2021). Website of apertium, a free/open-source machine translation platform. http://www.apertium.org.
 
Aweisi A, Arora D, Emby R, Rehman M, Tanev G, Tanev S (2021). Using web text analytics to categorize the business focus of innovative digital health companies. Technology Innovation Management Review, 11(7/8): 65–78.
 
Bergstra J, Bardenet R, Bengio Y, Kégl B (2011). Algorithms for hyper-parameter optimization. Advances in Neural Information Processing Systems 24. Curran Associates, Inc., New York.
 
Beręsewicz M, Pater R (2021). Inferring job vacancies from online job advertisements. Statistical Working papers. Eurostat, Luxembourg.
 
Daas P, de Wolf N (2021). Identifying different types of companies via their website text. In: Symposium on Data Science and Statistics (SDSS). Virtual, June 2-4, 2021.
 
Daas P, Tennekes M, De Miguel B, De Miguel M, Santamarina V, Carausu F (2022). Web intelligence for measuring emerging economic trends: The drone industry Statistical Working papers. Eurostat, Luxembourg.
 
Daas P, van der Doef S (2020). Detecting innovative companies via their website. Statistical Journal of IAOS, 36(4): 1239–1251.
 
De Kunder M (2022). The size of the world wide web (the internet). https://www.worldwidewebsize.com/.
 
Devlin J, Chang MW, Lee K, Toutanova K (2018). Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint: https://arxiv.org/abs/1810.04805, 13 pages.
 
Elkan C, Noto K (2008). Learning classifiers from only positive and unlabeled data. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Y Li, B Liu, S Sarawagi, eds.). Las Vegas, Nevada, USA. August 24–27, 2008, 213–220. ACM.
 
ESSnet (2020). Web page provinding an overview of the experimental statistics developed in the context of essnet big data workpackage C on enterprise characteristics. https://ec.europa.eu/eurostat/cros/content/wpc-experimental-statistics_en.
 
Fasttext (2022). Webpage of fasttext language detect v1.0.3. https://pypi.org/project/fasttext-langdetect/.
 
Florescu D, Karlberg M, Reis F, Rey Del Castillo P, Skaliotis M, Wirthmann A (2014). Will ‘big data’ transform official statistics? Quality in Official Statistics Conference. Vienna, Austria. June 2-5, 2014.
 
Gentzkow M, Kelly B, Taddy M (2019). Text as data. Journal of Economic Literature, 57(3): 535–574.
 
GitHub WIH Drones (2022). Web intelligence hub drone companies. https://github.com/eurostat/wih_drones_companies.
 
Gökk A, Waterworth A, Shapira P (2015). Use of web mining in studying innovation. Scientometrics, 102(1): 653–671.
 
GOPA (2021a). Data Retrieval, Deliverable 2. Report 2 of the project Web Intelligence for Measuring Emerging Economic Trends: The Drone Industry. Eurostat, Luxembourg.
 
GOPA (2021b). Deliverable 1. Report 1 of the project Web Intelligence for Measuring Emerging Economic Trends: The Drone Industry. Eurostat, Luxembourg.
 
Höchtl J, Parycek P, Schöllhammer R (2015). Big data in the policy cycle: Policy decision making in the digital era. J. Org. Comp. Elec. Com., 26(1–2): 147–169.
 
Kitchin R (2015). The opportunities, challenges and risks of big data for official statistics. Statistical Journal of the IAOS, 31(3): 471–481.
 
Kowsari K, Jafari Meimandi K, Heidarysafa M, Mendu S, Barnes L, Brown D (2019). Text classification algorithms: A survey. Information, 10(4).
 
Kühnemann H, van Delden A, Windmeijer D (2020). Exploring a knowledge-based approach to predicting nace codes of enterprises based on web page texts. Statistical Journal of the IAOS, 36(3): 807–821.
 
Larose D, Markov Z (2007). Data Mining the Web: Uncovering Patterns in Web Content, Structure, and Usage. Wiley-Interscience, Hoboken, NJ.
 
Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. (2011). Scikit-learn: Machine learning in python. Journal of Machine Learning Research, 12: 2825–2830.
 
Pires T, Schlinger E, Garrette D (2019). How multilingual is multilingual BERT? In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4996–5001. Association for Computational Linguistics, Florence, Italy.
 
Powell B, Nason G, Elliott D, Mayhew M, Davies J, Winton J (2018). Tracking and modelling prices using web-scraped price microdata: Towards automated daily consumer price index forecasting. Journal of the Royal Statistical Society: Series A, 181(3): 737–756.
 
PUlearn (2021). Website of the pulearn python library v0.07. https://pypi.org/project/pulearn.
 
Rothaermel F (2019). Strategic Management. McGraw-Hill Education, New York.
 
Song M, Wu YF (2008). Handbook of Research on Text and Web Mining Technologies. Information Science Reference, Hershey, NY.
 
United Nations (2014). Fundamental Principles of Official Statistics. United Nations Statistic Division, New York.

PDF XML
PDF XML

Copyright
2023 The Author(s). Published by the School of Statistics and the Center for Applied Statistics, Renmin University of China.
by logo by logo
Open access article under the CC BY license.

Keywords
bag of words classification model multiple languages text

Funding
This research was performed as part of the study “Web intelligence for measuring emerging economic trends: the drone industry” led by GOPA under the framework contract on Methodological Support (Ref. 2018.0086) for Eurostat.

Metrics
since February 2021
540

Article info
views

288

PDF
downloads

Export citation

Copy and paste formatted citation
Placeholder

Download citation in file


Share


RSS

Journal of data science

  • Online ISSN: 1683-8602
  • Print ISSN: 1680-743X

About

  • About journal

For contributors

  • Submit
  • OA Policy
  • Become a Peer-reviewer

Contact us

  • JDS@ruc.edu.cn
  • No. 59 Zhongguancun Street, Haidian District Beijing, 100872, P.R. China
Powered by PubliMill  •  Privacy policy