Harvesting bibliography multi-thread, safe and ethical web crawling

Harry Tursulistyono Yani Achsan; Wahyu Catur Wibowo; Wahyuningdiah Trisari Harsanti Putri; M. Muhtar Baswara Achsan; Quintin Kumia Dikara Barcah

doi:10.1109/ICACSIS.2018.8618262

Harvesting bibliography multi-thread, safe and ethical web crawling

Harry Tursulistyono Yani Achsan, Wahyu Catur Wibowo, Wahyuningdiah Trisari Harsanti Putri, M. Muhtar Baswara Achsan, Quintin Kumia Dikara Barcah

Department of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Citation (Scopus)

Abstract

Web mining is an important techniques for it enables extraction of data and information from the web for further needs. Although there are a lot of web databases that holds bibliography data, The Online Computer Library Center (OCLC) owns the largest web databases on bibliography in the world. A large number of bibliography data, surely needs more than manual downloading. In this research we conduct an experimentation to harvest bibliography data using multi-threading process that is fast, safe and ethical. Using C# programming language and Visual Studio IDE, We were successful in harvesting five million bibliography data, without being penalized by the source site.

Original language	English
Title of host publication	2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	355-360
Number of pages	6
ISBN (Electronic)	9781728101354
DOIs	https://doi.org/10.1109/ICACSIS.2018.8618262
Publication status	Published - 2 Jul 2018
Event	10th International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018 - Yogyakarta, Indonesia Duration: 27 Oct 2018 → 28 Oct 2018

Publication series

Name	2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018

Conference

Conference	10th International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018
Country/Territory	Indonesia
City	Yogyakarta
Period	27/10/18 → 28/10/18

Access to Document

10.1109/ICACSIS.2018.8618262

Cite this

Achsan, H. T. Y., Wibowo, W. C., Putri, W. T. H., Achsan, M. M. B., & Barcah, Q. K. D. (2018). Harvesting bibliography multi-thread, safe and ethical web crawling. In 2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018 (pp. 355-360). Article 8618262 (2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICACSIS.2018.8618262

Achsan, Harry Tursulistyono Yani ; Wibowo, Wahyu Catur ; Putri, Wahyuningdiah Trisari Harsanti et al. / Harvesting bibliography multi-thread, safe and ethical web crawling. 2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 355-360 (2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018).

@inproceedings{12684dcdadb94b6ea987e30361d78b6b,

title = "Harvesting bibliography multi-thread, safe and ethical web crawling",

abstract = "Web mining is an important techniques for it enables extraction of data and information from the web for further needs. Although there are a lot of web databases that holds bibliography data, The Online Computer Library Center (OCLC) owns the largest web databases on bibliography in the world. A large number of bibliography data, surely needs more than manual downloading. In this research we conduct an experimentation to harvest bibliography data using multi-threading process that is fast, safe and ethical. Using C# programming language and Visual Studio IDE, We were successful in harvesting five million bibliography data, without being penalized by the source site.",

author = "Achsan, {Harry Tursulistyono Yani} and Wibowo, {Wahyu Catur} and Putri, {Wahyuningdiah Trisari Harsanti} and Achsan, {M. Muhtar Baswara} and Barcah, {Quintin Kumia Dikara}",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE. All Rights Reserved.; 10th International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018 ; Conference date: 27-10-2018 Through 28-10-2018",

year = "2018",

month = jul,

day = "2",

doi = "10.1109/ICACSIS.2018.8618262",

language = "English",

series = "2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "355--360",

booktitle = "2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018",

address = "United States",

}

Achsan, HTY, Wibowo, WC, Putri, WTH, Achsan, MMB & Barcah, QKD 2018, Harvesting bibliography multi-thread, safe and ethical web crawling. in 2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018., 8618262, 2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018, Institute of Electrical and Electronics Engineers Inc., pp. 355-360, 10th International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018, Yogyakarta, Indonesia, 27/10/18. https://doi.org/10.1109/ICACSIS.2018.8618262

Harvesting bibliography multi-thread, safe and ethical web crawling. / Achsan, Harry Tursulistyono Yani; Wibowo, Wahyu Catur; Putri, Wahyuningdiah Trisari Harsanti et al.
2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 355-360 8618262 (2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Harvesting bibliography multi-thread, safe and ethical web crawling

AU - Achsan, Harry Tursulistyono Yani

AU - Wibowo, Wahyu Catur

AU - Putri, Wahyuningdiah Trisari Harsanti

AU - Achsan, M. Muhtar Baswara

AU - Barcah, Quintin Kumia Dikara

PY - 2018/7/2

Y1 - 2018/7/2

N2 - Web mining is an important techniques for it enables extraction of data and information from the web for further needs. Although there are a lot of web databases that holds bibliography data, The Online Computer Library Center (OCLC) owns the largest web databases on bibliography in the world. A large number of bibliography data, surely needs more than manual downloading. In this research we conduct an experimentation to harvest bibliography data using multi-threading process that is fast, safe and ethical. Using C# programming language and Visual Studio IDE, We were successful in harvesting five million bibliography data, without being penalized by the source site.

AB - Web mining is an important techniques for it enables extraction of data and information from the web for further needs. Although there are a lot of web databases that holds bibliography data, The Online Computer Library Center (OCLC) owns the largest web databases on bibliography in the world. A large number of bibliography data, surely needs more than manual downloading. In this research we conduct an experimentation to harvest bibliography data using multi-threading process that is fast, safe and ethical. Using C# programming language and Visual Studio IDE, We were successful in harvesting five million bibliography data, without being penalized by the source site.

UR - http://www.scopus.com/inward/record.url?scp=85062416089&partnerID=8YFLogxK

U2 - 10.1109/ICACSIS.2018.8618262

DO - 10.1109/ICACSIS.2018.8618262

M3 - Conference contribution

AN - SCOPUS:85062416089

T3 - 2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018

SP - 355

EP - 360

BT - 2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 10th International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018

Y2 - 27 October 2018 through 28 October 2018

ER -

Achsan HTY, Wibowo WC, Putri WTH, Achsan MMB, Barcah QKD. Harvesting bibliography multi-thread, safe and ethical web crawling. In 2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 355-360. 8618262. (2018 International Conference on Advanced Computer Science and Information Systems, ICACSIS 2018). doi: 10.1109/ICACSIS.2018.8618262

Harvesting bibliography multi-thread, safe and ethical web crawling

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint

Cite this