Building Indonesian local language detection tools using Wikipedia data

Puji Martadinata; Bayu Distiawan Trisedya; Hisar Maruli Manurung; Mirna Adriani

doi:10.1007/978-3-319-31468-6_8

Building Indonesian local language detection tools using Wikipedia data

Puji Martadinata, Bayu Distiawan Trisedya, Hisar Maruli Manurung, Mirna Adriani

Department of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Citations (Scopus)

Abstract

The widespread use of social media today has generated lots of research interest towards information retrieval, natural language processing, and also machine learning. The vast diversity of languages used on social media creates the need for accurate automated language identification tools. In this research, we develop a language identification tool that can help automatically identify social media posts in Indonesian, Javanese, Sundanese, and Minangkabau. The latter three are some of the most widely spoken regional languages in Indonesia. We conducted experiments to compare three popular methods used to develop language identification tools, namely N-grams, statistical models, and the Small Words technique. Our experiments conducted using articles on internet for training and tested using social media data that we constructed, show that the statistical method obtains the best result among all the methods used.

Original language	English
Title of host publication	Worldwide Language Service Infrastructure - 2nd International Workshop, WLSI 2015, Revised Selected Papers
Editors	Donghui Lin, Yohei Murakami
Publisher	Springer Verlag
Pages	113-123
Number of pages	11
ISBN (Print)	9783319314679
DOIs	https://doi.org/10.1007/978-3-319-31468-6_8
Publication status	Published - 2016
Event	2nd International Workshop on Worldwide Language Service Infrastructure, WLSI 2015 - Kyoto, Japan Duration: 22 Jan 2015 → 23 Jan 2015

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	9442
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	2nd International Workshop on Worldwide Language Service Infrastructure, WLSI 2015
Country/Territory	Japan
City	Kyoto
Period	22/01/15 → 23/01/15

Keywords

Language identification
Language model
N-gram
Statistical method
Twitter
Wikipedia

Access to Document

10.1007/978-3-319-31468-6_8

Cite this

Martadinata, P., Trisedya, B. D., Manurung, H. M., & Adriani, M. (2016). Building Indonesian local language detection tools using Wikipedia data. In D. Lin, & Y. Murakami (Eds.), Worldwide Language Service Infrastructure - 2nd International Workshop, WLSI 2015, Revised Selected Papers (pp. 113-123). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9442). Springer Verlag. https://doi.org/10.1007/978-3-319-31468-6_8

Martadinata, Puji ; Trisedya, Bayu Distiawan ; Manurung, Hisar Maruli et al. / Building Indonesian local language detection tools using Wikipedia data. Worldwide Language Service Infrastructure - 2nd International Workshop, WLSI 2015, Revised Selected Papers. editor / Donghui Lin ; Yohei Murakami. Springer Verlag, 2016. pp. 113-123 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{95aad414093140e0885231c664613316,

title = "Building Indonesian local language detection tools using Wikipedia data",

abstract = "The widespread use of social media today has generated lots of research interest towards information retrieval, natural language processing, and also machine learning. The vast diversity of languages used on social media creates the need for accurate automated language identification tools. In this research, we develop a language identification tool that can help automatically identify social media posts in Indonesian, Javanese, Sundanese, and Minangkabau. The latter three are some of the most widely spoken regional languages in Indonesia. We conducted experiments to compare three popular methods used to develop language identification tools, namely N-grams, statistical models, and the Small Words technique. Our experiments conducted using articles on internet for training and tested using social media data that we constructed, show that the statistical method obtains the best result among all the methods used.",

keywords = "Language identification, Language model, N-gram, Statistical method, Twitter, Wikipedia",

author = "Puji Martadinata and Trisedya, {Bayu Distiawan} and Manurung, {Hisar Maruli} and Mirna Adriani",

note = "Publisher Copyright: {\textcopyright} Springer International Publishing Switzerland 2016.; 2nd International Workshop on Worldwide Language Service Infrastructure, WLSI 2015 ; Conference date: 22-01-2015 Through 23-01-2015",

year = "2016",

doi = "10.1007/978-3-319-31468-6_8",

language = "English",

isbn = "9783319314679",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "113--123",

editor = "Donghui Lin and Yohei Murakami",

booktitle = "Worldwide Language Service Infrastructure - 2nd International Workshop, WLSI 2015, Revised Selected Papers",

address = "Germany",

}

Martadinata, P, Trisedya, BD, Manurung, HM & Adriani, M 2016, Building Indonesian local language detection tools using Wikipedia data. in D Lin & Y Murakami (eds), Worldwide Language Service Infrastructure - 2nd International Workshop, WLSI 2015, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9442, Springer Verlag, pp. 113-123, 2nd International Workshop on Worldwide Language Service Infrastructure, WLSI 2015, Kyoto, Japan, 22/01/15. https://doi.org/10.1007/978-3-319-31468-6_8

Building Indonesian local language detection tools using Wikipedia data. / Martadinata, Puji; Trisedya, Bayu Distiawan; Manurung, Hisar Maruli et al.
Worldwide Language Service Infrastructure - 2nd International Workshop, WLSI 2015, Revised Selected Papers. ed. / Donghui Lin; Yohei Murakami. Springer Verlag, 2016. p. 113-123 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9442).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Building Indonesian local language detection tools using Wikipedia data

AU - Martadinata, Puji

AU - Trisedya, Bayu Distiawan

AU - Manurung, Hisar Maruli

AU - Adriani, Mirna

N1 - Publisher Copyright: © Springer International Publishing Switzerland 2016.

PY - 2016

Y1 - 2016

N2 - The widespread use of social media today has generated lots of research interest towards information retrieval, natural language processing, and also machine learning. The vast diversity of languages used on social media creates the need for accurate automated language identification tools. In this research, we develop a language identification tool that can help automatically identify social media posts in Indonesian, Javanese, Sundanese, and Minangkabau. The latter three are some of the most widely spoken regional languages in Indonesia. We conducted experiments to compare three popular methods used to develop language identification tools, namely N-grams, statistical models, and the Small Words technique. Our experiments conducted using articles on internet for training and tested using social media data that we constructed, show that the statistical method obtains the best result among all the methods used.

AB - The widespread use of social media today has generated lots of research interest towards information retrieval, natural language processing, and also machine learning. The vast diversity of languages used on social media creates the need for accurate automated language identification tools. In this research, we develop a language identification tool that can help automatically identify social media posts in Indonesian, Javanese, Sundanese, and Minangkabau. The latter three are some of the most widely spoken regional languages in Indonesia. We conducted experiments to compare three popular methods used to develop language identification tools, namely N-grams, statistical models, and the Small Words technique. Our experiments conducted using articles on internet for training and tested using social media data that we constructed, show that the statistical method obtains the best result among all the methods used.

KW - Language identification

KW - Language model

KW - N-gram

KW - Statistical method

KW - Twitter

KW - Wikipedia

UR - http://www.scopus.com/inward/record.url?scp=84961712303&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-31468-6_8

DO - 10.1007/978-3-319-31468-6_8

M3 - Conference contribution

AN - SCOPUS:84961712303

SN - 9783319314679

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 113

EP - 123

BT - Worldwide Language Service Infrastructure - 2nd International Workshop, WLSI 2015, Revised Selected Papers

A2 - Lin, Donghui

A2 - Murakami, Yohei

PB - Springer Verlag

T2 - 2nd International Workshop on Worldwide Language Service Infrastructure, WLSI 2015

Y2 - 22 January 2015 through 23 January 2015

ER -

Martadinata P, Trisedya BD, Manurung HM, Adriani M. Building Indonesian local language detection tools using Wikipedia data. In Lin D, Murakami Y, editors, Worldwide Language Service Infrastructure - 2nd International Workshop, WLSI 2015, Revised Selected Papers. Springer Verlag. 2016. p. 113-123. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-31468-6_8

Building Indonesian local language detection tools using Wikipedia data

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this