User Perception Analysis of Online Learning Platform “Zenius” During the Coronavirus Pandemic Using Text Mining Techniques

Access to online learning platforms is expected to support online learning activities during the Coronavirus (COVID-19) pandemic. In their online learning activities, students experience several issues related to Internet access, learning content, and learning evaluation. Online learning platforms are indispensable to provide good learning resources that are easily accessible to students. This study aims to explore data from user reviews of the Zenius platform on Google Play Store to determine the priorities for service improvement by the provider. Topic modeling and sentiment analysis are applied to extract useful information based on user sentiment regarding the topics discussed in application reviews. The results of the topic analysis show that topic trends in user reviews focus on live classes, tryouts, the subject matter, user accounts, tutorial videos, and free learning access. Meanwhile, the Net Reputation Score (NRS) is an assessment method to evaluate a product or service based on user opinions published on social media. The results of a NRS reveal that the service aspects that should be prioritized to improve the online learning platform are related to tryouts and user accounts. Conversely, access to free learning received the highest score. Users benefit from free access to all learning content during their learning from home activities.


Introduction
The spread of the coronavirus disease (COVID-19) has impacted social and economic activities in Indonesia, including the education sector. In 115 days from the beginning of March 2020 to the end of June 2020, 50 thousand cases were recorded, and the next 50 thousand new cases were registered in just 32 days (Annur 2020). COVID-19 is an infectious disease caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus, which can cause disorders of the respiratory system, acute pneumonia, and even death (WHO 2020). The Minister of Education and Culture responded immediately to the situation by issuing Surat Edaran Nomor 4 Tahun 2020. For the sake of continuing Nevertheless, unusual changes in the teaching and learning process have caused a culture shock in a society that is accustomed to face-to-face learning (Azzahra 2020). Students prefer face-to-face learning, especially for subjects that require hands-on learning (Mawarni et al. 2020). Other issues experienced by students are unstable Internet access, limited Internet quotas, and difficulties in understanding the subject matter (Mawarni et al. 2020). Student awareness of the possibility to evaluate learning strategies and learning outcomes is still low during online learning activities (Sa'dyah and Basuki 2020). These issues are also experienced by students in three Sub-Saharan African countries (Ghana, Kenya, and South Africa), where there is a very low level of satisfaction with Internet connection, the cost of Internet packages, and the reliability of online education (Pete and Soko 2020). Indian students' perception of conventional learning and online learning shows that the lack of Internet connectivity and limited access to online learning platforms hinder the transition of students to online learning environments in developing countries (Kamble et al. 2021). Besides, the use of technology has been shown to help in the design and delivery of the subject matter during the COVID-19 pandemic in Palestine, whose experience can serve as a roadmap for online learning to support post-pandemic education in developing countries (Shraim and Crompton 2020). Online learning strategies making use of technology can help reduce the transactional distance between teachers and students and offer opportunities to increase cognitive presence, social presence, and teaching presence (Holbeck and Hartman 2018). Online learning platforms are indispensable to provide good learning resources that are easily accessible to students. Therefore, online learning platform providers need to constantly innovate and optimize their services. The effectiveness of online learning platforms in providing alternative learning resources during this pandemic should be assessed by measuring user perceptions as an indicator of success and feedback for providers.
As the use of the Internet increases, user perceptions can be conveyed through various online media, and content published by users can become a source of information for fellow users (Hu and Chen 2016). Collecting user data is a way to obtain more information about different changes or behaviors among users. Moreover, data can be used to give insight into problems in current implementations. The insight gained by interpreting user data can help providers determine platform service improvements necessary to address the users' specific concerns. Application reviews on Google Play Store can be a source of textual data to see how users of an online learning platform respond to the services that have been provided. Review data contain implicit information about the opinions and experiences of users (Masrury et al. 2019). However, there are no rules for writing reviews on Google Play Store; the text can be written freely. This results in a variety of writing formats. Large amounts of review data and unstructured data require the use of a technique to be efficiently processed into useful information. The information contained in review data can be extracted using the text mining technique (Feldman and Sanger 2006). Topic modeling is one of the text mining approaches that can find groups of words that represent a topic in a given document (Blei et al. 2003). Topic modeling can be applied to cluster user reviews into several topics, revealing hidden themes related to the platform. Additionally, we conducted sentiment analysis to determine user satisfaction with a few platform themes based on the result of the topic modeling. Sentiment analysis quantifies the perception of users by classifying the sentiments expressed in reviews. It can identify emotions expressed textually in user reviews as positive or negative (Liu 2010). Results from both methods provide insights into user perceptions of the quality of the platform's services for the provision of learning resources.
Previous work on topic modeling and sentiment analysis has been carried out by Aziz et al. (2018) to reveal information regarding the level of user satisfaction with Surabaya government services based on tweet data. Other research extracted useful information from a collection of tweet data to examine public opinion on Uber . However, previous works conducted topic modeling only to identify topics of conversation in each label or category. Topic modeling can be further applied to cluster textual data into more dynamic labels or categories. Moreover, previous studies have not used a product or service assessment method based on user sentiment on social media. Thus, this study aims to analyze useful extracted information based on user perceptions of online platform services for learning from home activities. This information is extracted using topic modeling and sentiment analysis through user reviews. The results of the study are expected to be useful for online learning platform providers, helping them determine the priorities for service improvement.

Text Mining
Text mining is the process of exploring the existing patterns in textual data sources to draw information and knowledge from large amounts of data (Feldman and Sanger 2006). The purpose of text mining is to assist in analyzing the connectedness of documents based on words that represent their contents to obtain useful information (Berry and Kogan 2010). The data that is used as input in the text mining process is unstructured, whereas data mining uses structured data or databases as the input (Han et al. 2012). Nonetheless, text mining applies data mining concepts and techniques because it requires giving the text a more structured form before carrying out further analysis.

Topic Modeling
Topic modeling is a form of unsupervised learning that allows for the efficient processing of a large amount of textual data. It aims to find out the structure of hidden topics in a collection of documents. In each document implied the topics, distribution of topics, and determination of topics per word (Blei et al. 2003). Latent Dirichlet allocation (LDA) is one of the most popular methods of topic modeling. The LDA model generates several keywords that are weighted on each topic. Thus, the LDA model can determine the dominant topic in each document by calculating the proportional value of the keywords contained in the text. Data can be clustered based on the similarity of the dominant topic.

Sentiment Analysis
Sentiment analysis consists in the computational study of opinions, appraisals, and emotions through entities, events, and attributes (Liu 2010). Sentiment analysis is carried out to classify the emotional tendency of an opinion into a positive or negative category. This study uses a lexicon-based approach with the SentiStrengthID algorithm. The algorithm generates positive and negative values to detect the strength of sentiments in an opinion (Wahid and Azhari 2016). If positive and negative values are equal, the sentiment is considered neutral. The table layout of a confusion matrix is used for the evaluation process. Confusion matrixes can help to measure the performance of the SentiStrengthID algorithm. F1-score is a performance measure that can be computed from the confusion matrix. It uses both a precision score and a recall score to understand the performance of the classifier from imbalanced data based on the formula shown in Equation 1 (Kulkarni et al. 2020):

Net Reputation Score
User sentiment in textual data can be explored further to find information on user assessment. The Net Reputation Score (NRS) is an assessment method to evaluate a product or service based on user opinions on social media. This method generates a score for a product or service based on the percentage of positive and negative comments posted by users, according to the Equation 2 (Meteor 2017): (2) The scores obtained range from 100% to -100%. This score can be used to determine the quality of a product or service and infer priorities for improvement.

Methodology
Prior research has discussed the use of topic modeling and sentiment analysis to extract useful information from textual data sources. However, these studies were conducted using different workflows. Aziz et al. (2018) conducted sentiment analysis and topic modeling on textual data obtained from Twitter to identify satisfaction with government service. Data was processed into information by grouping tweets into positive and negative classes. An LDA approach was used to identify topics that tend to have a strong influence on negative or positive opinions. Alamsyah et al. (2018), for their part, extracted useful information from tweets about a ride-hailing service provider. They mapped public opinions by analyzing text sentiments in a time-series format and developing a topic model to understand topics related to each sentiment and identify the topics being discussed. Alamsyah et al. (2019) also conducted content analysis to understand public opinion about an onlinebased sharing economy business model that disrupted innovation in the tourism industry. Content analysis was performed using tweet data. The results provide insights by identifying the words most frequently used in social conversations. Topic modeling with the LDA method is able to present the most popular topics to give a comprehensive understanding of how people react to the phenomenon. Additionally, a trained model was built by classifying reviews based on several aspects through manual annotations. Hariyani et al. (2019) conducted a lexicon-based sentiment analysis to find out students' assessments of the quality of educational services. They explained that the SentiStrengthID algorithm can be used to identify sentiment based on the polarity of a text. Text mining in this study is carried out using the model shown in Figure 1.
User review data were used as input in this text mining process. Text mining requires giving the text a more structured form before carrying out content analysis. We adopted LDA as a method for topic modeling. In addition, lexicon-based sentiment analysis was performed using the SentiStrengthID algorithm of Wahid and Azhari (2016), and the data was analyzed thanks to Jupyter Notebook. The final result is expected to define satisfaction scores for service aspects of the online learning platform and shed light on the users' perceptions. This study consists of three stages: data collection, data preprocessing, and content analysis.

Data Collection
Data was collected using a crawling technique on Google Play Store. The data consist of reviews by users of the Zenius application, one of the most popular online learning platforms in Indonesia according to the DSResearch report (DailySocial.id 2020). All of the platform's service lines have also increased significantly since students have had to carry out learning from home activities (Burhan 2020). The services of this platform are intended for grade 1 to grade 12 (K-12) students, who have become the primary users. Review data are in Bahasa because the service provider operates only in Indonesia. This study focuses on reviews of the services by users, including relevant opinions and experiences. Review data were filtered to remove duplicate and irrelevant data.

Data Preprocessing
Some processes were applied to structure the still-unstructured review data. In this study, data preprocessing was carried out, which included case folding, lexical normalization, tokenization, stopword removal, and stemming.
1. Case folding is a process of converting all letters in the review text to lowercase. Nonalphabetic characters (not expressing "a-z"), such as periods (.) and commas (,), were omitted because they are delimiters. We did not use any external libraries but worked with the Regular Expression and String modules available in Python. The output at this stage is more consistent review texts. 2. Lexical normalization aims to convert slang words into canonical words. We identified and normalized nonstandard ("Bahasa Alay") word variants by utilizing a word dictionary from Salsabila et al. (2018). Thus, this stage reduced the number of words that should have one meaning but are considered different. 3. Tokenization consists in splitting a sentence into word pieces called tokens. The sentences in the review text are separated into words using the "word tokenize" function of the NLTK library. 4. Stop-word removal aims to remove words that provide low information in a review text.
Negation words in the review text affect the results of the sentiment analysis and are therefore excluded from the stop-word removal process for tokens used in sentiment analysis. The Indonesian stop-list for stop-word removal in this study was sourced from Tala (2003). 5. Stemming is performed to remove inflections and reduce words to their root form. This stage is used to limit the number of occurrences of different words that share the same root meaning. The stemming process in Bahasa uses the Sastrawi library. Inflected words in a sentence affect the results of the sentiment analysis. Therefore, the change of words into their root forms was not carried out for the features used in the sentiment analysis process.

Content Analysis
Content analysis was conducted to analyze user perceptions of online learning platform services. It can be used as an alternative to survey research to understand the context of the data, gain knowledge, obtain new insights, and represent data to guide action. Two text mining techniques were used at this stage, namely topic modeling and sentiment analysis.
1. Topic modeling was carried out to identify the dominant topic in each review text. Thus, review data can be clustered based on the similarity of their dominant topic. A specific number of topics was determined before the topic modeling process. We built some LDA models with 2 to 18 topics, using the Mallet version of the Gensim library. Some topic models are built to determine the best model. This study used topics generated by the LDA model as service aspects of the online learning platform.
2. The level of user satisfaction with these service aspects was determined by conducting sentiment analysis. The SentiStrengthID algorithm detects user sentiment in each review text via a polarity value. Based on the results of the sentiment analysis, user satisfaction was defined in three levels (positive, negative, and neutral). The level of user satisfaction with the service aspects of the platform was visualized in a histogram.
The final result was then obtained by calculating user satisfaction scores using the NRS method to determine priorities for service improvement. The insights extracted from user perceptions are very useful for the provider to develop effective platform services to increase user satisfaction.

Topic Trends in User Reviews
This section analyzes topic trends in the review data. Topic trends explain user interest in a discussion about online learning platform services. The optimal number of topics is six and is determined by summarizing the visualization of the LDA model. This number was chosen because the bubbles representing topics in the visualization of the LDA model are large enough, do not overlap, and are scattered across all quadrants as shown in Figure 2.

Figure 2. LDA Model Visualization
The number of topics specified should not be too large because this would impact the appearance of keywords that are repeated across several topics, making it difficult to conclude the topic. We used keywords to understand the essence of a topic. In addition, we read some of the most representative reviews to help identify the topic. Representative reviews have text that mentions the most specific keywords. The conclusions regarding the topics revealed by the LDA model are described in Table 2.
All topics discussed in user reviews were reflected in the keywords in Table 2. The first topic contains keywords such as ajar (teach), bagus (good), download (download), kelas (class), banget (really), keren (cool), vidio (video), tutor (tutor), senang (happy), and saran (advice). Adjectives identify abstract levels of customer satisfaction, bagus (good); keren (cool); and senang (happy), concerning possible subjects such as kelas (class), vidio (video), and tutor (tutor). The second topic concerns the users' opinion on the tryout service. Several user perceptions of the tryout service are conveyed abstractly, as represented by baik (good), sesuai (appropriate), and salah (incorrect). The third topic focuses on subject matters on the platform. User perceptions of the learning materials and concepts seem to be positive, which is reflected by mudah (convenient), jelas (clear), and lengkap (comprehensive). The fourth topic deals with user accounts. Some users have difficulty accessing the online learning platform, both when logging in and registering for a new account. The fifth topic touches on the level of user satisfaction with tutorial videos on the platform. Users are concerned about Internet connectivity and Internet quotas for accessing videos. The sixth topic contains user opinions about free access to learning content. The access provided helps students continue their education during the pandemic.

Translation:
After updating, the platform looks really smooth. Then, Zenius is the only tutoring platform that makes me a big fan of the tutors! It's really fun, their way of teaching isn't "generalized," so each tutor is unique. There is Bang Sabda, who is the best tutor, Bang Hilman who makes me love a history lesson, Bang Wisnu, and many more! In addition, the way they teach always emphasizes "concepts" that have been compiled using their own terms, so it feels like I was taught by peer tutors. Unlike other platforms where the learning method is very standard.

Translation:
Honestly, the platform is good, but I have signed up and can't log in. I have changed my accounts 3 times so I can still use the platform, but the phone number can't be registered twice. I have used my phone number, my mother's phone number, and also my father's phone number to re-register. I want to use it but I can't register anymore.

Translation:
The tutorial video is easy to understand. The explanation is also concise and clear. For the developer, please add the download feature because the network is slow in my current location. Thus, sometimes I can't watch videos on the Zenius platform. The Internet is good in the middle of the night. However, the Internet is slow in the morning until the night. Please add a download feature so I can still watch the video when I'm offline.

6
Free Learning Access 0.184*"ajar" + 0.122*"gratis" + 0.037*"anak" + 0.028*"guru" + 0.027*"online" + 0.025*"sukses" + 0.021*"sekolah" + 0.019*"manfaat" + 0.017*"rumah" + 0.017*"pandemi" The weight of each keyword reflects the importance of the keyword for the topic. We used these weights to calculate the proportion value of the keywords that contribute to a review text. Review texts carry a proportion value for each topic, and the highest proportion value becomes a reference to find the dominant topic in a review text. Then, user reviews were clustered based on the similarity of their dominant topics. These topics were used as service aspects of the online learning platform. The distribution of user reviews on each topic is shown in Figure 3. Figure 3 shows that each topic is featured in a large number of reviews, which means that they are interesting for users to discuss in-app reviews. The output of this section is used to determine the level of user satisfaction with the platform's services by conducting sentiment analysis.

User Sentiments Regarding Service Aspects
The frequency distribution was arranged in the form of a histogram to gain insight into user sentiments about each service aspect. User sentiments were identified using a lexicon-based algorithm, SentiStrengthID. An evaluation was carried out using a confusion matrix to assess the performance of the SentiStrengthID algorithm.

Figure 4. SentiStrengthID Evaluation Matrix
The rating value was used as a truth set, where a value of 1 indicates a negative review, and a value of 5 is interpreted as a positive review. There were 586 positive reviews and 96 negative reviews. An F1score assessment served as a reference because the number of positive and negative reviews was nonsymmetric. Based on Figure 4, the SentiStrengthID algorithm obtained an F1-Score of 92.75%. We conclude that the SentiStrengthID algorithm is reliable to classify the sentiment of user reviews.
A histogram was created to determine the patterns of user sentiments regarding each service aspect based on review data. Histograms can show user perceptions conveying positive, neutral, or negative opinions. This can give a broad overview of the level of user satisfaction with service aspects, described by the number of reviews expressing negative, neutral, and positive sentiments, as shown in Figure 5.

Figure 5. Histogram of User Sentiments
Figure 5 reveals that users were most satisfied with the subject matters and free learning access provided by the online learning platform, indicating that the service provider can meet its users' needs. Live class and tryout services also obtained a high number of positive reviews. However, the tryout service received negative user perceptions as indicated by the highest number of negative reviews. A high number of positive reviews does not necessarily indicate that the service aspect is perceived positively by users; a method is needed to calculate the user satisfaction score with certainty by using the number of positive reviews and the number of negative reviews. The output of this section was used to calculate the NRS in order to determine the user satisfaction score for each service aspect.

Assessment of Platform Services
A report on user perceptions of service quality on the online learning platform was generated. An assessment of online learning platform services was carried out using the NRS method, which calculates the difference in the percentage of reviews expressing positive and negative sentiments. NRS values are used to measure the user satisfaction score with the services provided by the online learning platform. The results of the NRS assessment are shown in Table 3. The user satisfaction scores for each service aspect are shown in Table 3. Service improvement can then be prioritized based on the lowest NRS scores. Two service aspects received an NRS below 50%, namely tryout and user account. Both service aspects are thus in critical need of immediate improvement by the service provider. Meanwhile, the highest NRS concerned free learning access (93.75%) and subject matter (92.04%), confirming that these aspects are satisfactory and relatively mature and showing that the focus of service improvement can be directed towards other aspects.

Discussion and Implications
This study aimed to explore review data on Google Play Store to extract information about user perceptions of online learning platform services. The satisfaction scores for each service aspect show that users are most satisfied with the free learning service provided by the platform. The user satisfaction score for this service is 93.75%. The reviews reveal that users are helped by free access to all learning content during learning from home activities. However, there are not many majors available for vocational high schools, and the existing subjects in Madrasah Aliyah are incomplete, preventing some users from using the platform. The subject matters on this platform are incomplete and some explanations are different from those taught by teachers in schools. Additionally, the arrangement of learning content is not sequential; as a result, users encounter difficulties when looking for subject matters. Nonetheless, subject matter as an aspect received a high user satisfaction score of 92.04%.
Meanwhile, the live class service allows the tutors to answer user questions in real-time. Users are satisfied with this service, as indicated by a user satisfaction score of 82.43%. Users complain that the explanations of the tutor are not to the point, that the tutor's language of interaction is too casual, that the tutor teaches too fast, and that the handwriting of the tutor on the blackboard is not visible on smartphones with small screens. Tutorial videos obtained a low satisfaction score of 61.98%. Many users complain that the video resolution is too high and recommend that the provider make learning content that can be accessed with minimum bandwidth and provide an app feature to adjust the video resolution.
Concerning the tryout service, users noted problems with the tryout history, which disappears after updating the application, as well as answer choices that are missing or do not follow the tryout questions, and answers to tryout questions that are considered empty even though they have been filled. This aspect obtained the lowest satisfaction score (34.91%), followed by user accounts (44.14%). Users have difficulty logging on to the platform even when using Facebook, Google, and Twitter accounts. Meanwhile, the registration process was reported to fail often despite multiple attempts. During the registration process, users received a message saying that their Internet connection was lost when the signal and Internet quotas were actually in good condition. This study provides suggestions for improving platform performance that focus on the service quality of tryouts and user accounts.
Concerning practical implications, providers have a good chance to provide great online learning platform services by extracting useful information from large amounts of review data to be used as insights. Providers can periodically evaluate their online learning platform by exploring review data using text mining techniques. They can learn about user perceptions and specific issues by conducting content analysis based on the results of topic modeling and sentiment analysis and thus easily prioritize services that need to be improved. In particular, Zenius can optimize the service quality of tryouts by fixing technical issues that may affect the test scores obtained by the user and make the learning evaluation less precise. In addition, the platform provider can allow users to log in and register for a new account using a one-time password (OTP) sent to a phone number. It can also develop virtual assistance to facilitate access to its services for users.
This study also suggests that users have concerns about Internet connection and Internet quota availability when accessing tutorial videos. Excessively high and non-adjustable video quality hinders students in their online learning activities. This is similar to the results of a previous study, which concluded that the lack of Internet connectivity and access to online learning platforms hampers the transition of students to an online learning environment (Kamble et al. 2021). The conclusion regarding users' perceptions of borderless interactions with the tutors during live classes is in keeping with a previous study showing that online learning using technology can help reduce transaction distances between teachers and students (Holbeck and Hartman 2018). The user satisfaction score for the subject matter can indicate that online learning has a high potential to support post-pandemic education in developing countries, as stated by Shraim and Crompton (2020).

Conclusion
This study analyzed user perceptions of online learning platform services as a learning resource amid the COVID-19 pandemic, using text mining techniques. The dataset was gathered from Google Play Store reviews of one of the most popular online learning platforms in Indonesia. Review data reflect user perceptions that are conveyed objectively and voluntarily. This study conducted topic modeling and sentiment analysis to explore the dataset. Topic trends in user reviews concerned live classes, tryouts, subject matters, user accounts, tutorial videos, and free learning access. The service aspect that received the highest NRS is free learning access, with a user satisfaction score of 93.75%. After exploring user review data, it appears that users are helped by having free access to all learning content during learning from home activities. Conversely, the two service aspects that obtained the lowest NRS are tryouts and user accounts, with 34.91% and 44.14% user satisfaction, respectively. Users complained about missing tryout history preventing them from seeing discussions and learning from their mistakes when answering the questions. Additionally, users noted missing or inconsistent answer choices for the tryout questions and also reported that some answers to tryout questions are considered empty even though they have been filled. Meanwhile, complaints also concerned the difficult process of logging in and registering for accounts, which prevented users from accessing online learning platform services.
The results of the NRS assessment highlight priority aspects for improving platform services, namely tryouts and user accounts. The issues experienced by users are mostly technical in nature. Nevertheless, these issues may have an impact on the value of the lessons learned by the user and make the results of the evaluation of learning development less precise. Thus, this study brings useful insights to the provider based on feedback from the users. Suggestions for future study concern 1) the design and development of a system that can process review data periodically because data may change over time and 2) the validation of relevant user reviews thanks to a machine-learning model that may decrease data reduction in the filtering process.