|
Automated Question Answering for Thai Visa Services using GPT-4: System Architecture, Custom Dataset, and Performance Evaluation |
|---|---|
| รหัสดีโอไอ | |
| Creator | Aulia Akhrian Syahidi |
| Title | Automated Question Answering for Thai Visa Services using GPT-4: System Architecture, Custom Dataset, and Performance Evaluation |
| Publisher | Faculty of Informatics, Mahasarakham University |
| Publication Year | 2569 |
| Journal Title | Journal of Applied Informatics and Technology |
| Journal Vol. | 8 |
| Journal No. | 2 |
| Page no. | 262526 |
| Keyword | Automated Question Answering, GPT-4Large Language Models, Thai Immigration Services, Visa Information System |
| URL Website | https://ph01.tci-thaijo.org/index.php/jait |
| Website title | Journal of Applied Informatics and Technology |
| ISSN | 3088-1803 |
| Abstract | Thailand’s immigration system includes various visa types, such as Tourist(TR), Business (Non-Immigrant B), Education (Non-Immigrant ED), EDPlus, Media (Non-Immigrant M), Official (Non-Immigrant F), Depen-dent/Spouse (Non-Immigrant O), SMART, Transit, Long-Term Resident(LTR), Retirement (O-A), and Long-Stay Retirement (O-X), each gov-erned by distinct and frequently updated policies. This complexity causesconfusion and difficulties for foreign nationals in understanding requireddocuments, eligibility criteria, and policy changes, as shown by a surveyinvolving 75 respondents in Bangkok. Without scalable and accurate so-lutions, these challenges can reduce service efficiency and public trust. Toaddress this, the study employs GPT-4, a Large Language Model (LLM),optimized with prompt engineering and few-shot learning to automaticallyanswer visa-related questions with accurate, clear, and policy-compliantresponses. A simulated dataset of 1,600 question-answer pairs was createdfrom official Thai immigration policies. GPT-4’s performance was sys-tematically evaluated using automatic metrics (BLEU, ROUGE-L, ME-TEOR) and human assessments of accuracy, clarity, and user satisfaction,providing a robust methodological basis for evaluating its effectiveness inpublic service. Results show strong performance across most categories,with the highest BLEU scores observed in Tourist (0.82), Business (0.81),and Long-Term Resident (LTR) (0.78) visa-related queries. Correspond-ing peak ROUGE-L scores were 0.86 for Tourist, 0.84 for Business, and0.81 for Long-Term Resident (LTR) visas, while METEOR scores peakedat 0.79 (Tourist), 0.78 (Business), and 0.76 (Long-Term Resident (LTR)).The system achieved an overall BLEU score of 0.75, ROUGE-L of 0.79,and METEOR of 0.73. Human evaluators rated GPT-4’s responses withan average score of 4.5 for accuracy, 4.6 for language clarity, and 4.4 foruser satisfaction (on a 5-point scale). A Pearson correlation of 0.78 be-tween BLEU and human-rated accuracy indicates high alignment betweenautomated and human evaluation. These results highlight GPT-4’s po-tential in enhancing public-facing services such as immigration services byproviding accurate, clear, and policy-aligned responses. Future work willfocus on multilingual support, real-time policy updates, and deploymentin live service environments |
