Latest Projects
Gov-RAG: A Retrieval-Augmented Generation Framework for Enhancing E-Government Services
Authors: Miao Yu and Hailiang Chen
Abstract:
The rapid evolution of Large Language Models (LLMs) has opened new opportunities for applying AI in high-stakes domains such as government and legal services. However, deploying LLMs involves challenges like outdated training data, limited explainability, and reliability issues due to hallucinations. This study introduces Gov-RAG, a novel Retrieval-Augmented Generation (RAG) framework, specifically tailored to address the unique demands of e-government applications. Grounded in trust in automation theory and domain-specific practical adaptation needs, the Gov-RAG framework achieves five key goals: (1) In-depth domain knowledge, (2) Factual accuracy in sensitive domains, (3) Human-centered explainability, (4) Low-cost real-time updates, and (5) Hallucination reduction. The framework incorporates a real-time updated comprehensive dataset, aggregating national laws, regulations, and government policies. To enhance retrieval precision, Gov-RAG combines dense and sparse vector retrieval with inverted index full-text retrieval and applies domain-specific hierarchical slicing. We also develop a fact-based evaluation framework to measure hallucinations across different LLM architectures, an aspect that was not properly measured in prior work. Rigorous computational experiments show that while fine-tuning improves stylistic alignment with standard answers, it also increases hallucination. Gov-RAG can improve factual consistency and reduce contradictions, consistently outperforming base LLMs and alternative frameworks in governmental applications. Furthermore, RAG with fine-tuned LLMs exhibits degraded performance in answering questions that require historical knowledge. Gov-RAG exhibits balanced capabilities across different test datasets. Online experiments with real citizens further validate the feasibility and impact of Gov-RAG on user performance, user perceptions, and user behaviors. Results indicate significant improvements in task performance, user-perceived explainability, efficiency, and user sentiment during engagement compared to base LLMs. Additionally, we delve into human-centered perceptions and adopt explainable machine learning approaches to identify what drives user performance improvement.
Download this paper at SSRN
ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience
Authors: Ruiyun Rayna Xu, Yue (Katherine) Feng, and Hailiang Chen
Abstract:
The advent of ChatGPT, a large language model-powered chatbot, has prompted questions about its potential implications for traditional search engines. In this study, we investigate the differences in user behavior when employing search engines and chatbot tools for information-seeking tasks. We carry out a randomized online experiment, dividing participants into two groups: one using a ChatGPT-like tool and the other using a Google Search-like tool. Our findings reveal that the ChatGPT group consistently spends less time on all tasks, with no significant difference in overall task performance between the groups. Notably, ChatGPT levels user search performance across different education levels and excels in answering straightforward questions and providing general solutions but falls short in fact-checking tasks. Users perceive ChatGPT’s responses as having higher information quality compared to Google Search, despite displaying a similar level of trust in both tools. Furthermore, participants using ChatGPT report significantly better user experiences in terms of usefulness, enjoyment, and satisfaction, while perceived ease of use remains comparable between the two tools. However, ChatGPT may also lead to overreliance and generate or replicate misinformation, yielding inconsistent results. Our study offers valuable insights for search engine management and highlights opportunities for integrating chatbot technologies into search engine designs.
Download this paper at SSRN
The Impact of ChatGPT Launch on the Demand for Content Generation Services in the Freelancing Market
Authors: Ziqing Kim Yuan and Hailiang Chen
Abstract:
The ongoing debate surrounding AI’s impact on the labor market has been fueled by the rise of generative AI. While some argue that it may displace jobs, others suggest it could create new opportunities and improve productivity. This study investigates the impact of the ChatGPT launch on the demand for human content generation services, using a difference-in-differences approach and data from the online labor marketplace Fiverr. Our investigation reveals a significant decrease in demand for human content generation services following the ChatGPT launch. Moreover, we explore the heterogeneity of the impact, identifying differential effects across service types, human writers’ education levels, and service pricing. Notably, we observe a significant increase in demand for idea planning services, while demand for text reviewing services and services provided by human writers with a PhD degree remains unaffected by the launch of ChatGPT. These results contribute to the ongoing debate on the impact of generative AI on the labor market and offer practical guidance for human writers navigating this evolving AI-driven landscape.
Download this paper at SSRN
SocioLink: Leveraging Relational Information in Knowledge Graphs for Startup Recommendations
Journal of Management Information Systems 40(2) 655-682. 2023.
Authors: Ruiyun Rayna Xu, Hailiang Chen, and J. Leon Zhao
Abstract:
While venture capital firms are increasingly relying on recommendation models in investment decisions, existing startup recommendation models fail to consider the uniqueness of venture capital context, including two-sided matching between investing and investee firms and a lack of information disclosure requirements on startups. Following the design science research paradigm and guided by the proximity principle from social psychology, we develop a novel framework called SocioLink by depicting and analyzing various relations in a knowledge graph via machine learning. Our experimental results show that SocioLink significantly outperforms state-of-the-art startup recommendation methods in both accuracy and quality. This improvement is driven by not only the inclusion of social relations but also the superiority of modelling relations via knowledge graph. We also develop a web-based prototype to demonstrate explainable artificial intelligence. This work contributes to the FinTech literature by adding an innovative design artifact—SocioLink—for decision support in the investment context.
Download this paper at SSRN
Listening in on investors' thoughts and conversations
Journal of Financial Economics 145(2) 426-444. 2022.
Authors: Hailiang Chen and Byoung-Hyoun Hwang
Abstract:
A large literature in neuroscience and social psychology shows that humans are wired to be meticulous about how they are perceived by others. We propose that impression-management considerations also end up guiding the content that investors transmit via word-of-mouth. We analyze server-log data from one of the biggest investment-related websites in the United States, as well as experimental data. Consistent with our proposition, we find that investors more frequently share articles that are more suitable for impression management, even when such articles less accurately predict returns. Additional analyses suggest that high levels of sharing can lead to overpricing.
Download this paper at SSRN
Fake News, Investor Attention, and Market Reaction
Information Systems Research 32(1) 35-52. 2021.
Authors: Jonathan Clarke, Hailiang Chen, Ding Du, and Yu Jeffrey Hu
Abstract:
Does fake news in financial markets attract more investor attention and have a significant impact on stock prices? We use the SEC crackdown of stock promotion schemes in April 2017 to examine investor attention and the stock price reaction to fake news articles. Using data from Seeking Alpha, we find that fake news stories generate significantly more attention than a control sample of legitimate articles. We find no evidence that article commenters can detect fake news. Seeking Alpha editors have only modest ability to detect fake news. The broader stock market appears to price fake news correctly. The stock price reaction to the release of fake news is not significantly different than a matched control sample over short and longer-term windows. We conclude by presenting a machine learning algorithm that is successful in identifying fake news articles.
Download this paper at SSRN
Signal or Noise in Social Media Discussions: The Role of Network Cohesion in Predicting the Bitcoin Market
Journal of Management Information Systems 37(4) 933-956. 2020.
Authors: Peng Xie, Hailiang Chen, and Yu Jeffrey Hu
Abstract:
Prior studies have shown that social media discussions can be helpful in predicting price movements in financial markets. With the increasingly large amount of social media data, how to effectively distinguish value-relevant information from noise remains an important question. We study this question by investigating the role of network cohesion in the relationship between social media sentiment and price changes in the Bitcoin market. As network cohesion is associated with information correlation within the discussion network, we hypothesize that less cohesive social media discussion networks are better at predicting the next-day returns than more cohesive networks. Both regression analyses and trading simulations based on data collected from Bitcointalk.org confirm our hypothesis. Our findings enrich the literature on the role of social media in financial markets and provide actionable insights for investors to trade based on social media signals.
Download this paper at SSRN