Can ChatGPT Help Prospective Voters Get the Information They Need?

A November 2022 poll found that most voters still use search engines to find trusted election information. For most of the internet age, search engines sorted through an index of websites that are relevant to a user’s search terms. Today, we’re witnessing a shift from that format as popular generative artificial intelligence (AI) tools become more mainstream. 

Researchers and everyday people are leveraging tools such as OpenAI’s ChatGPT, Google’s Bard, or Microsoft’s Bing chatbot as a potential alternative to established search engines, offering voters a promising source for election information. However, problems with incompleteness, timeliness, and inaccurate information in prompts limit its effectiveness. We found a number of instances of misleading or incomplete information about how to vote, which can cause confusion or hinder a voter’s ability to exercise the right to vote effectively.  

Until these problems are addressed, voters should continue using their preferred authoritative sources for election information, and developers should take meaningful steps to address concerns about the reliability of their tools. For this blog, we chose to focus on one tool i.e., ChatGPT. However, there are similar tools that we will focus on in future research. 

What is ChatGPT 

ChatGPT is trained to generate coherent and contextually relevant responses to various prompts. Since ChatGPT is trained on millions of human-written sources and continually learns from human feedback, it can generate human-like responses. It is pre-trained on a large corpus of text data from books, Wikipedia, research articles, websites, social media platforms, code, and datasets – like Webtext, which consists of over 8 million web pages – but the majority of this training data was published before September 2021.  

How effective is ChatGPT?

To gauge the effectiveness of ChatGPT in providing accurate election information, we used the state of Illinois as a use case and asked ChatGPT about:  

1. Voting eligibility criteria 
2. Voting locations 
3. Methods of voting 
4. Current congressional candidates 

Note: This research was completed before GPT-4 was released on March 14. OpenAI claims “GPT-4 scores 40% higher than GPT-3.5 on our internal adversarial factuality evaluations” meaning GPT-4 may produce more factually reliable information. 

Voting Eligibility Criteria

We started by asking a broad question about the eligibility criteria to vote in Illinois, and ChatGPT responded with the correct voting eligibility requirements, including citizenship requirements, voting age, and residential requirements. It’s response also included the correct deadline to register to vote, 28 days. Since the model learns from human feedback, we wanted to evaluate the consistency in its response. We asked the same question three times and got a few different responses. For example, in two of three answers, the bot included ‘Not be declared mentally incompetent by a court of law’ as an eligibility criterion and mentioned ‘Not claim the right to vote anywhere else,’ in one instance, but not the other two. It is important to note that inconsistent or incomplete answers may mislead users. If voters have questions or concerns related to voting in their jurisdiction, they should check with their state or local election official.  

Adding more specifics resulted in a more detailed response.

Voting Locations

Broad prompts about where to vote and voting location information without added context result in generic responses, some including information to contact local election officials, local government, or the country’s embassy or consulate. While some information may be useful, it is best to check with official sources to verify your voting location.

When asked about places to vote in Chicago, it provided a link to the Chicago Board of Election Commissioners website, outlining the steps to find a voter’s polling place according to their address, and the phone number for the Chicago Board of Election’s office. This demonstrates that more specific prompts may generate more valuable information.

Voting Methods

After understanding how ChatGPT answers questions around who can vote and where they can do so, we then asked it what voting options are available. To better test ChatGPT’s reasoning, we asked a question about curbside voting. For context, in June 17, 2021, Illinois passed legislation to allow curbside voting for voters with disabilities. When we asked “where can I vote in Chicago?” the chatbot did not include curbside voting in its answers, but when asked specifically about curbside voting it responded correctly.

We wanted to check if asking about curbside voting influenced its response about allowable voting modes in Illinois. It did not influence its response potentially because curbside voting may not be commonly associated with other voting methods in the training data.

Limitations of ChatGPT – Current congressional candidates

It is crucial for users to understand that the information ChatGPT is trained on is not the most current, so candidates, campaign, and voting information may have changed. We assessed this by asking pointed questions about congressional candidates in Illinois.

Since the model we researched has not been trained effectively on data post-September 2021, ChatGPT could not give the latest data on the 2022 congressional candidates in Illinois. When asked about the current congressional representatives from Illinois, it provided a list of representatives who were in office in 2021 along with a disclaimer that this information may have changed with the next elections, although it is able to reason that congressional elections are scheduled for 2022.

Conclusion

Significant advancements in generative AI may supplement several aspects of our lives, such as writing and research or certain occupational tasks, and could seemingly substitute as a reliable resource for information, but it is important to exercise discretion regarding its applications, especially when there are significant implications on our democracy. It is easy to mislead users when solely relying on unconfirmed sources, such as ChatGPT and other chatbots, for answers. The bot has limitations of prompt length and information training, and often does not answer with complete or consistent information. We would caution users to check ChatGPT answers with reliable resources such as government websites or their local election boards.

The capabilities for fetching accurate election information are promising, but there is much room to grow and refine for ChatGPT to be useful to voters. The Bipartisan Policy Center’s Digital Democracy and Technology teams plan to subsequently investigate further generative AI and its implications on democracy and society, so please stay tuned.