The Pitfalls Of ChatGPT; We Tested It In Montana

I make use of ChatGPT a lot to help give me inspiration for article titles.  Some days, there isn't as much in the creative section of my mind as I'd like, so I lean on it to get me thinking in the right direction.

Get our free mobile app

In using this service, I've tested the article titles given by ChatGPT for Search Engine Optimization, SEO, and the scores are generally very low.  However, it's not a bad place to start.

This got me wondering, just how reliable is the information given by ChatGPT.  You'd think that as it's AI, it has the whole of the web to work with, so it's information would be spot on.

Think again.

Putting ChatGPT Facts To The Test

I asked ChatGPT to find "10 cozy cafes and where they are in Central Montana".

This is what it generated:

1. Main Street Perk - Lewistown
2. The Badlander's Coffee House - Great Falls
3. Nourish Café - Helena
4. Wheatgrass Saloon - Livingston
5. Gourmet Cellar & Deli - Bozeman
6. The Daily Coffee Bar - Whitefish
7. Chalet Market Café - Belgrade
8. The Roost - Missoula
9. The Square Café - Dillon
10. The Coffee Factory - Butte

Let's put on the fact-check hat.  

1. Main Street Perk is in Laurel
2. From my search, there's no Badlander's Coffee House in Great Falls or anywhere
3. There's a Nosh Cafe in Helena, but no Nourish Cafe
4. It appears there IS or WAS a Wheatgrass Saloon in Livingston.  It's unclear if it's still open
5. Gourmet Cellar is in Livingston.  I couldn't find a Gourmet Cellar & Deli
6. I couldn't find a Daily Coffee Bar in the entire state of Montana
7. There is a Chalet Market and Deli, but it's in Bozeman.  I'll call a technical on this since Bozeman and Belgrade aren't that far apart.
8. There is a Rustler's Roost restaurant in Missoula.  The only thing that came up as The Roost is a rental in the Missoula area
9. There was no business in Dillon that's name even resembles The Square Cafe
10. I could find no Coffee Factory in Butte.

So, in this one, unscientific test, it appears that ChatGPT got 2 facts right out of 10, and technically, one wasn't completely right. 15ish percent accuracy?

Then I asked ChatGPT if it felt it's facts were accurate.  This is the response:

As an AI language model, I strive to provide accurate and reliable information based on the data I have been trained on. However, it's important to note that I do not have real-time access to current events or the ability to verify the accuracy of every statement. The responsibility lies with the users to evaluate the information and cross-check it with other sources if needed. It's always a good practice to consult multiple sources and exercise critical thinking when assessing the accuracy of any information, including the responses provided by AI models like ChatGPT.

Maybe the days where machines take over completely isn't quite as close as we think.

via GIPHY