ChatGPT: The Good, The Bad and The Hyperbole

By BC, Chief Technologist | Published: February 8, 2023 in Blog

ChatGPT was launched in November of 2022, and has quickly attracted considerable industry attention. In that short time, many folks have editorialized on what ChatGPT could mean for society, for the IT industry, for jobs, for college essays, and so forth. Much of that analysis is an exercise in futurism: the predictions are not necessarily wrong, but they also anticipate implications that have not yet materialized, even if those outcomes seem likely.

In 2022, tools like Midjourney and DALL-E raised alarm bells with professional artists and illustrators, concerned that an entire category of paid work was going to disappear and be taken over by machine-generated imagery (and to add further insult: much of that generation process was trained using illustrations obtained without license or consent of the creators). We are starting to see the same type of alarm raised by professional writers and content creators. Buzzfeed, for example, recently laid off 12% of its staff and announced its intention to supplement its writers with AI-generated content; the stock market rewarded this move with huge increases to share prices.

ChatGPT in IT organizations

Many claims about ChatGPT (and about ML solutions in general) will almost certainly become serious considerations within the next decade.

ChatGPT is not yet what people are predicting it’ll become. That’s not to say that we should be comfortable sticking to the status quo: the status is very clearly no longer quo. Even with ChatGPT’s current limitations there are non-trivial, pragmatic ways that it can change information technology right now.

Let’s talk about three areas:

1. ChatGPT as a search engine;
2. ChatGPT and written content; and
3. ChatGPT and coding activities.

In this blog, we’ll cover the search engine topic; later blogs will cover the subsequent sections.

ChatGPT as a Modern Search Engine 

Since the earliest days of the Internet, the search engine has been organized like a phone book: it provides the addresses (links) to help you find the information you’re looking for. You enter your search criteria, scan the list of returned results, looking for a promising option, and then click on the best candidates.  

There are downsides to this kind of approach: in a relatively benign example, SEO optimization is the practice of gaming the search algorithm to your organization’s benefit. Google has come under increasing criticism that their search capability is growing less and less useful as their search results are polluted with ads and fake content.  

ChatGPT, by contrast, doesn’t offer up links, but instead offers up information. It can correctly answer queries such as “What else has Bella Ramsey been in?” Or “What was the 70s detective show with the messy detective who says ‘Just one more question…’”.  

While that might not seem especially ground-breaking, it can lead to interesting changes of behaviour that have the opportunity to provide a real, viable competitor to Google (competing search engines—such as Duck Duck Go—exist, but most of those follow the same index model of Google). Google, itself, has been very conservative about changing the behaviour of search as search’s ad revenue is the largest income source for the company. 

ChatGPT isn’t a good source to find, say, plumbers in downtown Toronto, but it can provide answers to questions such as “What are the top 3 internationalization frameworks for React apps?” Or “When would you recommend the different AWS queuing services?” The benefit of those types of answers for researching technical solutions is fairly significant. 

There is a danger of ChatGPT becoming seen as a source of truth: the immediate concern, of course, is that it’s not 100% accurate, and that its training data is about a year and a half out of date. (Because of its tendency toward “confident, even if wrong” responses, someone recently referred to ChatGPT as “Mansplaining-as-a-service”, and that’s not inaccurate).  

In the longer term, it’s unclear how the answers will evolve over time: it’s one thing to acknowledge that a particular technology or framework is the new hotness and ask ChatGPT questions about that. But it’s quite another thing to recognize that a formerly top framework is no longer actively recommended by solution architects (for whatever reason): ChatGPT doesn’t provide much insight into how it arrived at its answers, and we can’t know if it’s giving too much weight to out-of-date information. 

Up Next:

In our next blog, we’ll tackle the next topic: ML-authored content.

More recent stories