ChatGPT: The Good, The Bad and The Hyperbole

Part 3: ChatGPT and Coding Activities

By BC, Chief Technologist | Published: February 23, 2023 in Blog

In our first two blogs, ChatGPT as a search engine and ChatGPT and written content, we discussed some of the new ways that people are using ChatGPT: both as a general search tool, and as a tool to help author (or finesse) written content. In this blog, we’re going to look at ChatGPT’s ability to perform coding activities.

ChatGPT and Coding Activities 

One of the surprising things that many YouTube videos and blog articles about ChatGPT bring up is the fact that ChatGPT can code. And that’s not untrue. I can clearly prompt ChatGPT, Write me a JavaScript program to calculate the 500th Fibonacci number” or Write an authentication function for Spring Rest.”  

ChatGPT can do these things, but that’s barely a scratch compared to the effort of a real application development project. ChatGPT can also debug” code, all examples I have seen are comparable to syntax- or spelling-type error checking available in almost every IDE.  

So, I confess that I find the current claims of ChatGPT’s coding prowess to be unpersuasive. But I’m not saying that ChatGPT can’t be useful for coding-related tasks. 

ChatGPT is currently causing panicked meetings in colleges and high schools across the continent regarding its implications for essay writing. And I think we may have to entertain similar concerns for technical interviews. I think that ChatGPT can pass most exercises I might use in a technical interview, but I’m also confident that ChatGPT can’t do the job I need of an application developer. What gaps does that expose when I’m assessing technical interviewees? (I grant that high schools and colleges have pedagogical constraints that don’t apply to technical interviewing, so their apprehensions might be thornier to solve than mine). 

While I would not expect ChatGPT to code a whole application, there are other code-related use cases. Some development tasks are exploratory: when assessing new tools, for example, I might spend a certain amount of time figuring out how to get a quick sample up and running. I can easily ask ChatGPT questions like, Can you show me how to set up a sample Vue.js application?” Or Can you show me some sample Go code to connect and send a message to SQS?” Or Can you write a JUnit test for FizzBuzz?” ChatGPT gives me fast and accurate answers to such prompts. 

Google often fails me in these circumstances: I frequently find out-of-date instructions that have propagated to countless technical blogs and content sources. I can easily spend a few hours sifting through different tutorials trying to find one that’s up-to-date and that works correctly. The absence of publication dates on so many web sites exacerbates the problem. There’s a danger that ChatGPT will encounter the same relevance problem, but in the short term, ChatGPT looks to be more-recently tuned than Google, and it appears to produce better results. 

Much like I argued that ChatGPT might be the first point of call” for certain types of search queries, I think that it can become the first point of call” for some questions that we might have sought out on Stack Overflow. ChatGPT is not yet good at questions such as This framework produced this unusual error message; how do I fix that?” But it’s arguably better (and more timely) with questions such as what are the popular Swift frameworks for feature flags?” That could lead to some interesting pressures on Stack Overflow. 

To my mind, here’s an important thing to recognize: while today, ML/AI-generated code is in its infancy, that’s going to change quickly. At the moment, other ML/AI tools such as Copilot and Codex are more specifically tuned to the code creation space and thus ChatGPT might not be the tool that pushes for change in the way we do application development. But the change is coming. 

As much as I want to take comfort in the belief that the work I do requires a certain amount of skill and experience, very well-informed experts have different assessments that give me pause. Sam Altman, CEO of OpenAI, talks about the evolving perceptions of how ML/AI will impact jobs. Even five years ago, people predicted that ML/AI will start to cannibalize repetitious, low end” jobs such as manufacturing, or low skill white collar jobs such as admin, but that high-skill jobs such as programming or highly creative jobs such as illustration or photography will be safer longer. In actuality, he argues, what we’re seeing is the opposite: that ML/AI is having a much bigger impact on the high-skill / creative end of that spectrum, and that trend is only going to accelerate over the next decade. 

While nobody is yet predicting the end of application developers, I do think that we should be ready for fairly dramatic change in terms of what application development is going to look like over the next decade. If nothing else, the pressure to produce more, significantly faster, will be high. 

In Conclusion 

I agree with the assessment that ML/AI will be the major technology that will shape the next decade of IT systems and services. And those changes are going to be much more profound than Cloud technology. While some application development projects will interact with ML/AI systems in a way that resembles straight-forward integration with something involving AI/ML models”, the biggest changes are going to involve radical rethinking of foundational ways we build applications. ChatGPT feels like a tipping point because, I believe, it is a tipping point.  

ChatGPT, itself, isn’t going to bring about that radical change, but I think we will be better prepared for how IT services are going to be upended if we embrace and think through some of the ways that these ML/AI technologies are going to change the way we work.