ChatGPT / Agile #8: If you use ChatGPT output without review or editing, you’re an imbecile

I’ve read a few “mainstream” yarns on AI and ChatGPT in the last week or so — more and more are being published. It made me realise that I hadn’t emphasised this aspect of using ChatGPT strongly enough in Part #6 Some thoughts on capabilities. So I’m rectifying that flaw here.

This is Part 8 of an open-ended series on ChatGPT and how to use it in Agile Project Management. For other parts, check out the master index for this series.

The TL;DR

I read a great article (potential paywall) by legendary linguist Noam Chomsky and two others in the New York Times today (9th March 2023). This yarn revisited the core topic of ChatGPT capabilities vs the endless hyperbolic claims being made in other channels. The conclusion: AI language models such as ChatGPT cannot think in any way related to human thought. They are great at making language jump through hoops in a way that looks like thinking, but it isn’t. Like any tool, it has its uses. Unlike many tools — within its “sweet spot” — the capabilities are wide-ranging and powerful. But, as when you use a screwdriver to stir paint or chisel wood, you own the results. As I mentioned in Part #6, there’s nothing I’ve seen coming out of ChatGPT that I would use directly without editing, verification or some other form of control. If you do build a business or process that uses ChatGPT outputs directly, you are, quite literally, an imbecile.

The NYT Yarn

The article by Noam Chomsky, Ian Roberts and Jeffrey Watumull is a long read and probably behind a paywall. Here’s a short summary generated by ChatGPT and edited by me.

Machine learning programs like ChatGPT operate through statistical pattern recognition and are not reasoning or using language like humans do.
Humans possess an innate “operating system” for language that allows us to create complex sentences and logical principles from minimal exposure to information. ChatGPT, by comparison, consumes huge amounts of data without being able to explain causation.
Machine learning programs lack the most critical capacity of any intelligence: to explain what is not the case and what could and could not be the case, which is the mark of true intelligence.
Unlike humans, machine learning programs are not limited to the kinds of explanations we can rationally conjecture.
ChatGPT and its brethren are stuck in a prehuman or nonhuman phase of cognitive evolution and cannot distinguish between the possible and impossible.
Machine learning enthusiasts often prioritize correct predictions over explanations, but highly improbable theories are more valuable for advancing scientific knowledge.

“True intelligence is demonstrated in the ability to think and express improbable but insightful things.” — Chomsky, Roberts and Watumull (2023)

Machine learning programs are limited in balancing creativity with constraint, which can lead to ethical concerns.
Microsoft’s Tay chatbot’s offensive content in 2016 highlights the dangers of machine learning programs without the capacity to reason from moral principles.

There’s much more to the yarn, but that’s as much as I feel comfortable sharing. (FYI, I probably edited about 40% of the ChatGPT summary).

My Take on This

Intent and Control

As an author/content creator, I believe I have a moral responsibility to ensure that what I write says what I mean and that I have considered the effect or impact it will have on anyone who reads it. That is quite separate from any moral responsibility to not intentionally cause damage by making shit up or saying hurtful or damaging things. By the way, I believe I shouldn’t create such harmful content. However, even if you disagree with me and want to create such content, you should at least be deliberate in formulating your hurtful content as you intended.

The .45 Pistol in the Playpen

I have a mental model of tools that can do damage if used by unskilled or uncaring people. It’s like giving a loaded .45 pistol to a group of kids at kindergarten and expecting nothing bad to happen.

“Damage” might be as benign as creating a detailed schedule that looks great but is fundamentally wrong or incomprehensible. Or it might be creating the harmful content I referred to above. Or anything in between.

But ChatGPT is like giving that kindergarten a container load of weapons intended for the Ukrainian military.

What can go wrong?

The Two Faces of a Natural Language Interface

The nature of the interface — being a natural language with the superficial veneer of human speech — lulls us into this false sense of security that we don’t get from a traditional computer interface. This generality is part of the huge benefit that the chatbot model offers, but it has the downside of avoiding the triggers that keep us aware of what we are interacting with.

Most apps are clearly apps — we are constantly reminded of this as much from faults as working functionality — and they are highly modal: in this screen, we’re loading data. On that screen, we’re displaying data, and on another screen, we’re setting options and so forth. Natural language modality exists but is far more subtle and provides fewer clues. One of the key skills to learn when using ChatGPT is how to cue the model into the mode we want.

But, use the wrong prompt or trigger the wrong modality, and you’ll get the wrong answer. If not wrong, then flawed.

ChatGPT Anthropomorphisation

Because of both the natural language interface and the quality of the responses, I’ve noticed how easy it is to anthropomorphize (develop a perception that treats something as human) ChatGPT and to fall into a state of thinking I’m in a real dialogue with another person. As with movies, we “suspend disbelief” and adopt a social modality.

But it’s critical that users of ChatGPT find a way to maintain a critical and objective relationship with the model when using it.

A tool for experts that produces content for non-experts

Although ChatGPT sits on top of billions (or is it trillions) of data points, the content it produces is (at best) skilled rather than expert. It’s useful as a starting point but needs expansion and fleshing out.

And sometimes it is wrong. The number of identifiable factual errors has been small in my personal use of the tool. But I’ve read of far greater problems experienced by other users.

So, you have to be pretty expert in the domain you’re chatting about to verify the output: quality, accuracy and completeness.

You can’t rely on ChatGPT to do all your thinking for you.

The Bottom Line

I love ChatGPT — I’m still learning and finding new use cases where its power, properly applied, has huge productivity benefits.

But like any tool, especially the sharp ones, it can cut both ways.

I’m very interested in finding out from readers if they have a real use-case in which the ChatGPT output can be used without human intervention. I’m happy to be proven wrong, but until then I’ll say this: it’s dangerous and irresponsible to use the output of ChatGPT without review/edit and especially to warrant or imply that it is human-generated.