ChatGPT / Agile #10: The Short-term Thinking of Lazy Prompt Engineering

In this yarn, we’ll first cover the downside and short-term thinking of using canned prompts. It might look attractive but it is really bad for you. I’ll explain why it will probably keep you in the “Suck Zone” of ChatGPT mediocrity (at best).

I’ll then summarise some prompt types from one of the books I reviewed in the last post. I’ll compare this to the list of prompt types that ChatGPT says it supports.

This is a second yarn about Prompt Engineering — a truly important skill if you’re going to make the most of ChatGPT. Or make the most of any AI capability that sports a general-purpose Natural Language interface.

In the first yarn, I diarised my early exposure to Prompt Engineering and looked at a few books.

I developed — in partnership with ChatGPT — two tables of prompt types but bugger me if I find out that the Medium editor doesn’t support tables entered directly. After wasting an hour looking at various claimed workarounds, I put these into Google Sheets. Feel free to check them out here: AI Language Model Prompts Summary

So, to our first topic — lazy Prompt Engineering

Don’t Copy+Pasta Pre-written Prompts!

At least, not for long. Use them as training material to get you started. Then ensure you understand how they work and either adapt or ditch them.

I see a growing stream of products available in which someone has developed a canned list of prompts that can be copied and pasted (“copy+pasta”) into the ChatGPT.

Many of these resources are loosely categorised lists of prompts. At least one tries to define a problem statement and provides some prompts to help solve those problems.

If you don’t understand it, don’t use it. It’s challenging enough to learn how to make the tool work correctly and understand/validate the output. I suspect that if you’re predisposed to copy+pasta canned prompts, then you might also be inclined to use the output uncritically.

Maybe this will work for you — it’s certainly quick. And maybe it’s not going to work.

There’s a conceptual model that’s used in Agile to describe people or teams who use a tool but don’t understand how it works: it’s called the “Cargo Cult” mentality. I won’t spend any time on it, but it’s not good. Check out the link or google the term or even ask ChatGPT!

What’s the Alternative? Learn some basic things about the tool!

We all start out as novices at some particular skill or as tool users. And when we start out, we all “suck” at it. But with the right kind of practice and help, we get better.

Kathy Sierra describes it like this:

Along the way they’ll hit many milestones, some more crucial than others. But of all points on the curve, the two most important are crossing the Suck Threshold and crossing the Badass Threshold. in “Badass: Making Users Awesome” (p. 67). O’Reilly Media. Kindle Edition.

The emphasis here is on the word “right”. Just using the tool won’t help you get out of what Kathy Sierra calls “The Suck Zone”. You have to try different learning strategies to make the kind of progress shown in the above diagram.

Copying and pasting prompts that someone else has put together is almost guaranteed to keep you in the Suck Zone forever.

Don’t Forget the Science!

Amongst all the breathless excitement about this new technology, it’s easy to forget (wilfully ignore?) that there is a whole field of science devoted to Artificial Intelligence. There’s another (even older) field of science devoted to Linguistics that is even more fundamental to ChatGPT.

Beyond the digestible yarns in general publications, these sciences get very complicated — very quickly. Steven Wolfram’s article I mentioned in one of the early parts of this series is about as complex as I can wade through, yet it’s still considered a general article.

So, my investigation into Prompt Engineering lead me into two similar but distinct worlds: what general science calls something and what OpenAI call it. Keep note of this distinction as we progress.

The First thing to Learn: Prompt Types

When I first started with ChatGPT, I did something really weird: I read some of OpenAI’s documentation. One of the pages listed a few “text generation types” and said something like ‘this is a partial sample’. You can see this below by type and example.

ChatGPT Prompt Types

Being more familiar now with the tool, I simply asked it what prompt types it supports. I think I used the word “technique”. It took me a couple of attempts, but once I finally included the word “exhaustive”, ChatGPT gave me a list that it affirmed was everything. You can see it in the tab “GPT Prompt Techniques” of the AI Language Model Prompts Summary GoogleSheet.

Initially, the list was 64 prompt types, but as I requested more information, like a description and examples, it ended up at 60. I still haven’t found out why. Perhaps the “exhaustive” list is longer than 64.

Generic NLP Prompt Types

I said in my previous article that I liked Ibrahim John’s book “The Art of Asking ChatGPT for High-Quality Answers: A Complete Guide to Prompt Engineering Techniques”. And I still do. He gives some functional structure to the content, unlike most of the others. Unfortunately, like most of the books I cited, the writing isn’t easy and feels a bit rushed. I suppose this is unsurprising in the current marketplace.

So I listed out the prompt types and gave them to ChatGPT.

John includes 23 Prompt types as chapter headings, but the interesting thing is that ChatGPT expanded this to 38 different types

Comparing and Contrasting the Generic Prompt Types with ChatGPT’s Prompt types

Here I used ChatGPT straight off the bat:

AOP: please compare the GPT prompt techniques in PromptTypesv1 with the  
general NLP prompt types in  NLPPromptsv1. what are the overlaps and  
differences?

Note: the IDs “PromptTypesv1” and “NLPPromptsv1” are the handles I gave to the two lists to make it easier to refer to them in more complex prompts. “NLPPromptsv1” is the extended Ibrahim John list of 38 items, and “PromptTypesv1” is the ChatGPT list of 60 (or 64) items.

I’m not going to put GPT’s response in a code block for simplicity of formatting. I’ll just bracket it with a heavy line.

=======================

“GPT: Sure, here are some overlaps and differences between the prompt types in PromptTypesv1 and NLPPromptsv1:

Overlaps:

• Multiple Choice prompts
• Summarization prompts
• Adversarial prompts
• Clustering prompts
• Text classification prompts
• Text generation prompts

These prompt types are used in both PromptTypesv1 and NLPPromptsv1 and involve generating responses that fall into predefined categories, summarizing text, challenging the model with adversarial input, grouping similar concepts together, classifying text into categories, and generating new text based on a given input or prompt.

Differences:

• PromptTypesv1 includes a broader range of prompt types, such as creative prompts, opinion-based prompts, hypothetical prompts, personal experience prompts, historical prompts, artistic prompts, and so on.
• NLPPromptsv1 includes prompt types that are specifically related to natural language processing tasks, such as named entity recognition prompts, sentiment analysis prompts, language modeling prompts, and parsing prompts. These prompt types are more focused on specific NLP subtasks.
• NLPPromptsv1 also includes some prompt types that are related to training or fine-tuning language models, such as reinforcement learning prompts, curriculum learning prompts, and interpretable soft prompts.

Overall, PromptTypesv1 is a more general list of prompt types that can be used for a wide range of tasks, while NLPPromptsv1 is focused specifically on prompt types that are relevant to natural language processing.”

[GPT ends]

=========================

Remember, in the above, “NLPPromptsv1” is the extended Ibrahim John list of prompt types, and “PromptTypesv1” is the ChatGPT of prompt types.

For users of ChatGPT, please refer to the full details in the tab “GPT Prompt Techniques” of the AI Language Model Prompts Summary GoogleSheet.

The Bottom Line

Learning a new tool always takes time, and you always have to balance that time investment against the likely payoff that will derive from becoming more skilled. You may not become a “Badass” at ChatGPT, but there’s no point in being in the Zone of Suck.

This is a very interesting problem and has many parallels with learning Agile itself. I’ll have to find a way to weave those ideas into this series somehow.

In the next post on Prompt Engineering, I will cover the second thing you have to learn: Prompt Structure.

Stay Tuned.