Why "Reverse Prompt Engineering" is the Magic Key to Production-Ready Prompts

The advent of Large Language Models (LLMs) is like the birth of the internet. And just as the internet connected the world in unprecedented ways, LLMs are now connecting the dots of Artificial Intelligence.

Naturally, this breakthrough has sparked the development of countless AI applications and tools. And “Prompt Engineering”, the art of crafting queries that elicit the desired responses from these LLMs, has thus become a hot topic.

The rise in the demand for “production-ready” prompts

With this influx of LLM-based applications, the need for what can be termed “production-ready” LLM prompts has emerged. These “production-ready” prompts need to be meticulously crafted and re-iterated to ensure they are concrete, precise, and capable of handling the given use case in the best way possible.

Now it goes without saying that the effectiveness of these prompts will make or break the performance of the AI application they are empowering.

However, Prompt Engineering is more of an “Art” than “Science”

While there are guidelines to follow, the entire process of crafting prompts does have a nature of subjectivity to it.

And just like art, this entire process revolves around creativity and experimentation. In fact, successful prompt engineering requires an understanding of technology as well as the nuances of language.

It’s very different from traditional programming…

Traditional Programming is like painting with a fine brush

These Programming languages are more “deterministic” in nature. When you write code in Python, Java, or any other language, the code behaves exactly as expected, following predefined rules and structures.

It’s like using a fine brush to meticulously paint every detail on the canvas. There’s no room for interpretation or creativity. Each line of code is executed with exact precision, just as a fine brush captures the finest details of a painting.

In contrast, Prompt Engineering is like using a broad brush

When you create a prompt for an AI model, it’s like making a bold, general stroke. You basically give an overall idea of what you want, and then you let the LLM interpret it. The LLM might not pay meticulous attention to every intricate detail of your request.

It’s like a painter using broad strokes to convey the essence of a scene without focusing on every blade of grass or leaf on a tree.

To illustrate this with an example, see how summarizing a given block of text can be achieved through different prompts:

• “Please provide a concise summary of the following text:<text>”
• “Summarize the key points in the following text:<text>”
• “Can you give me a brief overview of: <text>”

… and so on.

So then how can we even craft a good “production-ready” prompt?

It’s fair to think that using very specific keywords is the key to control in prompt engineering. However, as we have seen above, keywords are not as rigid in language models as they are in Traditional Programming languages.

Achieving a good “production-ready” prompt is thus no small feat, and hence, it is evident that we need some sturdy assistance in setting up these prompts.

Enter “Reverse Prompt Engineering”

The concept of “Reverse Prompt Engineering” has emerged as a powerful technique that brings a certain level of precision to AI interactions.

It’s like reverse engineering for AI, allowing us to leverage the generative capabilities of Large Language Models (LLMs) to craft precise and effective prompts. Basically, we provide the desired output to the LLM and then request it to generate the most accurate prompt that can produce such an output.

Now “Reverse Prompt Engineering” is of 2 types:

1. Macro
2. Micro

Let us take a look at each of these types in detail.

“MACRO” Reverse Prompt Engineering

In the “Macro” approach, we first present the model with the desired output or a specific scenario, and then instruct it to generate a prompt capable of either reproducing this desired output or accurately handling the specified scenario.

Thus, the Macro approach has two sub-types:

1. Example-Based
2. Scenario-Based

MACRO Type-1: “Example-Based”

Example-based Reverse Prompt Engineering is a powerful technique that involves creating a prompt template by analyzing existing examples of desired output and then using this template to consistently generate similar results.

To illustrate this, let’s assume that we are a Fintech company and we wish to generate taglines for our finance products. Suppose we admire the iconic MasterCard tagline, “There are some things money can’t buy. For everything else, there’s MasterCard”, and we now want to write a prompt that can help us generate similar taglines.

Let’s see how we can do that through the “Example-Based” Reverse Prompt Engineering:

• STEP-1: Explain the complete task. We begin by informing the LLM that we need its assistance in analyzing a provided tagline and summarizing its core message to turn it into a prompt.

[Our prompt]---------------

You are an expert in "Reverse Prompt Engineering".
You are able to reverse-engineer prompts from
a text given to you. 

I will be providing you with a "tagline". 
Please try to be as specific as possible
and come up with a prompt that is based
on the tone, style, and any other 
properties you consider relevant. 

I will be using the prompt you generate
in the future to generate similar taglines. 

Let me know if you have understood and 
I will then move ahead and provide 
the tagline to you. 

If you need any clarifications, 
please feel free to ask them.

[ChatGPT's Response]---------------

I understand your request. 
Please go ahead and provide the tagline, 
and I will generate a prompt
based on the tone, style, 
and other relevant properties. 

If you have any specific criteria 
or guidelines for the prompt, 
feel free to share them as well.

• STEP-2: Provide clarifications (if required). If GPT asks for any clarifying questions, we’ll answer them. In our case, GPT seems to understand the task well, so we can skip this step.
  
  
• STEP-3: Provide the “Example”. Now, we share the MasterCard tagline we admire.

[Our prompt]---------------
Okay, the tagline is as follows:

MasterCard: 
'There are some things money can't buy. 
For everything else, there's MasterCard.'

[ChatGPT's Response]---------------
Thank you for providing the tagline. 
Here's a prompt generated based on 
the tone and style of the tagline:

"Create a compelling marketing slogan 
for a credit card company 
that conveys the idea that while 
some things in life are priceless, 
this credit card is the solution 
for all your financial needs. 

Make it impactful and memorable."

• STEP-4: Convert the generated prompt into a prompt template. With the prompt created, instruct the LLM to transform this prompt into a reusable template.

[Our prompt]---------------
Can you convert this into a prompt 
template with fill in the blanks?

[ChatGPT's Response]---------------
Certainly! Here's a prompt template 
with fill-in-the-blanks:

"Create a compelling marketing slogan 
for a [Type of Financial Product] that 
conveys the idea that while some things 
in life are [Adjective], 
this [Type of Financial Product] is the solution 
for all your [Type of Financial Needs]. 

Make it impactful and memorable."

MACRO Type-2: “Scenario-Based”

While “Example-Based” Reverse Prompt Engineering excels when we have a clear target output in mind, there are situations where we need to create prompts tailored to specific scenarios. This is where the “Scenario-Based” Reverse Prompt Engineering method comes into play.

This is a more methodical and interactive approach where we provide a “Scenario” to the model and then ask it to create a prompt for handling it.

For example, let’s say we have the transcript of a conversation between a User and an Agent, and we want the model to:

Now yes, the prompt template created through ‘Example-Based’ Reverse Prompt Engineering might not be perfect, but it does offer a solid foundation that can be easily refined and repurposed.

1. Give us a “Summary”
2. Analyze the “Sentiment”
3. Tell us if the User’s issue got resolved.

Now, let’s figure out how to create the perfect prompt for this scenario.

This process involves multiple steps. Let’s break down each step.

• STEP-1: Priming the model. Here, we offer context to the LLM by initially asking for an explanation of “Reverse Prompt Engineering” and requesting a few examples to illustrate it. We do this in the following way…

[Our prompt]---------------
Can you tell me more about 
"Reverse Prompt Engineering"? 
Also, please provide some examples

[ChatGPT's Response]---------------
"Reverse Prompt Engineering" could 
potentially refer to the process 
of reverse-engineering the prompts 
or instructions used in various AI systems.
In natural language processing tasks, 
AI models are often provided...

• STEP-2: Further priming the model by asking it to act as an expert in “Reverse Prompt Engineering”. With the model now having context about “Reverse Prompt Engineering,” we proceed to ask the model to act as an expert in this field. We also provide specific instructions on the role it should play.

[Our prompt]---------------
You are an expert in 
"Reverse Prompt Engineering". 
That is, when presented with a scenario, 
you excel at crafting the ideal 
input prompt for any Large Language Model (LLM), 
ensuring it produces the precise outcome.

Let me know if you have understood 
and I will then proceed in 
giving you a scenario.

[ChatGPT's Response] ---------------
I understand the concept you're describing. 
If you'd like assistance with crafting 
an input prompt for a Large Language Model 
based on a specific scenario, 
please go ahead and provide the scenario, 
and I'll do my best to help you formulate 
the ideal prompt for the desired outcome.

• STEP-3: Providing the “Scenario” and requesting the model to ask for clarifications. Once the model has been primed, we introduce the specific “Scenario” we wish to address. At this point, we take a crucial step by inquiring whether the model has a clear understanding of the scenario and if it needs any clarification or has questions to ensure the prompt is accurately crafted.

[Our Prompt]---------------

[Scenario Starts]

The LLM's task is to analyze 
a conversation between a human agent 
and a customer. From this transcript, 
the LLM will provide:

1. SUMMARY: A brief summary of the conversation.
2. SENTIMENT: The customer's sentiment.
3. RESOLUTION: A simple "Yes" or "No" 
response to indicate 
whether the customer's issue was resolved.

[Scenario ends]

Now that you have seen the scenario, 
are there any questions you have?

[ChatGPT's Response] ---------------
Thank you for providing the scenario. 
To ensure the Large Language Model (LLM) 
can provide the desired outputs, 
I have a few clarifying questions:

1. How will the conversation transcript 
be provided to the LLM? 
Will it be included in the input prompt, 
or will the LLM be given access 
to the transcript in some other way?


2. Can you specify the criteria 
for determining the customer's sentiment? 
Are you looking for a general sentiment analysis 
(e.g., positive, negative, neutral) 
or a more detailed analysis 
(e.g., angry, satisfied, frustrated)?


3. Are there any specific keywords 
or phrases in the conversation 
that the LLM should pay attention to 
when determining whether the customer's issue 
was resolved? 
How should the LLM identify the resolution
in the conversation?


4. Is there any desired format 
for the SUMMARY output, 
such as character or word limits?


Once I have this information, 
I can help you formulate an ideal 
input prompt for the LLM.

• STEP-4: Providing the needed clarifications, if any. If the model does ask for clarifications (as it did in our case), we proceed to offer the required information to ensure it has a precise grasp of the scenario.

[Our Prompt]---------------

Sure, let me clarify point-by-point:

1. The transcript will be provided 
as a part of the prompt. 
The transcript will mostly be in the format of:
AGENT: <this is what the agent said>
CUSTOMER: <this is what the customer said>


2. Only general sentiment analysis 
- Positive, Negative, and Neutral


3. Mostly, the AGENT will ask the customer 
if their issue was resolved. 
So we can use that as a way 
to determine if the issue was resolved or not. 

If the AGENT does not ask this question, 
then feel free to skip determining the RESOLUTION 
and respond back with "null"


4. Please keep the SUMMARY below 200 words.

• STEP-5: Waiting for the prompt to be generated. Finally, the model leverages this information to generate a prompt that is custom-tailored to the given scenario, optimizing the outcome for the intended interaction.

Thank you for the clarification. 
Based on your inputs, 
here's an ideal input prompt for the 
Large Language Model (LLM) to perform the tasks of 
analyzing the conversation transcript 
and providing the desired outputs:


Conduct a conversation analysis between a human agent and a customer. 
Please provide the following:

1. SUMMARY: 
A brief summary of the conversation 
(limit: 200 words)


2. SENTIMENT: 
The general sentiment of the customer 
(choose from: Positive, Negative, Neutral)


3. RESOLUTION: 
Determine if the customer's issue was resolved 
by checking if the AGENT asked the customer 
if their issue was resolved. 
If this question is asked, 
respond with "Yes." If not, respond with "null."


Transcript:
AGENT: <this is what the agent said>
CUSTOMER: <this is what the customer said>

This entire process of scenario-based reverse prompt engineering combines human intuition and machine-generated precision to create prompts that are finely tuned to the context, resulting in highly effective interactions with AI models.

“MICRO” Reverse Prompt Engineering

“Micro” Reverse Prompt Engineering comes in handy when things get a bit tricky, and the “Macro” approach isn’t giving us the best prompt.

With the “Micro” approach, we zoom into the prompt and focus on the specific keywords we need.

The core difference is, unlike “Macro,” we’re not asking the model to come up with the whole prompt. We are instead asking for its help in figuring out the right keywords we should use to ensure the prompt we create is crystal clear for the model.

It’s like getting a little extra guidance for a more accurate outcome.

Let’s understand this with an example —  Training a Chatbot

Assume we are training a Chatbot. Now training a Chatbot requires exposing it to diverse user inputs, and thus, the process of generating multiple variations of a given user query/utterance becomes crucial.

Our task, therefore, is to use an LLM for generating multiple linguistic and semantic variations of a given user utterance.

In the “Macro” approach, we would have just explained the entire scenario and asked the LLM to generate the complete prompt. However, “Micro” Reverse Prompt Engineering takes a more nuanced path.

• STEP-1: Seeking Guidance for the Right Keyword. Rather than overwhelming the LLM with the entire task, we start by asking for guidance on the technical terms for the process we need.

[Our Prompt]---------------

What is the specific technique called when you 
rephrase a given user utterance in different words 
to create linguistic and semantic variations 
for chatbot training?

[ChatGPT's Response] ---------------
The specific technique you are referring to 
is called paraphrasing.
 
Paraphrasing involves expressing the same meaning 
as the original statement but 
using different words or structures. 
In the context of chatbot training, 
paraphrasing user utterances helps create a more 
diverse set of training data, allowing the chatbot 
to better understand and respond to 
a variety of user inputs. 

It contributes to the robustness and flexibility of 
the chatbot's language understanding capabilities.

• STEP-2: Crafting a Clear Prompt. Now that we know ‘Paraphrasing’ is the right keyword for our task, our next step is to craft a prompt that precisely guides the model. So we can use prompts like…

[Our Prompt]---------------
Please generate accurate paraphrases 
of the following user utterance...

or,

[Our Prompt]---------------
You are an advanced paraphrasing system. 
Given a user input like...

Thus, by focusing on specific keywords and seeking guidance rather than complete prompt generation, this technique cleverly avoids overwhelming the LLM. Instead, it seeks the LLM’s guidance bit by bit, helping us build accurate prompts in a more manageable way.

So we are writing prompts to generate prompts? Sensing a paradox here

Yes, we’re in this interesting loop of writing prompts to make prompts. It does sound like a paradox, and it is.

The Reverse Prompt Engineering Paradox

As we have seen, Reverse Prompt Engineering is where we generate a prompt using an LLM model, but the twist is, that the input itself is a prompt.

Thus, it is fair to wonder, how can we be sure that this entire technique is even reliable. How do we know that the prompts we are using to generate the “production-ready prompts” were accurate in the first place?

Well, this is a bit paradoxical, but we’re okay with it.

Why?

Because the prompts we use for Reverse Prompt Engineering are way simpler than the final “production-ready prompts” we’re trying to create with this technique. Thus the chances of making a mistake are much lower.

So, even with the paradox in play, we can trust Reverse Prompt Engineering to help us create those “production-ready prompts” with confidence.

But are these generated prompts really “production-ready”?

Sometimes yes, sometimes no. Often no.

It is important to note that crafting successful production-level prompts is an iterative process. It takes some refining and tweaking. Thus, claiming that prompts from Reverse Prompt Engineering are spot-on right away would be both unfair and inaccurate.

But it does give us a strong leg up

The strength of Reverse Prompt Engineering comes from the fact that it gives us valuable insight into the most effective way to structure a prompt so that the LLM can understand the task clearly.

In that way, Reverse Prompt Engineering saves us from a lot of iterations by giving us a solid first draft of what would eventually become our final “production-ready” prompt.

So, rather than viewing it as the final destination, think of Reverse Prompt Engineering as the crucial first step in the journey of crafting precise and effective prompts.