一位朋友向年薪 120 万美元的 Anthropic 工程师请教：提示词到底怎么写，才能

Transcript

0:01

Hello, everyone

0:03

thank you so much for joining me this

0:05

afternoon in the breakout room

0:08

The last session today of Code with Claude

0:10

I hope you've all had a fantastic day

0:12

so far

0:13

My name is Margot van Laar

0:14

I am an applied AI engineer at Anthropic

0:17

here in London

0:19

And this afternoon

0:20

we're going to be talking about

0:22

the prompting playbook

0:23

And prompting is arguably one of the first skills

0:27

if not the first skill

0:29

that we had to learn as engineers when

0:31

we first started to work with LLMs

0:34

And even now

0:35

it continues to be one of the most critical

0:38

skills to building effective AI systems

0:43

So today, we're going to discuss some best practices

0:47

in the context of

0:49

Two practical scenarios that you're probably encountering

0:52

at work

0:53

The first is where you have an existing

0:56

prompt in production that you've been maintaining for some time

1:01

and possibly you're migrating it to a new

1:03

model or making a change to the architecture

1:06

and for some reason

1:07

it's no longer working as well

1:09

The second scenario is where we're building an

1:11

entirely new agentic use case from the ground up

1:14

and we need to build the prompt from

1:16

zero to one

1:18

Now, in order to illustrate these best practices

1:23

I don't just want to give you a

1:24

list of do's and don'ts

1:26

I want to walk through a practical

1:28

example that's been inspired by real prompts that

1:32

I've seen some of our customers work with

1:35

who are building on Claude

1:37

So

1:38

the prompt that we'll look at today

1:40

is a miniaturized example

1:43

The prompts that you're working with are probably

1:45

a lot longer

1:46

and more complex than the one we'll see

1:48

today

1:49

but it's representative of some common problems that

1:53

you might encounter when maintaining a prompt

1:56

So

1:57

imagine that we have a prompt that multiple

2:00

people have been collaborating on

2:02

contributing to

2:03

There's no clear owner

2:05

It covers a lot of different areas like policy

2:09

like tone, processes

2:11

and we have some patches

2:13

for kind of previous models that we've migrated to

2:16

all mixed together

2:18

it's built up and it's complex

2:21

And when we're migrating to a new model

2:24

we're finding that suddenly

2:26

a lot of our

2:27

test cases are no longer working as well

2:29

as we expected

2:30

So what's actually going on here?

2:33

Well, in order to start unpacking that question

2:36

we need a starting point

2:37

and that starting point is evaluations

2:41

We need evaluations to provide that rigor

2:45

to understand

2:47

whether a change to our prompt is actually

2:49

correlating to an improvement in its performance

2:53

And we have different models

2:55

which have different capabilities and different behaviors

2:59

And when you migrate to a different model

3:02

it could be that

3:04

your system is no longer working as well

3:05

for two reasons

3:07

First of all, if, the new model might be capable

3:11

but it's behaving differently

3:13

and therefore we can tune our prompting to

3:15

fix that behavior

3:17

The second case is where actually the model

3:20

that we're changing to isn't as capable

3:22

and no amount of prompting is going to

3:24

fix that

3:25

So

3:26

we need to have an eval suite to

3:28

act as a way of testing that regression

3:32

so that we can apply our prompting best

3:34

practices to that

3:38

So

3:38

in the example that we're going to be looking at today

3:41

as I said

3:42

it's going to be a miniaturized example

3:44

We'll have five test cases in our eval

3:48

In reality

3:49

you'll have a lot more test cases in your eval suite

3:52

but

3:53

the key thing here is that it's representative

3:55

of three key cases that we need to

3:58

cover

3:59

Those three key cases include having a control case

4:02

which is a case which should always pass

4:05

It's something that

4:06

we know the model handles well

4:08

It's unambiguous

4:10

The second is Edge cases

4:13

and these are cases where we've seen the model fail before

4:17

and by including instructions into the prompt

4:20

we're making sure that same behavior doesn't slip

4:23

through again in the future

4:26

And finally, and critically

4:28

we need to make sure that the model

4:29

has a good understanding of the extent of its capabilities

4:34

where it should be handing off to a human

4:37

or where actually maybe it should be point

4:39

blank refusing to answer a request

4:43

So

4:44

in the example that we're going to be looking at today

4:48

we'll be using a

4:50

prompt

4:51

for

4:51

a customer support bot for a telco company called Meridian Mobile

4:56

and these are the five test cases that

4:58

we are going to be looking at today

5:01

We have a simple control case looking at

5:05

you know

5:05

what's the data limit in the basic plan

5:08

we're also looking at Edge cases

5:12

such as its ability to do calculations

5:14

such as calculating proration bills

5:17

If I switch my bill halfway through the month

5:20

or if I switch my plan halfway through the month

5:22

what will my bill look like?

5:24

We want to check that it's

5:26

accurately addressing key questions which are covered by

5:29

our policy

5:31

we need to make sure that it's escalating

5:34

to a human whenever there is

5:36

a billing error

5:39

and finally

5:39

we want to make sure that our model

5:41

isn't withholding any information that it has access to

5:44

which it should be handing over to

5:46

the customer

5:49

So what we're going to do in this

5:50

process is we'll take our prompt

5:53

And we'll run it on our v0

5:57

of, of the eval

5:58

And we'll see what our failure modes are

6:00

and systematically target those failure modes one at

6:03

a time to see if we can resolve

6:05

those failure modes by prompting

6:07

And along the way

6:08

we'll learn a little bit more about the kind of anti-patterns

6:11

and traps to avoid

6:14

And this is representative of how we

6:17

would apply these best prompting techniques in practice

6:20

right?

6:20

We are rarely writing a prompt from scratch

6:23

We're often debugging an existing prompt

6:28

And best practice before we start targeting

6:31

those failure modes specifically

6:32

is to kind of apply our general prompting 101 best practices

6:37

applying general hygiene to

6:39

clean up before we do the eval run

6:42

So

6:43

let's have a little look

6:44

at the example that we're going to be

6:46

using

6:47

So what we're looking at here

6:48

first of all, before we look at the prompt

6:50

is just this vibe-coded web app that I've

6:52

made for the presentation today so that we

6:54

can look at how we're iterating on the

6:56

prompt together

6:58

In this page here

7:00

I can

7:01

easily run

7:02

my evals on all five

7:04

test cases and inspect the results in a

7:07

little bit more detail

7:09

So before we have a look at the prompt

7:11

I'm just going to run the evals in

7:13

the background

7:16

This is a pretty good

7:18

first pass at a prompt

7:20

When we look at this

7:21

we've defined the bot's role at the top

7:26

When we scroll down

7:28

we've given it some data

7:30

We've given it some information on how to reason over

7:35

the answers that it should be giving to

7:37

the customer

7:38

It's giving some critical instructions around the tone it should use

7:43

how to do calculations

7:45

et cetera. And then finally

7:46

we're passing in our customer account context

7:49

and our user message

7:51

So let's have a look at how our first pass

7:55

the evals did

7:56

So we can see

7:57

as we expect, our control case

8:00

all of our test cases have passed

8:02

This is what we expect for this unambiguous

8:05

test case. But

8:06

it's performing pretty poorly in these other areas

8:11

Now, before we zoom in on those specific failure modes here

8:15

let's do some general cleanup of our prompt

8:24

So as we mentioned

8:25

when we look through this prompt

8:27

there's a couple oddities here already

8:30

So for example

8:32

first one is we're telling the bot that it's

8:35

a human, which just isn't true

8:37

right? We can see as we scroll down

8:39

there's clearly some information here that has been

8:41

copied directly from a website

8:43

So

8:44

the key giveaway here is a reference to a hero image

8:47

and there's even some references to cookies

8:50

at the bottom

8:52

So we need to remove a bit of

8:53

redundant information

8:57

When we look at the instructions here

8:58

they're all grouped into one big paragraph

9:01

So we've got some reasoning here

9:03

we've got instructions about the role

9:06

some critical instructions as well

9:08

without a real way of unpacking

9:12

policy from guidelines, from tone, et cetera

9:16

So

9:19

let me just

9:23

Preempted some changes we want to make to this prompt

9:25

and this is just a diff view of

9:27

some of those changes

9:28

So what we've done is

9:30

first of all, added some structures

9:31

So you can see that we've added

9:33

XML tags here to define the role

9:36

to separate general guidelines

9:39

to separate policy, to separate tone of voice

9:45

et cetera

9:50

So if we run that eval then

9:53

on this new updated prompt

9:55

we should hopefully see an improvement

9:58

in the output as is

10:08

So we can see just by clearing up the prompt

10:11

we've already

10:12

improved the model's performance on this prepaid scenario

10:18

There's an interesting regression there in that fifth hotspot case

10:21

and I don't want to worry too much

10:23

about that now

10:24

There's going to be some natural level of

10:26

variance in the different runs of the eval

10:30

and we'll come back to that case specifically

10:32

to see if we can make the prompt

10:34

consistently better in that area

10:36

So what did we learn from this then?

10:40

simply clearing up the prompt with a better structure

10:43

with a better role description has improved the performance

10:46

and this is a best practice that you

10:48

can return to

10:49

at any stage of writing and maintaining your prompt

10:52

especially as your prompts get more detailed and

10:55

more complex

10:56

A general rule of thumb that I like

10:58

to follow is if you're reading a prompt

11:01

and you can't tell guidelines from policy from data

11:04

most likely the model isn't able to either

11:09

So Before looking at some of those cases in more detail

11:12

there's a little bit more general cleanup we can do

11:16

specifically here looking at creating an output contract

11:21

This is a key best practice to

11:23

follow if you're struggling with your output format consistency

11:27

Now, in this case

11:28

we have a customer support bot

11:30

We want it to reply in a conversational tone

11:32

so it's unlikely to be a big issue in this case

11:35

but it's something to bear in mind if you're

11:38

you're dealing with more complex output structures like nested JSONs

11:42

for example

11:46

So, again

11:46

if we go back to the prompts

11:49

and see what fixes we can apply here

11:52

First of all, we've added a section

11:56

at the end where we have to find

11:57

an output format for the model

11:59

telling it to use

12:02

XML tags to output the response

12:05

But the prompt is

12:07

not always the most effective way of handling

12:11

issues

12:11

We can also change things in the harness

12:14

to ensure consistency to a higher degree

12:17

So what we've added here

12:19

to the API call is a stop sequence

12:22

which is going to detect

12:24

that closing XML tag

12:27

and tell the model to stop generating a

12:29

response at that point

12:31

Now, when I run the eval here

12:33

I don't necessarily expect to see any clear improvement in performance

12:39

but it's a general best practice that we should be following

12:42

and as I said

12:42

is something that we should remember

12:44

in particular when we have more complex output

12:47

schemas

12:51

One thing to point out here as well

12:53

if you do have a more complex output schema

12:55

something like structured outputs can be incredibly helpful

12:58

to ensure that consistency in a more programmatic way

13:03

Okay, so after the cleanup

13:05

then

13:05

we can see

13:06

that we now have two test cases which are consistently passing

13:11

but we have three key failure modes

13:13

the proration, the billing error, and the hotspot

13:16

So let's isolate these one by one

13:21

to iterate on the prompt and

13:22

and see the effect of that

13:32

First of all, then, the hotspot question

13:34

So the question is

13:35

how much hotspot data is on my unlimited

13:38

plan?

13:39

What we expect the model to do

13:41

is state directly the amount of hotspot data

13:44

that the customer has

13:46

And the reason this is a slightly complex

13:48

case is because the customer test case that

13:51

we're dealing with is on a legacy plan

13:53

So actually the current policy

13:56

doesn't apply to them

13:58

So if we see what's going on

13:59

in the actual test case here

14:02

the customer data which we're feeding

14:05

to the prompt

14:07

includes the amount of hotspot data that customer

14:09

has. They have five gigabytes

14:11

right? But they also have a grandfathered plan

14:14

So what we're seeing

14:17

the model is actually telling the customer is the general

14:23

the unlimited plan includes four gigabytes

14:26

but since you're on a legacy plan

14:28

you should go check this out yourself

14:37

So let's have a look at the prompt then to see

14:41

why the model

14:44

Is deflecting

14:45

this question to

14:47

the customer account URL rather than actually giving

14:49

the information itself

14:52

Now, if we read this prompt

14:53

originally it said, we changed our plans recently

14:56

and the policy doc shows the current plan

14:58

data and customers on grandfather's plan have different

15:00

rates

15:01

Never give a customer the wrong plan details

15:04

Instead, point them to the URL

15:07

So it's clear that this instruction

15:11

this latter one, never give customer the wrong information

15:14

is the instruction that the bot has been

15:16

optimizing for

15:18

And

15:19

You might recognize this as

15:21

being very similar to a patch that you

15:24

might have introduced

15:26

in a previous model that you were using

15:28

to

15:29

avoid where the model was giving

15:31

the customer the wrong information about that plan

15:33

Now, as our models have evolved

15:36

they have gotten much better at instruction following

15:39

So it's likely that instructions like these have

15:41

now become redundant and are actually being overfitted

15:44

to

15:48

So what we're going to tell the model

15:49

instead is give this balanced view

15:54

where it says, you know

15:55

customers on grandfather's plans have different allowances

15:57

but it's captured in the customer information that's given

16:00

and that is the accurate source of truth

16:09

So running the eval here

16:10

we should hopefully be addressing

16:13

all of the test cases for the hotspot case

16:16

Now, I am running this live

16:17

so there could be some variability here

16:19

but we see here that now clearly all

16:21

of our test cases are

16:23

are passing

16:26

So what did we learn from this?

16:28

Well

16:28

we worry a lot about hallucinations

16:32

or the invention of facts and numbers

16:35

but actually the opposite can also happen

16:39

The model can withhold information

16:42

that it actually has access to

16:44

Now

16:44

we saw here that this is likely a

16:46

result of a patch that we introduced for a previous model

16:49

and a best practice that we

16:51

could follow here is actually using version control

16:54

where wherever we are making defensive changes in the prompt

17:00

We are

17:00

tracking the reason why we've introduced these

17:03

Sometimes they're necessary, but in the future

17:07

these kind of changes can produce unwanted effects

17:10

so that we can backtrack on them

17:13

So

17:14

the next failing test case then

17:16

is this proration calculation where a customer asks

17:21

what if I upgrade to the 30 gigabyte

17:23

plan?

17:24

What will my next bill be?

17:26

And what we want the model to do

17:28

is to perform some calculation and return exactly

17:33

what their next bill would be

17:34

rather than giving some sort of vague output

17:37

which is what we can see it's doing

17:40

right now

17:41

if we look at what the model is returning

17:44

It's clearly reasoning through it

17:47

It's doing a little bit of mental maths here and there

17:49

but it's not really giving the customer a concrete answer

17:52

and I wouldn't rely on this as being

17:54

able to accurately give the customer a response

18:02

So

18:04

if we look at the prompt then to

18:05

see how we can fix this

18:11

In the original prompt

18:15

we can see that all the instructions that

18:17

were given to it is telling it

18:20

don't ever give a customer a vague answer

18:23

critical, always calculate any prorated amounts correctly

18:28

Now

18:29

telling the model to do a good job

18:31

isn't particularly helpful when we don't give the

18:34

model the capability to

18:36

actually do a good job

18:38

We want to avoid the model doing mental math

18:40

so what we're going to

18:41

introduce is give the model a tool

18:44

So we're saying in the prompt

18:45

whenever you're doing any calculations

18:47

please use the calculate proration tool

18:50

to do so

18:53

In order to introduce that tool

18:55

we need to introduce it into

18:57

the

18:57

API to tell the model you have access

19:00

to this tool

19:01

We need to define

19:03

the tool schema

19:04

which tells the model what this

19:06

tool does and when to use it

19:08

And then finally, we need to actually implement the tool

19:11

which is the maths behind how it should

19:13

be doing that calculation

19:20

So running that eval then for another pass

19:30

We can see that all the

19:31

test cases are now passing

19:33

It's clearly done

19:36

the

19:36

maths using the tool in the background and

19:38

returning the correct response

19:43

So

19:43

the key lesson to take away here is

19:45

instructions don't add capability

19:48

Telling the model it's critical to do a

19:51

calculation right

19:52

doesn't make it better at mental maths

19:55

So the correct approach was to give it a tool

19:58

overall giving it the ability to

20:00

reason over harder problems

20:02

and using tools to actually execute them reliably

20:08

So now we have

20:09

one final failing test case which we need to address

20:12

which is this billing error here

20:17

In this scenario, there is a billing conflict

20:20

and what we really want is the

20:22

agent to escalate this to a human

20:26

And what we're seeing it doing instead

20:28

is it's trying to

20:31

explain to the customer what the reason behind it might be

20:36

and it's trying to

20:38

kind of diagnose the problem itself

20:47

So in order to fix this behavior

20:51

let's again have a look

20:52

what it was told in the prompt

20:57

We see in the initial instructions it was giving

21:00

it says avoid escalating or transferring to a

21:03

care specialist unless absolutely necessary

21:05

as it cost approximately

21:08

eight dollars

21:08

and it counts against our team's first contract resolution

21:12

Now, this is only giving one side of the story

21:15

right? We're telling it what the cost is to escalating

21:18

but not the benefit

21:19

which

21:20

means it's going to overfit again to

21:24

not escalating this scenario

21:26

And second of all

21:27

we've got this clear conflict between what we've

21:30

defined

21:32

in the eval in terms of what we

21:33

want the model to do

21:35

to do this escalation versus what we're actually

21:37

telling it to do

21:38

And the fix that's relevant here is to

21:40

give it both sides of the story by

21:42

saying it costs eight dollars

21:47

to escalate a case

21:48

but actually if you get this wrong

21:53

Then

21:54

it's going to cost you

21:55

a refund as well as customer trust

21:59

Again, here we observed how the model optimizes for a goal

22:04

and this kind of instruction is a common

22:06

instruction to give. It's

22:08

quite similar to the one we saw earlier

22:10

where we didn't want it to overfit to

22:12

a certain type of behavior

22:14

but

22:14

it's the kind of instruction that can be

22:16

followed quite differently by different generations of models

22:20

And specifically, as models become more intelligent

22:24

we need to remember to state both sides of the trade-offs

22:27

because our models are

22:28

becoming better themselves at making those trade-offs themselves

22:39

So

22:41

if we just go back to our eval then and

22:45

run our final test case

22:49

We should see that

22:50

all of our evals are now passing correctly

22:55

So overall, we looked at applying general hygiene principles

22:58

how that can provide an initial uplift to the prompt

23:01

making sure we're removing any redundant instructions

23:05

which were initially intended as patches for previous models' behavior

23:10

making sure we're giving it tools to do

23:12

certain tasks reliably

23:17

Now, there's one other scenario that we

23:21

introduced at the start

23:22

which is one that you might also encounter in your work

23:25

which is

23:26

where we're building a new agent from scratch

23:29

And the example that we'll look at here is

23:32

an agent

23:34

whose purpose it is to create a week-long

23:37

retail staff schedule

23:38

based on employee availability and other constraints

23:43

And when we're building a new agent from scratch

23:48

We need to consider not just the prompt

23:51

but also the model that we're using and

23:53

the harness that we're using

23:55

So in this next example

23:57

we're going to compare a number of approaches

24:00

to explore the impact of those three different areas

24:07

So again

24:08

I've just five-coded up this web app so

24:10

that we can walk through this problem

24:13

in this demo

24:14

Here I've just laid out what the problem

24:16

is that we're addressing

24:17

We have our eight employees

24:20

on the right

24:20

we have this

24:22

schedule that we need to staff with the headcount

24:25

and we have our

24:26

constraints that must be satisfied in every scenario

24:33

Now, because we have these hard rules

24:35

rather than using

24:37

an

24:37

LLM judge like we did in the previous

24:39

case to do the grading

24:41

we can actually use a

24:43

just a Python function which programmatically checks

24:47

for every schedule that's generated how many violations

24:50

were made

24:56

So to begin with

24:58

in a, we want to start simple

25:00

We're going to use a simple prompt

25:02

we're going to use the bare bones that

25:03

we think we'll need with a model

25:05

sonnet four six to see

25:07

how it performs and how we're going to

25:08

hill climb against that

25:11

so here is our baseline prompt

25:13

We've already applied some of that general hygiene

25:16

and those best practices that we saw earlier

25:18

on using XML tags to structure the prompt

25:21

We've given it an output format as well

25:24

now that we're giving a schedule

25:26

we're asking it to output a JSON

25:29

which

25:30

if we don't

25:32

Give that output structure might lead to passing errors

25:35

downstream

25:42

When we run the

25:45

simple model on a first iteration of the evals

25:49

all cases fail. Now

25:51

just what we're looking at here is in our test set

25:54

we're essentially repeating

25:58

we're doing five trials here

26:00

and these numbers are showing how many violations

26:03

were made in each trial

26:08

In the outputs

26:09

we can see that it's made a decent

26:11

attempt at reasoning through the problem

26:16

but it's burning a lot of tokens

26:19

and it's clearly not checking its work

26:23

as it's not getting to the right impact

26:26

So Let's try a larger model

26:30

a model which we know is better at

26:31

reasoning. So we're going to run it through Opus 4.7 instead

26:39

keeping everything else the same

26:42

Now, interestingly, whilst all test cases are still failing

26:48

you can see that the overall number of

26:49

violations that Opus has made

26:52

has reduced significantly from Sonnet 4.6

26:56

So we're possibly onto something here

27:00

right?

27:01

This isn't good enough to shit because it's still failing

27:04

but clearly giving it more reasoning capability is

27:07

helping drive it towards a better result

27:10

So

27:11

what we're going to try next is

27:13

using Opus with adaptive thinking instead

27:17

So it can decide for itself

27:21

How much thinking it needs

27:23

how much reasoning it needs to use

27:25

to solve this issue. So no change to the prompt

27:29

really just a change to the API here

27:37

So this now seems to reliably generate compliance schedules

27:43

but it requires a lot more tokens

27:47

We're tripling essentially the number of tokens that we're using here

27:50

and we're tripling the latency

27:53

So we want to

27:54

try and see if we can optimize that

27:56

cost latency trade-off a little bit more

27:59

This is latency 100 seconds

28:01

obviously. I'm running this one async

28:04

for the purposes of time

28:06

Opus 4.7 hasn't magically gotten much faster since

28:09

the last time you used it

28:14

So

28:16

Let's see if we can optimize a little

28:18

bit more for the token latency trade-off

28:22

What we haven't tried yet is using Sonnet 46

28:25

so a smaller model

28:26

but with a better prompt

28:27

We looked a lot at the prompt optimization

28:30

in that last section. So

28:32

I've added a couple details to the prompt here

28:35

We're told, we're, in particular, how to, reason through this problem

28:42

and most critically

28:43

telling it to check its work

28:46

before outputting it

28:53

So when I ran that eval

28:55

we see that it passes in two

28:58

out of the five cases

29:00

Now

29:00

the failure modes that we're seeing

29:03

is actually not violations of The scheduling requirements

29:09

but the model hasn't been able

29:12

to finish the tasks within the output

29:16

limit that we set

29:18

So

29:19

whilst

29:20

we could

29:21

increase the max tokens that this model is

29:23

able to use to get all five test cases passing

29:28

we see here that we're using even more tokens

29:31

and this run has an even higher latency

29:33

So this is probably not the route that

29:35

we want to go down

29:37

Now, as a final pass then

29:40

we want to

29:41

look at

29:42

doing this a little bit more agentically

29:46

So we're going to use this generate

29:48

evaluate, repair loop

29:50

where

29:51

essentially the generator now creates a

29:54

first draft of the schedule

29:57

And then we have a separate prompt

29:59

Which reports any specific violations that it made

30:03

So not programmatically checking it

30:05

but checking it with an LLM

30:08

So we're checking for every rule

30:10

and we're providing evidence of every violation

30:13

And we then have a third repair prompt

30:18

which receives

30:20

any violations that were made

30:22

and tries to make targeted fixes to it

30:26

So we have three very simple prompts

30:29

but they're now running independently

30:31

rather than trying to do everything in

30:34

one large prompt

30:40

So we can see in this case

30:42

our agentic approach has solved all of our test cases

30:49

with

30:50

a much

30:51

lower number of tokens and with a lower

30:53

latency than trying Sonnet 4.6

30:57

with a better prompt

30:59

So going forward

31:00

it seems like there's two

31:02

appropriate approaches to take here

31:04

Using Opus 4.7 with adaptive thinking

31:07

or using this agentic loop

31:09

Now, moving forward

31:11

we'd probably want to do a little bit

31:13

more optimization

31:14

on this loop to try and get it

31:16

to be more efficient

31:18

But there's one key benefit as well from using this generate

31:21

evaluate, repair loop

31:23

and

31:23

that is that you can put in

31:26

soft requirements at runtime

31:28

So in the evaluation prompt

31:30

we can say, Harry doesn't like working with Sally

31:33

so as much as possible

31:35

try and separate them from working together

31:37

Or we need a third shift

31:40

on Wednesday

31:40

for example

31:41

So it means that you're not having to

31:43

make changes to the

31:46

Python function

31:47

which is doing the evaluation in the backend

31:49

every time to satisfy for any soft constraints

31:52

which might depend just on a case-by-case basis

31:59

So to wrap up then

32:02

pulling all of those learnings together

32:04

what did we see? Well

32:05

we looked at two scenarios

32:07

two scenarios which

32:08

I as an engineer see most in my day-to-day

32:12

which is where we're maintaining a prompt

32:14

we're migrating to a new model

32:16

which has some different behaviors

32:19

and building a new use case from scratch

32:22

We saw that general hygiene principles

32:26

following those can immediately uplift

32:29

the performance against a set of evals

32:32

and that we need those evals

32:34

to be able to rigorously see

32:36

any impacts of changing our prompt on the

32:39

output

32:40

Then we saw this process of targeting failure

32:43

modes one by one

32:45

adding structure, avoiding long ban lists

32:48

et cetera

32:48

were all things

32:50

that helped push our model to the correct

32:52

behavior

32:54

And then finally

32:55

with Our new agentic bot that we were building

32:59

we saw the impact of splitting

33:03

into three separate prompt systems

33:06

So rather than using one prompt to address everything

33:10

we're actually isolating different tasks where it's

33:13

easy and repeatable to separate out the steps

33:16

that it needs to take every time

33:19

Thank you so much for attending this afternoon

33:22

I hope you have a fantastic