This is f*cking insane. One of my friends makes $1.2M a year as an Anthropic engineer Fable 5.2 dro

Transcript

0:01

Hello, everyone

0:03

thank you so much for joining me this

0:05

afternoon in the breakout room

0:08

The last session today of Code with Claude

0:11

I hope you've all had a fantastic day

0:12

so far

0:13

My name is Margot van Laar

0:15

I am an applied AI engineer at Anthropic

0:18

here in London

0:19

And this afternoon

0:21

we're going to be talking about

0:22

the prompting playbook

0:24

And prompting is arguably one of the first skills

0:28

if not the first skill

0:30

that we had to learn as engineers when

0:31

we first started to work with LLMs

0:34

And even now

0:36

it continues to be one of the most critical

0:39

skills to building effective AI systems

0:44

So today, we're going to discuss some best practices

0:48

in the context of

0:50

Two practical scenarios that you're probably encountering

0:53

at work

0:54

The first is where you have an existing

0:57

prompt in production that you've been maintaining for some time

1:03

and possibly you're migrating it to a new

1:05

model or making a change to the architecture

1:07

and for some reason

1:09

it's no longer working as well

1:11

The second scenario is where we're building an

1:13

entirely new agentic use case from the ground up

1:16

and we need to build the prompt from

1:17

zero to one

1:20

Now, in order to illustrate these best practices

1:24

I don't just want to give you a

1:26

list of do's and don'ts

1:28

I want to walk through a practical example

1:30

that's been inspired by real prompts that

1:35

I've seen some of our customers work with

1:37

who are building on Claude

1:40

So

1:41

the prompt that we'll look at today

1:42

is a miniaturized example

1:45

The prompts that you're working with are probably

1:47

a lot longer

1:49

and more complex than the one we'll see today

1:52

but it's representative of some common problems that

1:55

you might encounter when maintaining a prompt

1:59

So

1:59

imagine that we have a prompt that multiple

2:02

people have been collaborating on

2:05

contributing to

2:06

There's no clear owner

2:08

It covers a lot of different areas like policy

2:12

like tone, processes

2:14

and we have some patches

2:16

for kind of previous models that we've migrated to

2:19

all mixed together

2:22

It's built up and it's complex

2:24

And when we're migrating to a new model

2:27

we're finding that suddenly

2:29

a lot of our

2:30

test cases are no longer working as well

2:32

as we expected

2:34

So what's actually going on here?

2:36

Well, in order to start unpacking that question

2:40

we need a starting point

2:41

and that starting point is evaluations

2:45

We need evaluations to provide that rigor

2:49

to understand

2:50

whether a change to our prompt is actually

2:53

correlating to an improvement in its performance

2:57

And we have different models

3:00

which have different capabilities and different behaviors

3:03

And when you migrate to a different model

3:07

it could be that

3:08

your system is no longer working as well

3:10

for two reasons

3:11

First of all, if, the new model might be capable

3:16

but it's behaving differently

3:18

and therefore we can tune our prompting to

3:20

fix that behavior

3:22

The second case is where actually the model

3:24

that we're changing to isn't as capable

3:27

and no amount of prompting is going to

3:29

fix that

3:30

So

3:31

we need to have an eval suite to

3:33

act as a way of testing that regression

3:37

so that we can apply our prompting best

3:39

practices to that

3:43

So

3:43

in the example that we're going to be looking at today

3:47

as I said

3:47

it's going to be a miniaturized example

3:49

We'll have five test cases in our eval

3:54

In reality

3:54

you'll have a lot more test cases in your eval suite

3:58

but

3:58

the key thing here is that it's representative

4:01

of three key cases that we need to

4:03

cover

4:04

Those three key cases include having a control case

4:08

which is a case which should always pass

4:11

It's something that

4:12

we know the model handles well

4:14

It's unambiguous

4:16

The second is Edge cases

4:19

and these are cases where we've seen the model fail before

4:24

and by including instructions into the prompt

4:27

we're making sure that same behavior doesn't slip

4:29

through again in the future

4:32

And finally, and critically

4:34

we need to make sure that the model

4:36

has a good understanding of the extent of its capabilities

4:41

where it should be handing off to a human

4:44

or where actually maybe it should be point

4:46

blank refusing to answer a request

4:50

So

4:51

in the example that we're going to be looking at today

4:55

we'll be using a

4:57

prompt

4:58

for

4:58

a customer support bot for a telco company called Meridian Mobile

5:03

and these are the five test cases that

5:05

we are going to be looking at today

5:08

We have a simple control case looking at

5:12

you know

5:13

what's the data limit in the basic plan

5:16

we're also looking at Edge cases

5:19

such as its ability to do calculations

5:22

such as calculating proration bills

5:25

If I switch my bill halfway through the month

5:28

or if I switch my plan halfway through the month

5:30

what will my bill look like?

5:32

We want to check that it's

5:34

accurately addressing key questions which are covered by

5:37

our policy

5:39

we need to make sure that it's escalating

5:41

to a human whenever there is

5:44

a billing error

5:47

and finally

5:47

we want to make sure that our model

5:49

isn't withholding any information that it has access to

5:52

which it should be handing over to

5:54

the customer

5:57

So what we're going to do in this

5:58

process is we'll take our prompt

6:01

And we'll run it on our v0

6:05

of, of the eval

6:07

And we'll see what our failure modes are

6:09

and systematically target those failure modes one at

6:12

a time to see if we can resolve

6:14

those failure modes by prompting

6:16

And along the way

6:16

we'll learn a little bit more about the kind of anti-patterns

6:20

and traps to avoid

6:23

And this is representative of how we

6:26

would apply these best prompting techniques in practice

6:29

right?

6:29

We are rarely writing a prompt from scratch

6:32

We're often debugging an existing prompt

6:37

And best practice before we start targeting

6:40

those failure modes specifically

6:42

is to kind of apply our general prompting 101 best practices

6:46

applying general hygiene to

6:49

clean up before we do the eval run

6:52

So

6:52

let's have a little look

6:54

at the example that we're going to be

6:56

using

6:57

So what we're looking at here

6:58

first of all, before we look at the prompt

7:00

is just this vibe-coded web app that I've

7:02

made for the presentation today so that we

7:04

can look at how we're iterating on the

7:06

prompt together

7:08

In this page here

7:10

I can

7:11

easily run my evals on all five

7:14

test cases and inspect the results in a

7:17

little bit more detail

7:19

So before we have a look at the prompt

7:22

I'm just going to run the evals in

7:23

the background

7:27

This is a pretty good

7:28

first pass at a prompt

7:31

When we look at this

7:32

we've defined the bot's role at the top

7:37

When we scroll down

7:39

we've given it some data

7:41

We've given it some information on how to reason over

7:46

the answers that it should be giving to

7:48

the customer

7:49

It's giving some critical instructions around the tone it should use

7:54

how to do calculations

7:56

et cetera. And then finally

7:57

we're passing in our customer account context

8:00

and our user message

8:02

So let's have a look at how our first pass

8:06

the evals did

8:08

So we can see

8:08

as we expect, our control case

8:11

all of our test cases have passed

8:13

This is what we expect for this unambiguous

8:16

test case. But

8:18

it's performing pretty poorly in these other areas

8:23

Now, before we zoom in on those specific failure modes here

8:27

let's do some general cleanup of our prompt

8:36

So as we mentioned

8:37

when we look through this prompt

8:39

there's a couple oddities here already

8:42

So for example

8:44

first one is we're telling the bot that it's

8:47

a human, which just isn't true

8:50

right? We can see as we scroll down

8:51

there's clearly some information here that has been

8:53

copied directly from a website

8:56

So

8:56

the key giveaway here is a reference to a hero image

9:00

and there's even some references to cookies

9:03

at the bottom

9:05

So we need to remove a bit of

9:06

redundant information

9:10

When we look at the instructions here

9:11

they're all grouped into one big paragraph

9:14

So we've got some reasoning here

9:16

we've got instructions about the role

9:20

some critical instructions as well

9:21

without a real way of unpacking

9:25

policy from guidelines, from tone, et cetera

9:29

So

9:32

let me just

9:36

Free entered some changes we want to make to this prompt

9:39

and this is just a diff view of

9:40

some of those changes

9:42

So what we've done is

9:43

first of all, added some structures

9:45

So you can see that we've added

9:47

XML tags here to define the role

9:50

to separate general guidelines

9:53

to separate policy, to separate tone of voice

9:59

et cetera

10:04

So if we run that eval then

10:07

on this new updated prompt

10:09

we should hopefully see an improvement

10:12

in the output as is

10:23

So we can see just by clearing up the prompt

10:26

we've already

10:27

improved the model's performance on this prepaid scenario

10:32

There's an interesting regression there in that fifth hotspot case

10:36

and I don't want to worry too much

10:38

about that now

10:39

There's going to be some natural level of

10:41

variance in the different runs of the eval

10:45

and we'll come back to that case specifically

10:47

to see if we can make the prompt

10:49

consistently better in that area

10:51

So what did we learn from this then?

10:55

simply clearing up the prompt with a better structure

10:59

with a better role description has improved the performance

11:02

and this is a best practice that you

11:04

can return to

11:05

at any stage of writing and maintaining your prompt

11:08

especially as your prompts get more detailed and

11:11

more complex

11:12

A general rule of thumb that I like

11:14

to follow is if you're reading a prompt

11:16

and you can't tell guidelines from policy

11:20

from data

11:20

most likely the model isn't able to either

11:25

So Before looking at some of those cases in more detail

11:28

there's a little bit more general cleanup we can do

11:32

specifically here looking at creating an output contract

11:37

This is a key best practice to

11:40

follow if you're struggling with your output format consistency

11:44

Now, in this case

11:45

we have a customer support bot

11:46

We want it to reply in a conversational tone

11:49

so it's unlikely to be a big issue in this case

11:52

but it's something to bear in mind if you're

11:54

you're dealing with more complex output structures like nested JSONs

11:59

for example

12:03

So, again

12:03

if we go back to the prompts

12:06

and see what fixes we can apply here

12:09

first of all, we've added a section at the end

12:14

where we have to find an output format for the model

12:16

telling it to use

12:19

XML tags to output the response

12:23

But the prompt is

12:25

not always the most effective way of handling

12:28

issues

12:29

We can also change things in the harness

12:31

to ensure consistency to a higher degree

12:35

So what we've added here

12:37

to the API call is a stop sequence

12:40

which is gonna detect

12:42

that closing XML tag

12:44

and tell the model to stop generating a

12:47

response at that point

12:50

Now, when I run the eval here

12:52

I don't necessarily expect to see any clear improvement in performance

12:57

but it's a general best practice that we should be following

13:00

and as I said

13:01

is something that we should remember

13:03

in particular when we have more complex output

13:05

schemas

13:10

One thing to point out here as well

13:12

if you do have a more complex output schema

13:14

something like structured outputs can be incredibly helpful

13:17

to ensure that consistency in a more programmatic way

13:22

Okay, so after the cleanup then

13:24

we can see

13:25

that we now have two test cases which are consistently passing

13:30

but we have three key failure modes

13:32

the proration, the billing error, and the hotspot

13:35

So let's isolate these one by one

13:40

to iterate on the prompt and

13:41

and see the effect of that

13:51

First of all then

13:52

the hotspot question. So the question is

13:55

how much hotspot data is on my unlimited

13:57

plan?

13:58

What we expect the model to do

14:01

is state directly the amount of hotspot data

14:03

that the customer has

14:06

And the reason this is a slightly complex

14:08

case is because the customer test case that

14:11

we're dealing with is on a legacy plan

14:13

So actually the current policy

14:16

doesn't apply to them

14:18

So if we see what's going on

14:20

in the actual test case here

14:22

the customer data which we are feeding

14:26

to the prompt

14:27

includes the amount of hotspot data that customer

14:30

has. They have five gigabytes

14:31

right? But they also have a grandfathered plan

14:35

So what we're seeing

14:38

the model is actually telling the customer is the general

14:44

the unlimited plan includes four gigabytes

14:47

but since you're on a legacy plan

14:49

you should go check this out yourself

14:58

So let's have a look at the prompt then to see

15:02

why the model

15:05

Is deflecting

15:06

this question to

15:08

the customer account URL rather than actually giving

15:10

the information itself

15:14

Now, if we read this prompt

15:15

originally it said

15:16

we changed our plans recently and the policy

15:18

doc shows the current plan data and customers

15:20

on grandfather's plan have different rates

15:23

Never give a customer the wrong plan details

15:26

Instead, point them to the URL

15:29

So it's clear that this instruction

15:33

this latter one, never give customer the wrong information

15:36

is the instruction that the bot has been

15:38

optimizing for

15:40

And

15:41

You might recognize this as

15:43

being very similar to a patch that you

15:46

might have introduced

15:48

in a previous model that you were using to avoid

15:51

where the model was giving

15:53

the customer the wrong information about their plan

15:56

Now, as our models have evolved

15:59

they have gotten much better at instruction following

16:01

So it's likely that instructions like these have

16:03

now become redundant and are actually being overfitted

16:07

to

16:11

So what we're going to tell the model

16:12

instead is give this balanced view

16:17

where it says, you know

16:17

customers on grandfather's plan have different allowances

16:20

but it's captured in the customer information that's given

16:23

and that is the accurate source of truth

16:32

So running the eval here

16:33

we should hopefully be addressing

16:36

all of the test cases for the hotspot case

16:39

Now, I am running this live

16:41

so there could be some variability here

16:43

but we see here that now clearly all

16:45

of our test cases are

16:47

are passing

16:50

So what did we learn from this?

16:51

Well

16:52

we worry a lot about hallucinations

16:55

or the invention of facts and numbers

16:59

but actually the opposite can also happen

17:03

The model can withhold information

17:06

that it actually has access to

17:08

Now

17:08

we saw here that this is likely a

17:10

result of a patch that we introduced for a previous model

17:14

and a best practice that we

17:16

could follow here is actually using version control

17:19

where wherever we are making defensive changes in the prompt

17:24

We are

17:25

tracking the reason why we've introduced these

17:28

Sometimes they're necessary, but in the future

17:31

these kind of changes can produce unwanted effects

17:34

so that we can backtrack on them

17:38

So

17:39

the next failing test case then

17:41

is this proration calculation where a customer asks

17:46

what if I upgrade to the 30 gigabyte

17:48

plan?

17:49

What will my next bill be?

17:51

And what we want the model to do

17:53

is to perform some calculation and return exactly

17:58

what their next bill would be

18:00

rather than giving some sort of vague output

18:03

which is what we can see it's doing

18:05

right now

18:07

if we look at what the model is returning

18:10

It's clearly reasoning through it

18:12

It's doing a little bit of mental maths here and there

18:15

but it's not really giving the customer a concrete answer

18:18

and I wouldn't rely on this as being

18:20

able to accurately give the customer a response

18:28

So

18:30

if we look at the prompt then to

18:31

see how we can fix this

18:38

In the original prompt

18:42

we can see that all the instructions that

18:44

were given to it is telling it

18:46

don't ever give a customer a vague answer

18:50

critical, always calculates any prorated amounts correctly

18:55

Now

18:56

telling the model to do a good job

18:58

isn't particularly helpful

19:00

when we don't give the model the capability

19:03

to actually do a good job

19:05

We want to avoid the model doing mental math

19:07

so what we're going to

19:08

introduce is give the model a tool

19:11

So we're saying in the prompt

19:12

whenever you're doing any calculations

19:14

please use the calculate proration tool

19:17

to do so

19:20

In order to introduce that tool

19:22

we need to introduce it into

19:24

the

19:25

API to tell the model you have access

19:27

to this tool

19:28

We need to define

19:31

the tool schema

19:32

which tells the model what this

19:33

tool does and when to use it

19:36

And then finally, we need to actually implement the tool

19:39

which is the maths behind how it should

19:41

be doing that calculation

19:48

So running that eval then for another pass

19:58

We can see that all the

20:00

test cases are now passing

20:02

It's clearly done

20:04

the

20:04

maths using the tool in the background and

20:07

returning the correct response

20:11

So

20:12

the key lesson to take away here is

20:14

instructions don't add capability

20:17

Telling the model it's critical to do a

20:19

calculation right

20:21

doesn't make it better at mental maths

20:24

So the correct approach was to give it a tool

20:27

overall giving it the ability to

20:29

reason over harder problems

20:31

and using tools to actually execute them reliably

20:37

So now we have

20:38

one final failing test case which we need to address

20:41

which is this billing error here

20:46

In this scenario, there is a billing conflict

20:49

and what we really want

20:51

is the agent to

20:53

escalate this to a human

20:55

And what we're seeing it doing instead

20:58

is it's trying to

21:00

explain to the customer what the reason behind it might be

21:06

and it's trying to

21:07

kind of diagnose the problem itself

21:17

So in order to fix this behavior

21:21

let's again have a look

21:22

what it was told in the prompt

21:28

We see in the initial instructions it was giving

21:30

it says avoid escalating or transferring to a

21:33

care specialist unless absolutely necessary

21:36

as it cost approximately

21:38

eight dollars and it counts against our team's

21:40

fast contract resolution

21:42

Now, this is only giving one side of the story

21:46

right? We're telling it what the cost is to escalating

21:49

but not the benefit

21:50

which

21:51

means it's going to overfit again to

21:54

not escalating this scenario

21:57

And second of all

21:58

we've got this clear conflict between what we've

22:01

defined

22:03

in the eval in terms of what we

22:05

want the model to do

22:06

to do this escalation versus what we're actually

22:08

telling it to do

22:09

And the fix that's relevant here is to

22:12

give it both sides of the story by

22:13

saying it costs eight dollars

22:18

to escalate a case

22:20

but actually if you get this wrong

22:24

Then

22:26

it's going to cost you

22:27

a refund as well as customer trust

22:30

Again, here we observed how the model optimizes for a goal

22:36

and this kind of instruction is a common

22:38

instruction to give. It's

22:40

quite similar to the one we saw earlier

22:42

where we didn't want it to overfit to

22:44

a certain type of behavior

22:46

but

22:47

it's the kind of instruction that can be

22:48

followed quite differently by different generations of models

22:52

and specifically as models become more intelligent

22:56

we need to remember to state both sides of the trade-offs

22:59

because our models are

23:01

becoming better themselves at making those trade-offs themselves

23:12

So

23:14

if we just go back to our eval then and

23:18

run our final test case

23:22

We should see that

23:23

all of our evals are now passing correctly

23:28

So overall, we looked at applying general hygiene principles

23:31

how that can provide an initial uplift to the prompt

23:34

making sure we're removing any redundant instructions

23:38

which were initially intended as patches for previous models behavior

23:43

making sure we're giving it tools to do

23:46

certain tasks reliably

23:50

Now, there's one other scenario that we

23:54

introduced at the start

23:55

which is one that you might also encounter in your work

23:59

which is

24:00

where we're building a new agent from scratch

24:03

And the example that we'll look at here is

24:06

an agent

24:08

whose purpose it is to create a week-long

24:11

retail staff schedule

24:12

based on employee availability and other constraints

24:17

And when we're building a new agent from scratch

24:22

we need to consider not just the prompt

24:25

but also the model that we're using and

24:27

the harness that we're using

24:30

So in this next example

24:32

we're going to compare a number of approaches

24:35

to explore the impact of those three different areas

24:42

So again

24:43

I've just five-coded up this web app so

24:45

that we can walk through this problem

24:48

in this demo

24:49

Here I've just laid out what the problem

24:51

is that we're addressing

24:52

We have our eight employees

24:55

on the right we have this

24:57

schedule that we need to staff with the headcount

25:00

and we have our

25:02

constraints that must be satisfied in every scenario

25:08

Now, because we have these hard rules

25:11

rather than using

25:12

an

25:13

LLM judge like we did in the previous

25:15

case to do the grading

25:16

we can actually use a

25:19

just a Python function which programmatically checks

25:23

for every schedule that's generated how many violations

25:26

were made

25:32

So to begin with

25:34

in a, we want to start simple

25:36

We're going to use a simple prompt

25:38

we're going to use the bare bones that

25:40

we think we'll need with a model

25:41

sonnet four six to see

25:43

how it performs and how we're going to

25:45

hill climb against that

25:47

so here is our baseline prompt

25:49

We've already applied some of that

25:51

general hygiene and those best practices that we

25:54

saw earlier on using XML tags to structure

25:57

the prompt

25:58

We've given it an output format as well

26:00

now that we're giving a schedule

26:02

we're asking it to

26:05

output a JSON which If we don't

26:08

give that output structure might lead to passing errors

26:12

downstream

26:19

When we run the

26:22

simple model on a first iteration of the evals

26:26

all cases fail. Now

26:29

just what we're looking at here is in our test set

26:32

we're essentially repeating

26:36

we're doing five trials here

26:38

and these numbers are showing how many violations

26:41

were made in each trial

26:46

In the outputs

26:47

we can see that it's made a decent

26:49

attempt at reasoning through the problem

26:54

but it's burning a lot of tokens

26:57

and it's clearly not checking its work

27:01

as it's not getting to the right impact

27:04

So Let's try a larger model

27:08

a model which we know is better at

27:09

reasoning. So we're gonna run it through Opus 4.7 instead

27:18

keeping everything else the same

27:20

Now, interestingly, whilst all test cases are still failing

27:26

you can see that the

27:27

overall number of violations that Opus has made

27:31

has reduced significantly from Sonnet 4.6

27:35

So we're possibly onto something here

27:39

right?

27:40

This isn't good enough to ship because it's still failing

27:43

but clearly giving it more

27:45

reasoning capability is helping drive it towards a

27:48

better result

27:50

So

27:50

what we're gonna try next is

27:52

using Opus with adaptive thinking instead

27:57

So it can decide for itself

28:01

How much thinking it needs

28:02

how much reasoning it needs to use

28:05

to solve this issue. So no change to the prompt

28:09

really just a change to the API here

28:17

So this now seems to reliably generate compliance schedules

28:23

but it requires a lot more tokens

28:27

We're tripling essentially the number of tokens that we're using here

28:31

and we're tripling the latency

28:33

So we want to

28:34

try and see if we can optimize that

28:36

cost latency trade-off a little bit more

28:40

This is latency 100 seconds

28:42

Obviously

28:42

I'm running this one async

28:45

for the purposes of time

28:46

Opus 4.7 hasn't magically gotten much faster since

28:49

the last time you used it

28:55

So

28:57

Let's see if we can optimize a little

28:59

bit more for the token latency trade-off

29:03

What we haven't tried yet is using Sonnet 4.6

29:06

so a smaller model

29:07

but with a better prompt

29:08

We looked a lot at the prompt optimization

29:11

in that last section. So

29:13

I've added a couple details to the prompt here

29:16

We're told, we're, in particular, how to, reason through this problem

29:23

and most critically

29:25

telling it to

29:26

check its work before outputting it

29:34

So when I ran that EFL

29:37

we see that it passes in two

29:39

out of the five cases

29:42

Now

29:42

the failure modes that we're seeing

29:44

is actually not violations of The scheduling requirements

29:51

but the model hasn't been able

29:54

to finish the tasks within the output

29:58

limit that we set

30:00

So

30:02

whilst

30:03

we could

30:03

increase the max tokens that this model is

30:06

able to use to get all five test cases passing

30:11

we see here that we're using even more tokens

30:14

and this run has an even higher latency

30:16

So this is probably not the route that

30:18

we want to go down

30:20

Now, as a final pass then

30:23

we want to

30:24

look at

30:25

doing this a little bit more agentically

30:29

So we're going to use this generate

30:31

evaluate, repair loop

30:33

where

30:34

essentially the generator now creates a first draft

30:38

of the schedule

30:40

And then we have a separate prompt

30:43

Which reports any specific violations that it made

30:46

So not programmatically checking it

30:48

but checking it with an LLM

30:51

So we're checking for every rule

30:53

and we're providing evidence of every violation

30:57

And we then have a third repair prompt

31:01

which receives

31:04

any violations that were made

31:06

and tries to make targeted fixes to it

31:10

So we have three very simple prompts

31:13

but they're now running independently

31:15

rather than trying to do everything in

31:18

one large prompt

31:24

So we can see in this case

31:27

our agentic approach has solved all of our test cases

31:34

with

31:35

a much

31:36

lower number of tokens and with a lower

31:38

latency than trying Sonnet 4.6

31:42

With about a prompt

31:43

So going forward

31:44

it seems like there's two

31:46

appropriate approaches to take here

31:49

Using Opus 4.7 with adaptive thinking

31:52

or using this agentic loop

31:54

Now, moving forward

31:56

we'd probably want to do

31:57

a little bit more optimization

31:59

on this loop to try and get it

32:01

to be more efficient

32:03

But there's one key benefit as well from using this generate

32:06

evaluate, repair loop

32:08

And

32:08

that is that you can put in

32:11

soft requirements at runtime

32:14

So in the evaluation prompt

32:16

we can say, Harry doesn't like working with Sally

32:18

so as much as possible

32:20

try and separate them from working together

32:23

Or we need a third shift

32:25

on Wednesday

32:26

for example

32:27

So it means that you're not having to

32:29

make changes to the

32:31

Python function

32:32

which is doing the evaluation in the backend

32:34

every time to satisfy for any soft constraints

32:38

which might

32:39

depend just on a case by case

32:41

Basis

32:45

So to wrap up then

32:49

pulling all of those learnings together

32:51

what did we see? Well

32:52

we looked at two scenarios

32:53

two scenarios which

32:55

I as an engineer see most in my day-to-day

32:58

which is where we're maintaining a prompt

33:00

we're migrating to a new model

33:02

which has some different behaviors

33:05

and building a new use case from scratch

33:08

We saw that general hygiene principles

33:12

following those can immediately uplift

33:16

the performance against a set of evals

33:19

and that we need those evals

33:21

to be able to rigorously see

33:23

any impacts of changing our prompts on the

33:26

output

33:27

Then we saw this process of targeting failure

33:30

modes one by one

33:32

adding structure, avoiding long ban lists

33:35

et cetera

33:36

were all things

33:37

that helped push our model to the correct

33:40

behavior

33:41

And then finally

33:42

with our new agentic bot that we were building

33:47

we saw the impact of splitting

33:51

into three separate prompt systems

33:53

So rather than using one prompt to address everything

33:58

we're actually isolating different tasks where it's

34:01

easy and repeatable to separate out the steps

34:04

that it needs to take every time

34:07

Thank you so much for attending this afternoon

34:10

I hope you have a fantastic rest