One of my friends makes $1.2M a year as an Anthropic engineer. I asked him how he became so good at

Transcript

Last session today of Code with Claude

0:02

I hope you've all had a fantastic day

0:04

so far

0:05

My name is Margot van Laer

0:06

I am an applied AI engineer at Anthropic here in London

0:11

and

0:11

this afternoon we're going to be talking about

0:14

the prompting playbook

0:15

And prompting is arguably one of the first skills

0:19

if not the first skill

0:21

that we had to learn as engineers when

0:23

we first started to work with LLMs

0:26

and

0:26

even now it continues to be one of the most critical

0:30

skills to building effective AI systems

0:34

So today we're going to discuss some best practices

0:39

in the context of

0:41

Two practical scenarios that you're probably encountering

0:44

at work

0:45

The first is where you have an existing

0:47

prompt in production that you've been maintaining for some time

0:53

and possibly you're migrating it to a new

0:55

model or making a change to the architecture

0:58

and for some reason

0:59

it's no longer working as well

1:01

The second scenario is where we're building an

1:03

entirely new agentic use case from the ground up

1:06

and we need to build the prompt from

1:08

zero to one

1:10

Now, in order to illustrate these best practices

1:14

I don't just want to give you a

1:16

list of do's and don'ts

1:18

I want to walk through a practical

1:19

example that's been inspired by real prompts that

1:24

I've seen some of our customers work with

1:26

who are building on Claude

1:29

So

1:30

the prompt that we'll look at today

1:32

is a miniaturized example

1:35

The prompts that you're working with are probably

1:37

a lot longer

1:38

and more complex than the one we'll see

1:40

today

1:41

but it's representative of some common problems that

1:45

you might encounter when maintaining a prompt

1:48

So

1:48

imagine that we have a prompt that multiple

1:51

people have been collaborating on

1:54

contributing to

1:55

There's no clear owner

1:57

It covers a lot of different areas like policy

2:00

like tone, processes

2:03

and we have some patches

2:05

for kind of previous models that we've migrated to

2:08

all mixed together

2:10

it's built up and it's complex

2:13

And when we're migrating to a new model

2:16

we're finding that suddenly

2:18

a lot of our

2:19

test cases are no longer working as well

2:21

as we expected

2:22

So what's actually going on here?

2:25

Well, in order to start unpacking that question

2:28

we need a starting point

2:29

and that starting point is evaluations

2:33

We need evaluations to provide that rigor

2:37

to understand

2:38

whether a change to our prompt is actually

2:40

correlating to an improvement in its performance

2:45

And we have different models

2:47

which have different capabilities and different behaviors

2:51

And when you migrate to a different model

2:54

it could be that

2:56

your system is no longer working as well

2:57

for two reasons

2:59

First of all, if, the new model might be capable

3:03

but it's behaving differently

3:05

and therefore we can tune our prompting to

3:07

fix that behavior

3:09

The second case is where actually the model

3:12

that we're changing to isn't as capable

3:14

and no amount of prompting is going to

3:16

fix that

3:17

So

3:18

we need to have an eval suite to

3:20

act as a

3:21

way of testing that regression

3:24

so that we can apply our prompting best

3:26

practices to that

3:30

So

3:30

in the example that we're going to be looking at today

3:33

as I said

3:34

it's going to be a miniaturized example

3:36

We'll have five test cases in our eval

3:40

In reality

3:41

you'll have a lot more test cases in your eval suite

3:44

but

3:45

the key thing here is that it's representative

3:47

of three key cases that we need to

3:49

cover

3:51

Those three key cases include having a control case

3:54

which is a case which should always pass

3:57

It's something that

3:58

we know the model handles well

4:00

It's unambiguous

4:02

The second is Edge cases

4:05

and these are cases where we've seen the model fail before

4:09

and by including instructions into the prompt

4:12

we're making sure that same behavior doesn't slip

4:15

through again in the future

4:18

And finally, and critically

4:20

we need to make sure that the model

4:21

has a good understanding of the extent of its capabilities

4:26

where it should be handing off to a human

4:29

or where actually maybe it should be point

4:31

blank refusing to answer a request

4:35

So

4:36

in the example that we're going to be looking at today

4:39

we'll be using a

4:42

prompt

4:43

for

4:43

a customer support bot for a telco company called Meridian Mobile

4:48

and these are the five test cases that

4:50

we are going to be looking at today

4:53

We have a simple control case looking at

4:57

you know

4:57

what's the data limit in the basic plan

5:00

we're also looking at Edge cases

5:03

such as its ability to do calculations

5:06

such as calculating proration bills

5:09

If I switch my bill halfway through the month

5:12

or if I switch my plan halfway through the month

5:14

what will my bill look like?

5:16

We want to check that it's

5:18

accurately addressing key questions which are covered by

5:21

our policy

5:23

we need to make sure that it's escalating

5:25

to a human whenever there is

5:28

a billing error

5:30

and finally

5:31

we want to make sure that our model

5:33

isn't withholding any information that it has access to

5:36

which it should be handing over to

5:38

the customer

5:40

So what we're going to do in this

5:42

process is we'll take our prompt

5:45

and we'll run it on our v0

5:49

Of, of the eval

5:50

and we'll see what our failure modes are

5:52

and systematically target those failure modes one at

5:55

a time to see if we can resolve

5:57

those failure modes by prompting

5:59

And along the way

6:00

we'll learn a little bit more about the kind of anti-patterns

6:03

and traps to avoid

6:06

And this is representative of how we

6:08

would apply these best prompting techniques in practice

6:12

right?

6:12

We are rarely writing a prompt from scratch

6:15

We're often debugging an existing prompt

6:19

And best practice before we start targeting

6:22

those failure modes specifically

6:24

is to kind of apply our general

6:26

prompting 101 best practices of buying general hygiene

6:30

to

6:31

clean up before we do the eval run

6:34

So

6:35

let's have a little look

6:36

at the example that we're going to be

6:38

using

6:39

So what we're looking at here

6:40

first of all, before we look at the prompt

6:42

is just this vibe coded web app that

6:44

I've made for the presentation today so that

6:46

we can look at how we're iterating on

6:48

the prompt together

6:50

in this page here

6:52

I can

6:53

easily run

6:54

my evals on all five

6:56

test cases and inspect the results in a

6:59

little bit more detail

7:01

So before we have a look at the prompt

7:03

I'm just going to run the evals in

7:05

the background

7:08

This is a pretty good

7:10

fast pass at a prompt

7:12

When we look at this

7:13

we've defined the bot's role at the top

7:18

When we scroll down

7:20

we've given it some data

7:22

We've given it some information on how to

7:25

reason over

7:27

the answers that it should be giving to

7:29

the customer

7:30

It's giving some critical instructions around the tone it should use

7:35

how to do calculations

7:36

et cetera. And then finally

7:38

we're passing in our customer account context

7:41

and our user message

7:43

So let's have a look

7:44

at how our

7:46

first pass at the evals did

7:48

So we can see

7:49

as we expect, our control case

7:52

all of our test cases have passed

7:54

This is what we expect for this unambiguous

7:56

test case. But

7:58

it's performing pretty poorly in these other areas

8:03

Now, before we zoom in on those specific failure modes here

8:07

let's do some general cleanup of our prompt

8:16

So as we mentioned

8:17

when we look through this prompt

8:19

there's a couple oddities here already

8:22

So for example

8:24

first one is we're telling the bot that it's

8:27

a human, which just isn't true

8:29

right? We can see as we scroll down

8:31

there's clearly some information here that has been

8:33

copied directly from a website

8:35

So

8:36

the key giveaway here is a reference to a hero image

8:39

and there's even some references to cookies

8:42

at the bottom

8:44

So we need to remove a bit of

8:45

redundant information

8:49

When we look at the instructions here

8:50

they're all grouped into one big paragraph

8:53

So we've got some reasoning here

8:55

we've got instructions about the role

8:58

some critical instructions as well

9:00

without a real way of unpacking

9:03

policy from guidelines, from tone, et cetera

9:07

So... Let me just

9:14

Free and subtle changes we want to make to this prompt

9:17

and this is just a diff view of

9:18

some of those changes

9:20

So what we've done is

9:21

first of all, added some structure

9:23

So you can see that we've added

9:25

XML tags here to define the role

9:28

to separate general guidelines

9:31

to separate policy, to separate tone of voice

9:37

et cetera

9:41

So if we run that eval then

9:44

on this new updated prompt

9:47

we should hopefully see an improvement

9:49

in the output as is

10:00

So we can see just by clearing up the prompt

10:03

we've already

10:04

improved the model's performance on this prepaid scenario

10:09

There's an interesting regression there in that fifth hotspot case

10:13

and I don't want to worry too much

10:14

about that now

10:16

There's going to be some natural level of

10:18

variance in the different runs of the eval

10:22

and we'll come back to that case specifically

10:24

to see if we can make the prompt

10:25

consistently better in that area

10:28

So what did we learn from this then?

10:32

simply clearing up the prompt with a better structure

10:35

with a better role description has improved the performance

10:38

and this is a best practice that you

10:40

can return to

10:41

at any stage of writing and maintaining your prompt

10:44

especially as your prompts get more detailed and

10:47

more complex

10:48

A general rule of thumb that I like

10:50

to follow is if you're reading a prompt

10:52

and you can't tell guidelines from policy from data

10:56

most likely the model isn't able to either

11:01

So Before looking at some of those cases in more detail

11:04

there's a little bit more general cleanup we can do

11:08

specifically here looking at creating an output contract

11:13

This is a key best practice to

11:15

follow if you're struggling with your output format consistency

11:19

Now, in this case

11:20

we have a customer support bot

11:22

We want it to reply in a conversational tone

11:24

so it's unlikely to be a big issue in this case

11:27

but it's something to bear in mind if you're

11:29

you're dealing with more complex output structures like nested JSONs

11:34

for example

11:38

So again

11:38

if we go back to the prompts

11:40

and see what fixes we can apply here

11:44

first of all, We've added a section

11:48

at the end where we have to find

11:49

an output format for the model

11:51

telling it to use

11:54

XML tags to output the response

11:57

But the prompt is

11:59

not always the most effective way of handling

12:02

issues

12:03

We can also change things in the harness

12:06

to ensure consistency to a higher degree

12:09

So what we've added here

12:11

to the API call is a stop sequence

12:14

which is gonna detect

12:16

that closing XML tag

12:18

and tell the model to stop generating a

12:21

response at that point

12:23

Now, when I run the eval here

12:25

I don't necessarily expect to see any clear improvement in performance

12:31

but it's a general best practice that we should be following

12:33

and as I said

12:34

is something

12:35

that we should remember in particular when we

12:37

have more complex output schemas

12:43

One thing to point out here as well

12:45

if you do have a more complex output schema

12:47

something like structured outputs can be incredibly helpful

12:50

to ensure that consistency in a more programmatic way

12:55

Okay, so after the cleanup

12:57

then we can see

12:58

that we now have two test cases which are consistently passing

13:03

but we have three key failure modes

13:05

the proration, the billing error, and the hotspot

13:07

So let's isolate these one by one

13:13

to iterate on the prompt and

13:14

and see the effect of that

13:24

First of all, then, the hotspot question

13:26

So the question is

13:27

how much hotspot data is on my unlimited

13:29

plan?

13:31

What we expect the model to do

13:33

is state directly the amount of hotspot data

13:36

that the customer has

13:38

And the reason this is a slightly complex

13:40

case is because the customer test case that

13:43

we're dealing with is on a legacy plan

13:45

So actually the current policy

13:48

doesn't apply to them

13:49

So if we see what's going on

13:51

in the actual test case here

13:54

the customer data which we're feeding

13:57

to the prompt

13:58

includes the amount of hotspot data that customer

14:01

has. They have five gigabytes

14:03

right? But they also have a grandfathered plan

14:06

So what we're seeing

14:09

the model is actually telling the customer is the general

14:15

the unlimited plan includes four gigabytes

14:18

but since you're on a legacy plan

14:20

you should go check this out yourself

14:29

So let's have a look at the prompt then to see

14:33

why the model

14:35

Is deflecting

14:37

this question to

14:38

the customer account URL rather than actually giving

14:41

the information itself

14:44

Now, if we read this prompt

14:45

originally it said, we changed our plans recently

14:48

and the policy doc shows the current plan

14:50

data and customers on grandfather's plan have different

14:52

rates

14:53

Never give a customer the wrong plan details

14:56

Instead, point them to the URL

14:59

So it's clear that this instruction

15:03

this latter one, never give customer the wrong information

15:06

is the instruction that the bot has been

15:08

optimizing for

15:10

And

15:11

You might recognize this as

15:13

being very similar to a patch that you

15:16

might have introduced

15:17

in a previous model that you were using to avoid

15:21

where the model was giving

15:23

the customer the wrong information about their plan

15:25

Now, as our models have evolved

15:28

they've gotten much better at instruction following

15:30

So it's likely that instructions like these have

15:33

now become redundant and are actually being overfitted

15:36

to

15:40

So what we're going to tell the model

15:41

instead is give this balanced view

15:46

where it says, you know

15:46

customers on grandfather's plan have different allowances

15:49

but it's captured in the customer information that's given

15:52

and that is the accurate source of truth

16:00

So running the eval here

16:02

we should hopefully be addressing

16:05

all of the test cases for the Hotspot case

16:08

Now, I am running this live

16:09

so there could be some variability here

16:11

but we see here that now clearly all

16:13

of our test cases are

16:15

are passing

16:18

So what did we learn from this?

16:19

Well

16:20

we worry a lot about hallucinations

16:23

or the invention of facts and numbers

16:27

but actually the opposite can also happen

16:31

The model can withhold information

16:33

that it actually has access to

16:36

Now

16:36

we saw here that this is likely a

16:38

result of a patch that we introduced for a previous model

16:41

and a best practice that we

16:43

could follow here is actually using version control

16:46

where wherever we are making defensive changes in the prompt

16:52

we are

16:52

tracking the reason why we've introduced these

16:55

Sometimes they're necessary, but in the future

16:58

these kind of changes can produce unwanted effects

17:02

so that we can backtrack on them

17:05

So

17:06

the next failing test case then

17:08

is this proration calculation where a customer asks

17:13

what if I upgrade to the thirty gigabyte

17:15

plan?

17:16

What will my next bill be?

17:18

And what we want the model to do

17:20

is to perform some calculation and return exactly

17:24

what their next bill would be

17:26

rather than giving some sort of vague output

17:29

which is what we can see it's doing

17:32

right now

17:33

if we look at what the model is returning

17:36

it's clearly reasoning through it

17:39

it's doing a little bit of mental math here and there

17:41

but it's not really giving the customer a concrete answer

17:44

and I wouldn't rely on this as being

17:46

able to accurately give the customer a response

17:54

So

17:56

if we look at the prompt then to

17:57

see how we can fix this

18:03

In the original prompt

18:07

we can see that all the instructions that

18:09

were given to it is telling it

18:12

don't ever give a customer a vague answer

18:15

critical, always calculate any prorated amounts correctly

18:20

Now

18:21

telling the model to do a good job

18:23

isn't particularly helpful

18:25

when we don't give the model the capability

18:28

to actually do a good job

18:30

We want to avoid the model doing mental math

18:32

so what we're gonna

18:33

introduce is give the model a tool

18:35

So we're saying in the prompt

18:37

whenever you're doing any calculations

18:39

please use the calculate proration tool

18:42

to do so

18:45

In order to introduce that tool

18:46

we need to introduce it into

18:49

the

18:49

API to tell the model you have access

18:51

to this tool

18:53

We need to define

18:55

the tool schema

18:56

which tells the model what this

18:58

tool Does and when to use it

19:00

and then finally we need to actually implement the tool

19:03

which is the maths behind how it should

19:05

be doing that calculation

19:12

So running that eval then for another pass

19:22

We can see that all the

19:23

test cases are now passing

19:25

It's clearly done

19:28

the

19:28

maths using the tool in the background and

19:30

returning the correct response

19:35

So

19:35

the key lesson to take away here is

19:37

instructions don't add capability

19:40

Telling the model it's critical to do a

19:42

calculation right

19:44

doesn't make it better at mental maths

19:47

So the correct approach was to give it a tool

19:50

overall giving it the ability to

19:52

reason over harder problems and using tools

19:56

Actually execute them reliably

20:00

So now we have one final failing test case

20:03

which we need to address

20:04

which is this billing error here

20:09

In this scenario, there is a billing conflict

20:12

and what we really want

20:13

is the agent to

20:16

escalate this to a human

20:18

And what we're seeing it doing instead

20:20

is it's trying to

20:23

explain to the customer what the reason behind it might be

20:28

and it's trying to

20:29

kind of diagnose the problem itself

20:39

So in order to fix this behavior

20:43

let's again have a look

20:44

what it was told in the prompt

20:49

We see in the initial instructions it was giving

20:52

it says

20:52

avoid escalating or transferring to a care specialist unless absolutely necessary

20:57

as it costs approximately

20:59

eight dollars and it counts against our team's

21:01

fast contract resolution

21:04

Now, this is only giving one side of the story

21:07

right? We're telling it what the cost is to escalating

21:10

but not the benefit

21:11

which

21:12

means it's going to overfit again to

21:15

not escalating this scenario

21:18

And second of all

21:19

we've got this clear conflict between what we've

21:21

defined

21:24

in the eval in terms of what we

21:25

want the model to do

21:26

to do this escalation versus what we're actually

21:29

telling it to do

21:30

And the fix that's relevant here is to

21:32

give it both sides of the story by

21:34

saying it costs eight dollars

21:39

to escalate a case

21:40

but actually if you get this wrong

21:45

Then

21:46

it's going to cost you

21:47

a refund as well as customer trust

21:51

Again, here we observed how the model optimizes for a goal

21:56

and this kind of instruction is a common

21:58

instruction to give. It's

22:00

quite similar to the one we saw earlier

22:02

where we didn't want it to overfit to

22:04

a certain type of behavior

22:06

but

22:06

it's the kind of instruction that can be

22:08

followed quite differently by different generations of models

22:12

And specifically, as models become more intelligent

22:16

we need to remember to state both sides of the trade-offs

22:19

because our models are

22:20

becoming better themselves at making those trade-offs themselves

22:31

So

22:33

if we just go back to our eval then and

22:37

run our final test case

22:41

We should see that

22:42

all of our evals are now passing correctly

22:47

So overall, we looked at applying general hygiene principles

22:50

how that can provide an initial uplift to the prompt

22:53

making sure we're removing any redundant instructions

22:56

which were initially intended as patches for previous models' behavior

23:02

making sure we're giving it tools to do

23:04

certain tasks reliably

23:09

Now, there's one other scenario that we

23:12

introduced at the start

23:13

which is one that you might also encounter in your work

23:17

which is

23:18

where we're building a new agent from scratch

23:21

And the example that we'll look at here is

23:24

an agent

23:26

whose purpose it is to create a week-long

23:29

retail staff schedule

23:30

based on employee availability and other constraints

23:34

And when we're building a new agent from scratch

23:40

We need to consider not just the prompt

23:43

but also the model that we're using and

23:45

the harness that we're using

23:47

So in this next example

23:49

we're going to compare a number of approaches

23:52

to explore the impact of those three different areas

23:59

So again

24:00

I've just five-coded up this web app so

24:02

that we can walk through this problem

24:05

in this demo

24:06

Here I've just laid out what the problem

24:08

is that we're addressing

24:09

We have our eight employees

24:12

on the right

24:12

we have this

24:14

schedule that we need to staff with the headcount

24:17

and we have our

24:18

constraints that must be satisfied in every scenario

24:25

Now, because we have these hard rules

24:27

rather than using

24:29

an

24:29

LLM judge like we did in the previous

24:31

case to do the grading

24:33

we can actually use a

24:35

just a Python function which programmatically checks

24:39

for every schedule that's generated how many violations

24:42

were made

24:48

So to begin with

24:50

in a, we want to start simple

24:52

We're going to use a simple prompt

24:54

we're going to use the bare bones that

24:55

we think we'll need with a model

24:57

Sonnet 4.6

24:58

to see

24:59

how it performs and how we're going to

25:00

hill climb against that

25:03

so here is our baseline prompt

25:05

We've already applied some of that

25:07

general hygiene and those best practices that we

25:10

saw earlier on using XML tags to structure

25:12

the prompt

25:13

We've given it an output format as well

25:15

now that we're giving a schedule

25:18

we're asking it to output a JSON

25:21

which

25:22

if we don't

25:23

Give that output structure might lead to passing errors

25:27

downstream

25:34

When we run the

25:37

simple model on a first iteration of the evals

25:40

all cases fail. Now

25:43

just what we're looking at here is in our test set

25:46

we're essentially repeating

25:50

we're doing five trials here

25:52

and these numbers are showing how many violations

25:55

were made in each trial

26:00

In the outputs

26:01

we can see that it's made a decent

26:03

attempt at reasoning through the problem

26:08

but it's burning a lot of tokens

26:11

and it's clearly not checking its work

26:15

as it's not getting to the right impact

26:18

So let's try a larger model

26:22

a model which we know is better at

26:23

reasoning. So we're gonna run it through

26:27

Opus 4.7 instead

26:31

keeping everything else the same

26:34

Now, interestingly, whilst all test cases are still failing

26:39

you can see that the

26:40

overall number of violations that Opus has made

26:44

has reduced significantly from Sonnet 4.6

26:48

So we're possibly onto something here

26:52

right?

26:53

This isn't good enough just yet because it's still failing

26:56

but clearly giving it more

26:58

reasoning capability is helping drive it towards a

27:01

better result

27:02

So

27:03

what we're gonna try next is

27:05

using Opus with adaptive thinking instead

27:09

So it can decide for itself

27:13

How much thinking it needs

27:15

how much reasoning it needs to use

27:17

to solve this issue. So no change to the prompt

27:21

really just a change to the API here

27:29

So this now seems to reliably generate compliance schedules

27:34

but it requires a lot more tokens

27:39

We're tripling essentially the number of tokens that we're using here

27:42

and we're tripling the latency

27:45

So we want to

27:46

try and see if we can optimize that

27:48

cost latency trade-off a little bit more

27:51

this is latency a hundred seconds

27:53

obviously. I'm running this one async

27:56

for the purposes of time

27:58

Opus four seven hasn't magically gotten much faster

28:00

since the last time

28:02

you used it

28:06

so

28:08

Let's see if we can optimize a little

28:10

bit more for the token latency trade-off

28:14

What we haven't tried yet is using SONNET 46

28:17

so a smaller model

28:18

but with a better prompt

28:19

We looked a lot at prompt optimization

28:22

in that last section. So

28:24

I've added a couple details to the prompt here

28:27

We're told, we're in particular how to

28:31

reason through this problem

28:34

and most critically

28:35

telling it to check its work

28:38

before outputting it

28:45

So when I ran that eval

28:47

we see that it passes in two

28:50

out of the five cases

28:52

Now

28:52

the failure modes that we're seeing

28:54

is actually not violations of The scheduling requirements

29:01

but the model hasn't been able

29:04

to finish the tasks within the output

29:08

limit that we set

29:10

So

29:11

whilst

29:12

we could

29:13

increase the max tokens that this model is

29:15

able to use to get all five test cases passing

29:20

we see here that we're using even more tokens

29:23

and this run has an even higher latency

29:25

So this is probably not the route that

29:27

we want to go down

29:29

Now, as a final pass then

29:32

we want to

29:33

look at

29:34

doing this a little bit more agentically

29:38

So we're going to use this generate

29:40

evaluate, repair loop

29:42

where

29:43

essentially the generator now creates a

29:46

first draft of the schedule

29:48

And then we have a separate prompt

29:51

Which reports any specific violations that it makes

29:55

So not programmatically checking it

29:57

but checking it with an LLM

29:59

So we're checking for every rule

30:01

and we're providing evidence of every violation

30:05

And we then have a third repair prompt

30:09

which receives

30:12

any violations that were made

30:14

and tries to make targeted fixes to it

30:18

So we have three very simple prompts

30:21

but they're now running independently

30:23

rather than trying to do everything in

30:26

one large prompt

30:32

So we can see in this case

30:34

our agentic approach has solved all of our test cases

30:41

with

30:42

a much

30:43

lower number of tokens and with a lower

30:45

latency than trying Sonnet 4.6

30:49

with a better prompt

30:51

So going forward

30:51

it seems like there's two appropriate approaches to take here

30:56

using Opus four seven with adaptive thinking

30:59

or using this agentic loop

31:01

Now, moving forward

31:03

we'd probably want to do a little bit

31:05

more optimization

31:06

on this loop to try and get it

31:08

to be more efficient

31:10

But there's one key benefit as well from using this generate

31:13

evaluate, repair loop

31:15

and

31:15

that is that you can put in

31:18

soft requirements at runtime

31:20

So in the evaluation prompt

31:22

we can say, Harry doesn't like working with Sally

31:25

so as much as possible

31:26

try and separate them from working together

31:29

Or we need a third shift

31:31

on Wednesday

31:32

for example

31:33

So it means that you're not having to

31:35

make changes to the

31:37

Python function

31:38

which is doing the evaluation in the backend

31:40

every time to satisfy for any soft constraints

31:44

which might depend just on a case-by-case basis

31:51

So to wrap up then

31:54

pulling all of those learnings together

31:56

what did we see? Well

31:57

we looked at two scenarios

31:59

two scenarios which

32:00

I as an engineer see most in my day-to-day

32:04

which is where we're maintaining a prompt

32:06

we're migrating to a new model

32:07

which has some different behaviors

32:10

and building a new use case from scratch

32:14

We saw that

32:15

general hygiene principles following those can immediately uplift the performance against

32:23

a set of evals

32:24

and that we need those evals

32:26

to be able to rigorously see

32:28

any impacts of changing our prompt on the

32:31

output

32:32

Then we saw this process of targeting failure

32:34

modes one by one

32:37

adding structure, avoiding long ban lists

32:40

et cetera

32:40

were all things

32:42

that helped push our model to the correct

32:44

behavior

32:45

And then finally

32:47

with Our new agentic bot that we were building

32:51

we saw the impact of splitting

32:55

into three separate prompt systems

32:57

So rather than using one prompt to address everything

33:02

we're actually isolating different tasks where it's

33:05

easy and repeatable to separate out the steps

33:08

that it needs to take every time

33:11

Thank you so much for attending this afternoon

33:14

I hope you have a fantastic rest of

33:16

your day. Tell the situation and be like

33:18

hey, can you fix this for me?

33:20

It's incredible

33:22

Now, let's talk about best practices

33:26

And the first one is not going to be a surprise

33:29

but the first one is use cloud.md files

33:33

So remember that Claude Code

33:34

like I said, is an agent, and it has some tools

33:38

has some lightweight instructions in the prompt

33:40

but it doesn't really have memory

33:42

And so the main way we share state

33:45

across kind of sessions

33:47

or across our team when we fire up

33:48

Claude Code in the same code base over

33:51

and over again is this Claude.md file

33:53

So when we start Claude

33:55

what happens is if there's this Claude.md file

33:58

in the working directory

33:59

it's just plopped into context

34:01

it's plopped into the prompt

34:02

and basically what it says is

34:04

hey, Claude, by the way

34:04

these are

34:05

important instructions the developer left for you

34:08

Be sure to pay close attention to this

34:11

And there's various places you can put the

34:12

Claude.md file. You can put it in a

34:14

project

34:15

and check it in so all your teammates

34:16

share it

34:17

You can put one in your home directory

34:19

if there's things you just want Claude to always know about

34:21

regardless of what you're working on

34:23

And the things you put in here are things like

34:25

hey, by the way

34:27

maybe this is how you run the unit

34:28

tests

34:29

Or just so you know

34:30

to make kind of your searching and life easier

34:33

here's like

34:33

just like an overview of kind of how

34:35

this project is laid out

34:37

where the tests live

34:38

what different modules are

34:39

things like that

34:40

Or here's our style guide

34:42

All sorts of things like that to just

34:43

make Claude's life a bit easier

34:46

And you can build these things up over

34:47

time

34:50

The other thing you can do

34:51

which is important, is permission management

34:54

When you're running Claude Code

34:55

there's all sorts of different kind of permission

34:57

things flying by

34:59

Kind of out of the box

35:00

what happens when you start our tool is for read actions

35:03

if Claude is searching or reading

35:06

we just let it go

35:07

But once it starts writing

35:09

or running bash commands or doing things that could change

35:13

change stuff on your machine potentially

35:15

that's when we kick in this UI and

35:17

it says something like

35:18

yes, yes, always allow this, or no

35:22

I want to do something else

35:24

And using that permission management and being smart

35:27

about it can help you work faster

35:29

So there's something called auto accept mode

35:31

where if you're working with cloud code and

35:33

you press shift tab

35:35

cloud will just start working

35:37

There's things you can do

35:38

like you can configure cloud in the settings where specific commands

35:42

like on bash

35:43

like if you just are like tired of saying yes

35:46

run npm run test

35:48

you can just always approve that

35:49

So fiddling with your permission management

35:52

is a great way

35:53

to kind of speed up your workflow

35:55

Integration setup

35:57

So one thing that

35:58

is going to help you get the most

35:59

out of Cloud Code is remember

36:01

that it's great at the terminal

36:03

And if there's applications that you use

36:05

which have kind of a way to access them through CLI

36:08

and GitHub is a great example of that

36:10

they have a powerful tool called GH

36:12

you can

36:13

basically give more work to Cloud Code

36:16

And you can do that

36:17

either by just installing more CLI tools

36:20

or you can attach more MCP servers

36:23

I would say, just through experience

36:26

that if you're using something like

36:28

a CLI tool that's well known and well documented

36:31

and you're trying to choose between the CLI

36:33

tool and just installing it on your machine

36:36

and grabbing an MCP server

36:38

I would recommend using the CLI tool

36:41

and then also

36:42

if you internally have your own tools at Anthropic

36:46

we have something called Coup that does a

36:47

whole bunch of stuff for us

36:49

you can also tell Cloud about that

36:51

and you prob- that's the sort of thing

36:52

you'd put in Cloud to MD

36:55

And then context management

36:57

So remember that Claude is an agent

37:00

and when it's an

37:01

what it does, it calls these tools

37:03

and the context builds up and up over time

37:06

and at least for Anthropic

37:07

our models have a context window of 200,000 tokens

37:10

and you can max this thing out

37:12

So you kind of have two options when

37:14

you're in a long session with Claude

37:16

and you're working

37:16

and you're going back and forth

37:18

You'll see in the bottom right

37:19

you'll start to get this little warning that'll say

37:21

hey, your sona filled the context window

37:24

and kind of depending on what's going on

37:26

you have two options

37:27

You can run slash clear and just start over

37:29

and that clears everything out

37:31

except for, for instance, claude.md, or you can run slash compact

37:36

and what'll happen is basically it's like a

37:38

user message is inserted

37:40

and it just says something like

37:41

hey, I need to go summarize everything we've been up to

37:45

I'm going to give this to another developer

37:46

and they're going to pick up where I

37:47

left off

37:48

And then that summary is what kind of

37:50

seeds the next session

37:51

and you can go from there

37:52

We spend a lot of time tuning this

37:54

kind of Compact functionality

37:56

so that as you max out the contacts

37:58

window and then run compact

38:00

you can start back over and keep going

38:04

Efficient workflows, what can you do with Cloud Code

38:07

and how do you get the most out

38:08

of it?

38:08

So using planning and to-dos

38:10

I talked a little bit about this before

38:12

but one of the best things you can

38:14

do is when you open up Cloud Code

38:15

instead of saying, hey, I need you to fix this bug

38:18

you can say, hey, I have this bug

38:20

can you search around

38:21

figure out what's causing it

38:23

and just like tell me a plan how

38:24

we're going to fix it

38:25

And this can save you a lot of time

38:27

because you can verify

38:28

you can read Cloud's plan

38:30

and you can verify what it's going to

38:31

do

38:32

And then the other thing that we have

38:34

is we have this to-do list feature

38:36

So often when Cloud's working on a big task

38:39

it'll create a to-do list

38:41

And if you're kind of paying attention

38:42

you can kind of watch this to-do list

38:44

and if you see anything kind of weirder

38:46

in there or something that doesn't make sense

38:48

that's when you can press escape and say

38:50

hey, Cloud, let's change the to-do list

38:51

I think you're on the wrong path

38:53

Smart vibe coding

38:55

So

38:55

it's very tempting and it's very powerful to

38:58

just let Claude work and press enter and

39:00

see what happens at the end

39:01

I think there's a few things that can

39:03

help make this better

39:04

and there's, I think

39:04

a talk later today about just this for thirty minutes

39:08

but doing things like having test-driven development

39:11

having Claude make small changes

39:13

run the tests, make sure they pass

39:16

always having Claude do things like check the

39:18

TypeScript and the linting

39:19

and then commit regularly

39:21

so that if it's kind of going off the rails

39:23

you can always fall back and try again

39:26

You can use screenshots to guide and debug

39:28

so Claude is built on top of our models

39:31

which are multimodal

39:32

You can always just grab a screenshot

39:34

paste it in

39:35

or if you have a file somewhere that's an image

39:37

you can just say

39:38

hey, Claude, look at this mock.png

39:41

and then build the website for me

39:42

or whatever

39:44

And an advanced technique

39:46

so as you're getting used to using Claude

39:48

what are some things you can think about

39:51

to kind of push things to the next

39:53

level?

39:53

And one of the things we see

39:55

both internally and with customers

39:57

is when you've started to use this tool for a while

39:59

it's going to be very tempting to use

40:02

multiple Claudes at once

40:04

And so

40:04

I know people at Anthropic and a few

40:06

customers that run

40:07

four Claudes at the same time

40:08

There's various ways to do this

40:10

You can have it in Tmux or just different tabs

40:13

all sorts of crazy things

40:15

So I would challenge you

40:16

to try

40:17

getting multiple Claudes running at once and

40:19

kind of be orchestrating all these things

40:21

It's quite fun. I can only do two

40:24

but I know people that do four

40:26

Use escape. So escape is your best friend

40:29

While Claude is working

40:31

you can kind of keep an eye on

40:32

what it's up to

40:33

and you can press escape to stop it

40:35

and interject and say

40:37

hey, I think you're going on the wrong path

40:38

or I want you to do something else

40:40

Knowing when the right time to press escape

40:43

is versus just letting Claude figure it out

40:45

is key to getting the most out of

40:46

the tool

40:48

And there's a hidden feature

40:49

not too many people know about it

40:50

but if you press escape twice

40:52

you actually jump back in your conversation

40:54

You can go back and you can kind

40:56

of reset

40:58

Tool expansion and MCP

41:00

So this is taking it to the next

41:02

level. If you feel like

41:03

with Bash and with the tools that Claude

41:05

has that it still can't do something

41:08

this is when you should start looking at

41:09

MCP servers

41:11

And then headless automation. I think this is

41:13

a thing we're most excited about

41:15

but also we

41:16

are still trying to wrap our heads around internally

41:19

which is how can we use quad programmatically?

41:22

We have that in GitHub Actions

41:24

We want to figure out other creative places

41:27

we can start using it

41:28

I would challenge you all to do

41:30

the same

41:32

So with that said

41:34

I'm going to jump

41:35

over to my computer because there's one other best practice

41:39

which is

41:40

it's always good to stay on top of

41:43

everything that's new. So we're shipping super fast

41:45

I'm going to throw

41:46

I'm just going to go over a few

41:47

things that are new as of today

41:50

one thing is

41:52

when you're in cloud now and you fire it up

41:53

you can do slash model

41:55

You can see what model you're running on

41:57

I'm on default, which happens to be Sonnet

41:59

We can jump over to Opus

42:01

You can do the same thing in slash

42:02

config

42:04

Switch it here

42:07

So that's new. Make sure you're running the

42:09

model that works for you

42:11

There's another thing that's new about these models

42:14

which is you can say something like

42:18

can you figure out what's in this project?

42:20

And for a long time

42:21

for a while, we've had this, like

42:23

think hard or extended thinking

42:25

Now, this is great, but with our past models

42:29

we wouldn't let our model think between tool calls

42:32

and that's probably when the thinking matters most

42:34

So starting with Cloud 4

42:36

they can now, our models now think between tool calls

42:38

and we can watch this happen

42:40

So we have Cloud in this project

42:41

There's a few different files in here

42:43

and I'm just going to tell it to

42:44

think hard and

42:45

figure out what's in this project

42:47

And we can watch Cloud start to work

42:49

and so

42:49

the way you know you triggered thinking is

42:51

you'll see kind of this lighter gray text

42:54

and then it'll call some file

42:56

it'll call some tools

42:57

it'll read some stuff

42:58

and then we see some more thinking

43:00

And this is awesome

43:02

so I encourage you

43:03

when you're working on tasks and solving bugs

43:05

throw a think hard in there

43:09

And then the other thing

43:10

and you know what

43:12

we'll just throw it up real quick

43:14

is I have this in VS Code

43:16

but of course this is in JetBrains as well

43:18

but we have these new great integrations

43:21

with VS Code and JetBrains

43:23

we can do things like

43:24

Cloud's gonna know what file I'm in

43:26

what file am I in

43:31

That is not what I meant to say

43:32

but Cloud's gonna figure it out

43:35

And you can do things like this

43:47

So these are the sort of things I

43:49

would encourage you to stay on top of

43:51

We have a public

43:53

kind of

43:54

GitHub project called Cloud Code under Anthropic

43:57

You can post issues there

43:58

but we also post our changelog there

44:00

And so I

44:01

check this once a week and make sure

44:02

that I'm on top of all the new stuff we're shipping

44:04

because even I can't keep up with it

44:07

So with that said

44:08

we have like four minutes left

44:10

I'm happy to answer questions about anything Cloud

44:12

Code related. We have it here

44:14

I can live demo some stuff if you're

44:15

interested

44:18

let's do a few

44:22

Thanks, real quick. This might be obvious

44:24

but multiple cloud.md files in a project

44:29

I presume that's possible and it just figures

44:30

it out or not?

44:32

So there's a few options

44:33

Of course, like in the same directory

44:37

you couldn't

44:41

but you could have one here

44:44

And one in the subdirectory

44:45

and I think we changed this

44:47

so that all the subdirectory ones aren't read in

44:50

because like Anthropic, we have a monorepo

44:52

and people would open it at the top

44:53

and blow up their context with all the

44:55

Claude MDs

44:57

So we encourage Claude

44:59

when it's searching around and it discovers Claude.md files in

45:04

child directories that are relevant

45:05

to be sure to read them

45:07

But by default

45:08

it just reads the

45:09

Claude.md file in the current working directory when

45:11

you fire it up

45:13

and then also you can set one in

45:14

like, your home directory

45:16

there are things you can do

45:17

though. We have this new thing

45:19

like, in your Claude.md

45:20

you can start referencing other files

45:24

So you could, for instance

45:27

do something like this with an at sign

45:30

if you have other Claude.md files that you

45:31

just kind of know you always want to read in

45:34

to do something like that

45:37

Hi. Okay

45:39

I

45:40

have not had luck getting Claude to respect

45:42

my Claude MD

45:43

Like

45:44

there's one thing particular where I'll ask it to refactor something

45:48

and then it will leave inline comments explaining the

45:50

like, the what of it is

45:52

and it's, like, something that's extremely obvious

45:54

And so I'll tell it

45:55

like

45:56

go and remove any inline comments that describe

45:58

the what of what's happening

45:59

and then it will remove it

46:00

and then immediately do it again in

46:02

like

46:02

the same pass. So do you have any

46:03

strategies for dealing with that?

46:05

So there's kind of two things that

46:07

fix that. So that was actually kind of

46:09

a model problem. There's nothing in the prompt

46:11

We have actually a lot in the prompts

46:13

for 3.7 that said

46:15

whoa, do not leave comments

46:16

And despite that

46:17

the model just loves to leave comments

46:19

so it doesn't surprise me that your Claude

46:21

MD didn't help much either

46:22

We already did a lot

46:23

I did a lot of work to try

46:24

to tamp it down from

46:26

what happens out of the box

46:27

So we mostly fixed that in Claude 4

46:32

Now there might be some new weird behavior quirks

46:34

but the other thing we made better in

46:36

Cloud 4 is it's just better at following

46:37

instructions

46:39

and we've gotten a lot of feedback from early testers that

46:42

all of a sudden

46:43

whoa

46:44

my CloudMD is being followed way more closely

46:47

and it might be a good chance to

46:48

go look in your CloudMD and decide

46:50

do I still need this stuff?

46:52

Maybe I can take some of it out

46:54

Maybe I need to add a few new

46:55

things

46:56

So

46:56

moving over to the new models might be

46:58

a good time to

46:59

take another look at what's in there and

47:00

see what you need and what maybe

47:02

can go

47:04

for the record

47:05

I'm trying to think of something that you

47:06

might not have thought of

47:07

Yes. We're doing multi-agent execution and parallelization

47:11

Can you make it so that for four agents

47:13

say, agents two and three use the context from agent one

47:17

maybe agent four uses the context from agent

47:19

two at a certain point?

47:21

Yeah

47:22

yeah, et cetera. That's interesting

47:24

We're trying to, so kinda like I said at the beginning

47:27

we're trying to do the simple thing that works

47:29

which is just one agent that's

47:30

great at coding and does everything

47:33

I think we wanna figure that out

47:35

Probably what's going to happen is if you

47:37

wanted to do that

47:39

you would ask all your agents to probably

47:41

like, write to a shared markdown file or something like that

47:44

so they can all kinda

47:45

like, check in and communicate

47:47

sometimes, like, I'll be working with cloud.md

47:49

or cloud, and I'll just say

47:50

like, hey, I need you to write some stuff in

47:53

like, ticket.md for another developer

47:55

and then I'll fire up another cloud code

47:57

and I'll be like

47:57

hey, read ticket.md, like, another developer left this note for you

48:01

like, this is what you're gonna work on

48:03

So I'd think about trying to write that

48:05

state to a file

48:06

and then just kinda

48:07

like, count on the model's ability to just

48:09

like, read files and make sense of them

48:12

is probably the best you can do today

48:14

and maybe we'll figure out clever ways to expose that

48:18

in the product as something more native

48:22

Cool

48:24

All right

48:25

And with that said

48:26

I have some rare Claude

48:28

code stickers that I found in my backpack

48:30

so come find me. I'll be hanging out

48:32

over there or something

48:34

happy to chat. Thank you

48:40

Hey, can everyone hear me?

48:42

Thumbs up, all good?

48:43

All right

48:45

Everyone

48:46

I hope that you have had a fantastic

48:48

day at Code with Claude

48:49

London so far today. My name is Will

48:52

I'm on our engineering team at Anthropic

48:54

I sit on a team called Applied AI

48:57

What that means is I essentially split my

48:59

time between internal engineering work and time spent

49:03

building agents with customers. So folks

49:06

imagine that

49:07

you built

49:08

and shipped an agent to solve a problem

49:11

I'm sure that's something that a lot of

49:12

the folks in this room have actually done

49:15

And imagine this, that this agent worked fantastic

49:19

right? But it worked so well

49:21

that a few weeks after shipping

49:24

you were asked to add some additional capability

49:27

to the agent

49:28

A few weeks after that

49:29

you received more business requirements and you added

49:32

additional capability

49:34

This pattern continued and continued until before you know it

49:38

your system prompt had grown to

49:40

become several hundred lines long

49:43

You have dozens of tools and sub Agents

49:45

that exist for your agent

49:47

and because of the complexity

49:50

you've started to see regressions

49:52

in the areas that your agent was previously

49:55

accelerating in. So if this is you

49:57

you're not alone

49:58

We see this type of scenario happen pretty commonly with customers

50:02

and actually with ourselves included in that

50:05

So within this workshop

50:07

we are going to

50:08

simulate an agent that is essentially grown to

50:11

a complexity

50:12

where we start to see degradation in its

50:15

performance

50:16

We're then going to walk through some of

50:18

the decisions that we as engineers and architects make

50:23

in order to improve the design of our

50:25

agent

50:26

to restore the performance that we expect

50:29

with the additional capabilities. Specifically

50:32

we're going to make some decisions

50:33

around tools and skills and sub-agents

50:37

As we modernize the stack of our agents

50:40

we want to make sure that we're using

50:41

the right agentic primitives at the right time

50:45

So when do you use a tool

50:46

when do you use a skill

50:47

and when do you use a subagent?

50:49

We're going to talk through all of that

50:52

in this session. As I mentioned

50:53

folks, this session will be hands-on

50:56

So let's go ahead and get started

50:58

I first want to walk you through

51:01

our problem statement in our agent

51:03

So for the purposes of this session

51:05

we're going to be focusing on an agent

51:06

called StockPilot

51:09

This is an inventory management agent

51:12

that was designed by and for a mid-size

51:15

retailer

51:17

The agent that you see on the screen

51:18

can do several things. It can flag low levels of stock

51:21

it can forecast demand

51:23

it can pick suppliers

51:25

it can file POs

51:26

and ultimately it can write weekly reports

51:29

for the employees of this retailer

51:32

Now, none of these capabilities are particularly complex on their own

51:36

but again

51:37

the issue is that we've essentially

51:39

bolted capabilities onto our agent over time

51:43

without modernizing our architecture

51:46

This complexity has started to cause some problems

51:49

Let's take a look at the actual architecture

51:51

today of the agent

51:53

Folks, today the agent is facilitated by a single orchestrator

51:57

so you see the stock pilot orchestrator sitting

52:00

at the top of the screen

52:02

The agent has a system prompt

52:03

as I mentioned

52:04

that's grown to be about 400 lines long

52:08

It has 12 different tools

52:10

Three of those tools happen to be wrappers

52:13

around subagents with completely isolated context windows

52:17

If you have the repo pulled up

52:20

which we'll go into more detail in just a bit

52:22

there's an agent that's under a folder called before

52:25

which essentially walks through this

52:27

this agent exactly. So again

52:30

orchestrator, long system prompt, a lot of tools

52:33

we have a lot of subagents

52:36

The result of this

52:37

is that our evals have started to dip

52:39

So let's imagine how we got here for a moment

52:41

Again

52:41

like we built that agent up front to

52:44

solve a really specific problem

52:47

We received business requirements to say add maybe

52:50

some forecasting capability to our inventory management agent

52:54

So what we decided to do was essentially

52:55

just spin up a forecaster as a sub-agent

52:58

Again, later on, we

53:00

we received more requirements to add report writing

53:02

capability to our agent

53:04

so we decided to add another sub-agent

53:07

for that report writing capability

53:11

Again

53:11

our eval started to dip over time

53:14

because we added more and more complexity

53:17

while just bolting this capability on

53:19

So let's take just a little bit of

53:21

time and talk about eval specifically

53:25

For this agent, folks

53:27

we have twelve different eval tasks

53:30

Across five different types of graders

53:33

So my colleague gave a talk on evals

53:35

shortly before this

53:37

Evals will have a component within this workshop

53:40

but it won't be the main focus

53:41

I'll give you a quick summary of the

53:43

tactical evals that we're using for this agent

53:47

On the left side of the screen

53:48

you see some

53:49

IDs. You see several evals that start with

53:51

the letter R. This stands for regression

53:54

These are more realistic

53:56

single-turn tasks that we grade the model's capability

54:00

on. So imagine I give the model a task

54:03

the model comprehends that task in the for- within the agent

54:07

calls some tools

54:08

and then provides a response back to me

54:10

We're essentially evaluating that response

54:13

We also have some more complex

54:15

tasks that we're grading the model on

54:17

So you see those F

54:19

IDs

54:19

the IDs that start with

54:20

F on the left side of the screen

54:22

That stands for failure mode

54:23

In this case

54:24

we're evaluating the model

54:26

over a more complicated multi-turn task

54:30

That we're grading

54:31

Now, again

54:32

I won't go into evals too specifically

54:34

We have a number of different types of

54:36

graders that are both deterministic and non-deterministic

54:41

right? When I talk about not

54:42

when I talk about

54:43

deterministic evals

54:45

we're grading things like turn count and like

54:48

latency and like the number of tokens that

54:50

are used as our agent is completing a particular task

54:54

and we're tracking those deterministic metrics over time

54:58

We're also using the idea of LLM as

55:00

a judge to evaluate the non-deterministic characteristics of

55:04

our agent

55:05

So personality and tone and style and output quality

55:09

we're using a non-deterministic grader

55:12

as a part of our eval to evaluate our agent's

55:15

non-deterministic characteristics

55:18

Now

55:19

we're going to run the evals for our

55:20

agent in just a bit

55:21

but when you do

55:22

you'll find that the agent is struggling

55:24

a bit. I'll talk about some of these

55:26

evals in just a little bit more depth

55:28

So F1 on the screen

55:30

third from the bottom

55:31

this is essentially simulating a daily low stock sweep

55:35

So again, this is an inventory agent

55:37

We're simulating our ability to look through all

55:40

of our inventory and pull the low levels

55:41

of stock

55:43

This eval you'll find will actually fail

55:46

because the agent is going to do the right thing

55:48

but it's going to take a very

55:50

winding path to do so

55:52

So instead of taking the straightest line from

55:54

point A to point B

55:55

the agent is going to take a very

55:57

inefficient path. It's going to get to the right end

56:00

but it's going to fail the eval because

56:01

it's not at the efficiency that we'd like

56:04

F2 on the screen is another eval that

56:06

you'll see fail

56:08

This eval actually evaluates the

56:10

ordering process under a particular promotion package

56:14

This is going to fail because we

56:16

are using a subagent for this particular task

56:21

The subagent is actually getting the task right

56:24

but there's a communication breakdown between our subagent

56:28

and our orchestrator. This is a really common

56:30

point of failure that we see when customers have

56:32

have really complicated systems with a lot of

56:34

subagents

56:35

It's important to get the communication

56:38

between your subagents and your orchestrator just right

56:41

In the case of F2

56:42

like you see on the screen

56:43

this is an eval that's going to fail

56:45

because we have a breakdown in that communication

56:50

The last one that I'll highlight that you'll

56:51

see fails is R8 on the screen

56:55

R8 will essentially

56:56

check the forecasting during a particular promotion month

57:00

This eval is also going to fail

57:03

because we have two different policies that live

57:06

in very different parts of our system prompt

57:08

and actually end up contradicting each other

57:11

So I mentioned over time our system prompt has grown

57:14

we start to have some conflicts

57:15

and the model gets confused

57:18

leading towards a failure for this particular eval

57:21

Now, in the repo, you'll see it in the README

57:24

when we run these evals

57:25

you'll see that they're going to pass

57:27

up front at about 83%

57:30

which is okay

57:31

but if you work in the world of manufacturing

57:32

that is not okay. 17% failure

57:35

is a really expensive failure percentage

57:39

Now, let's double-click on R8 again

57:42

just so that we can understand a little

57:43

bit about what's happening behind the scenes

57:46

Again

57:46

R8 is where we're essentially calculating

57:49

the forecast

57:51

during a particular month with a promotion

57:53

And so in my

57:55

on my screen here

57:55

on the right side

57:56

where you see kind of the simulated terminal window

58:00

within the first block under the commented text

58:03

we can see that the agent pulled the

58:05

right forecasting baseline

58:07

and also pulled the right promotion multiplier

58:10

So forecasting baseline, 12 units a day

58:12

promotion multiplier, 3.1x. This is all correct

58:17

But in the calculation part below that

58:19

we can see that there was actually some

58:21

kind of hallucination that happened

58:23

Instead of using that 3.1x promo multiplier

58:27

the agent actually ended up using 1.35

58:30

So something happened along the way

58:33

A hint here is that the reason for

58:35

this is that we have context problems

58:37

So this isn't a model problem

58:39

It's an issue with our

58:40

the information that we're surrounding the model with

58:43

Our system prompt has grown to be really

58:45

long and is very confusing for the model

58:47

and has some conflicts in it

58:49

which lead to the issue that shows up

58:52

within this eval

58:54

So, folks

58:54

our objective in this workshop will first be

58:57

to run our suite of evals

59:00

We're going to triage the issues

59:02

and we're going to update the design of our agent accordingly

59:06

and then

59:06

we're going to do something that we call

59:08

internally hill climbing towards eval improvement

59:11

right? So we run our evals

59:12

we get a baseline

59:13

it's going to be about eighty-three percent

59:15

We're then going to optimize the architecture of our agent

59:18

and we're going to continue then running our

59:20

evals so that we climb on them

59:22

hopefully seeing the success percentage

59:25

improve over time

59:27

In this lab

59:28

we're also going to start with an agent

59:30

that is self-created on our messages API

59:35

again

59:35

if you have the repo

59:36

and you click on the before folder

59:38

I'll show you this in just a bit

59:40

this is an agent that is built from

59:42

scratch on our messages API

59:45

We're going to actually migrate that

59:47

that agent to cloud managed agents

59:52

Cloud-managed agents essentially allows us to

59:55

offload the messiness that comes with maintaining

59:59

an agentic harness

60:01

and

60:01

scaling agents safely and securely to thousands

60:06

and tens of thousands of users

60:07

right? Like if I want to build my

60:09

agent locally and run it locally

60:11

I can do that pretty quickly and pretty

60:13

easily

60:14

But the moment that I need to take that agent

60:17

I need to host it remotely

60:19

and I need to allow hundreds and

60:21

and, and thousands of users to

60:23

at the same time

60:24

engage with that agent

60:26

there's an infrastructure problem

60:27

there's a scaling problem

60:29

there's memory, there's security

60:30

there's so much that I have to account

60:32

for

60:33

So in order to offload that

60:35

so I can just worry about

60:37

the architecture of my agent itself and make decisions around tools

60:43

skills, and sub-agents

60:45

I'm going to offload everything else to cloud-managed agents

60:50

So again, to break that down just a bit

60:52

there's been a few talks on CMA so far today

60:56

but this is really where we're able to

60:57

separate the agent from the session details

61:02

from the sandboxed environment where tool calls are actually happening

61:07

again

61:07

this allows us to offload

61:09

particular parts of the stack to then worry about the

61:13

to then only worry about the design of

61:15

our agent itself

61:19

All right

61:20

I mentioned that we're going to get hands-on

61:22

in this workshop

61:23

We are going to go ahead and do

61:25

that right now

61:29

Now

61:29

what you see on the screen here

61:33

Is the workshop URL as well

61:36

if you haven't had a chance to grab it

61:37

feel free to go ahead and do so

61:39

this is where we're keeping all of the

61:41

different workshops throughout Code with Claude within London

61:44

so you can go back and revisit them

61:45

if helpful

61:46

Within this workshop, we're going to be working on agent decomposition

61:50

so that's going to be the name of

61:51

the folder that we're actually going to be

61:53

working within

61:57

Great

61:58

Let me jump forward here

62:02

Perfect. So the first thing that we're going

62:04

to do

62:06

As a part of this workshop is we're

62:07

first going to get a baseline

62:09

So when you open up that link

62:12

you'll first, clone the repo

62:15

so we're going to clone the repo locally

62:17

We have a UV project that's set up

62:19

so we're going to run UV

62:20

sync in order to make sure that we

62:22

have all of our packages and our dependencies

62:24

to be able to invoke the Anthropic

62:26

SDK

62:27

and then eventually deploy our agent to Claude

62:29

managed agents

62:30

So we can run UV

62:31

sync to do that. I mentioned previously that

62:33

we're going to need an API key

62:35

for this workshop as well

62:37

So using those credits that you got at

62:38

the start of this session

62:40

you can go to your Claude console account

62:42

and create an API key

62:44

If you copy the ENV example

62:46

you'll just have to manually copy your API

62:48

key into the ENV file that's created for you

62:52

Now, all the twelve evals that I previously walked you through

62:56

we have all of those set up already

62:57

So in order to get a baseline and run those evals

63:00

you have to run uv

63:02

run evals dash dash agent before

63:05

This is all in the README

63:06

but if you just run that command

63:09

you will be able to

63:12

actually go about running your evals

63:16

Now, in terms of our building here

63:18

we're going to take a number of steps

63:20

to actually go about running our evals

63:22

using Claude Code to triage the results of them

63:25

and then climbing accordingly on our agent

63:30

so we're first going to take some

63:32

we're going to take a look at our

63:33

the system prompt that we have for our

63:35

agent itself

63:36

so I mentioned earlier that our system prompt

63:38

is currently sitting at about 400 lines long

63:40

We've been stacking information on our system prompt

63:44

over and over again as we've continued to

63:46

get more business requirements. So our system prompt

63:48

is very long

63:49

We'll take a look at that

63:51

We are then going to take some time

63:52

to evaluate the tools that we're using

63:54

Right now, as I mentioned

63:56

we have 12 different tools

63:57

Three of them are actually kind of wrapped sub-agents

63:59

so we'll take a look to see what

64:00

we can do to make that more efficient

64:03

And then lastly

64:04

if there are any sub-agents that we really

64:06

need to make our agent effective

64:07

we're going to take a look at the best way

64:11

to actually construct sub-agents with cloud-managed agents

64:18

I'm going to jump back just for a

64:19

moment. There's one thing that I forgot to

64:21

mention for you as you get started

64:24

within the repo folder

64:25

there's two different, folders that you'll see

64:28

There's a before folder

64:30

and then there's a starter folder

64:32

Those contain two separate agents

64:34

So if you want to view the messages

64:36

API version of the agent

64:39

again

64:39

this is just me building my own agent

64:41

loop and my own agent harness around the

64:44

anthropic messages API to invoke cloud

64:47

You'll see that within the before folder

64:51

If you want to

64:52

view what that agent looks like when deployed

64:54

on cloud managed agents

64:57

you can look in the starter folder

64:58

which exists right below that

65:00

If you want to deploy your agent on cloud managed agents

65:04

you can run uv run deploy starter

65:08

So again, run your evals using the messages API version

65:13

dash dash agent before

65:15

You can then deploy your agent on cloud

65:17

managed agents. We already have it built for you

65:19

and it's really easy to use cloud code

65:21

to kind of compare the two

65:23

and understand exactly what's going on and what

65:25

some of the differences are

65:27

with cloud managed agents

65:31

Okay

65:31

so I'm going to jump over here and

65:32

we're just going to open up Cloud Code

65:34

and we are going to build together

65:36

I'm going to zoom in very far so

65:38

that you can see everything and so that

65:39

I can see everything and we'll just talk

65:41

through

65:42

exactly what happens when I run some of

65:44

these evals and we'll talk through

65:46

the process that we usually go through to

65:48

do what I just called hill climbing

65:51

on the evals themselves

65:53

Okay, so if you're looking at Cloud Code here

65:55

again

65:55

I just used Cloud Code to actually run

65:57

my evals because I want

65:58

Cloud's help in triaging what's going on

66:01

So this is me

66:03

I'm using Cloud Code. I have Opus 4.7

66:06

running as you can see on the screen

66:09

My effort level is set to extra high

66:11

I usually set effort as extra high with

66:13

Opus 4.7 and I forget about it

66:16

That's the effort level that I usually stay

66:18

on. We find that it gets great performance

66:21

with extra high effort altogether

66:24

Now, you can see on the screen

66:25

the first thing that I did was I

66:26

ran my evals. So

66:28

I used the bash capability in Claude Code

66:31

and I ran uv

66:31

run evals dash dash agent before

66:36

Claude actually went ahead and ran my evals

66:38

So I'm going to scroll down

66:39

and we're going to look at what Claude

66:41

found while actually running those

66:44

So you can see the response that we got

66:47

the results that we got from this eval

66:49

run was actually lower than what I told

66:51

you before. So we ran them

66:52

and we got 62%

66:54

which is worse than the 83% that we

66:56

started with

66:57

So we passed seven out of 12 of them

67:01

and it looks like Claude has provided us

67:03

with a diagnosis for the different

67:06

evals that we actually failed

67:10

Let's scroll down just a bit more

67:16

And

67:17

we

67:17

are going to use Claude to understand a

67:19

little bit more about why this actually happened

67:23

So you can see here

67:24

I am using Claude

67:26

to provide me some of the themes

67:28

around why we actually failed some of these evals

67:30

Again, this is a great technique

67:32

If you have evals for your agent

67:34

again, as Geary showed before this session

67:36

you can use Claude to actually go about

67:38

triaging these

67:39

So it looks like there's a few different

67:40

themes that Claude is figuring out based on

67:42

this agent. So the first thing

67:44

Claude is seeing that

67:46

our model is taking on a lot of work

67:49

that it should have tools in order to

67:51

do. So our model is doing a lot

67:52

of reasoning across information

67:54

that it just doesn't have the tools to

67:56

be able to complete

67:58

It looks like there is

68:00

some issues that we have with the enforcement

68:02

of output structure. So our model

68:04

and our sub-agents are producing information in a

68:07

particular output structure that doesn't align

68:11

exactly with, what we're looking for with

68:15

to, to pull the best performance from

68:18

from our agent

68:20

If I continue to scroll down here

68:22

you can see there was some policy issues

68:24

et cetera

68:25

as I mentioned before

68:26

we have a system prompt that's really long right now

68:29

and so Claude is seeing some confusions based

68:32

on the information that's found within the system prompt

68:36

So again

68:36

you can see Claude has found some root causes

68:40

Now, we're going to do a few different things here

68:42

Again

68:42

we're going to go one by one and

68:44

address some of the areas

68:46

that we're seeing issues on within our agent

68:48

So I'm going to scroll down here

68:50

and we are going to use

68:52

Claude Code to triage some things within our agent

68:57

Okay

68:57

so the first thing that I'm going to

68:58

ask Claude to do

68:59

we're going to talk through this

69:00

Claude is making some changes

69:01

which is great

69:03

system prompts tend to get very

69:05

very long when we accumulate agents over time

69:08

So the first prompt that I ran

69:10

if you're following along

69:11

feel free to go ahead and do this

69:13

I encouraged Claude to look at my agent.py file

69:16

which is where our main CMA

69:20

agent loop is located. Again

69:21

that's agent.py

69:23

And I essentially said

69:24

hey, Claude

69:25

do you have any thoughts on the system

69:26

prompt?

69:27

Maybe I can use skills instead of a

69:29

long-running system prompt for progressive disclosure

69:33

So the first thing that we'll talk about

69:34

is skills. There's been a few other sessions

69:37

on skills. The short definition that I like

69:39

to use

69:40

Is that skills are packaged in composable

69:43

information

69:44

that Claude has the ability to pull into

69:46

context whenever Claude realizes that it needs that

69:50

information to complete a particular task

69:53

right? Skills are really useful with Claude Code

69:56

Like if you need to

69:57

provide Claude information on your testing process

70:00

or if you want to package up your

70:02

brand and your UI components

70:04

and bundle them into a skill that Claude

70:07

can pull into context whenever needed

70:09

skills are fantastic

70:11

Skills are also

70:12

useful within the agents that you're building for

70:15

your customers. So if you're building

70:18

a product

70:19

and you are going to give that product to customers

70:22

you're building an agent

70:23

skills are great within that

70:26

In the case of the agent that we

70:27

have on the screen here

70:29

again

70:30

we have a lot of different policies

70:32

and a lot of procedures that go into

70:35

our inventory management system

70:38

As I accumulated requirements over time

70:42

instead of building skills

70:44

I decided to take all of that information

70:46

and keep appending it to my system prompt

70:49

so my system prompt got longer and longer

70:51

and longer over time

70:54

This is not something that we recommend you

70:56

do based on the introduction of skills

70:59

right? Leave the system prompt

71:01

only for the information that Claude needs in its mind

71:05

regardless of the task that you give it

71:08

Skills are fantastic for packaging information that Claude

71:12

is going to need some of the time

71:15

not all of the time

71:16

right? So if I ask Claude to go build a forecast

71:19

Claude is going to

71:22

go ahead and do that

71:23

Let's see

71:23

I lost my computer just for a second

71:26

Here we go

71:27

If I ask Claude to go ahead and build a forecast

71:30

right

71:30

Claude is not going to need forecasting information

71:34

unless I specifically ask it

71:36

to go ahead and

71:37

and build that forecast

71:39

right? So in the case of that particular task

71:42

I want Claude to pull forecasting information

71:45

into its context window

71:47

Skills are also fantastic

71:49

for making sure that you are being efficient with context

71:53

because if you

71:53

stuff all of this information into the system prompt

71:56

you're polluting that context window with information that

71:59

Claude does not need

72:02

in order to complete a particular task

72:05

So again, the first thing that I did

72:06

I'll zoom in just a bit more so

72:07

that you can see this

72:09

and I'll scroll up just a bit

72:13

I said, hey Claude

72:14

can you help me take a look through

72:16

my system prompt?

72:18

can I use skills instead?

72:21

my system prompt is too long and I

72:23

need some help. And so Claude did an

72:24

analysis of this

72:26

and realized that I have some pre-built skills

72:29

that I can use

72:30

to supplement information in my system prompt

72:33

So the first

72:34

correction or fix that we're going to make

72:36

to modernize our architecture here is we are actually going to

72:42

remove, Many of, much of the system prompt

72:47

and we're going to put that information into

72:48

skills

72:49

And so you can see here

72:50

the first thing that we're doing with Claude

72:53

is we are

72:53

activating a number of different skills that previously

72:57

were not there before

72:58

and we're actually swapping our system prompt to

73:01

be a short prompt instead of a long

73:03

one. So if you're curious

73:04

if you feel like you have a long

73:05

system prompt within the agents that you're building

73:08

feel free to take a look at this

73:09

to see the differences between what was like

73:11

a 400-line system prompt compared to about a

73:13

50-line system prompt

73:15

We've supplemented that

73:16

and we've switched a lot of that information

73:19

to skills

73:21

Great

73:23

I am now going to continue working with

73:25

Claude. So you can see we made those changes here

73:27

which is fantastic

73:29

there's some evals that I can go rerun

73:31

I'm going to ask Claude to do one more thing

73:33

and then we're going to

73:34

we're going to rerun some of our evals

73:36

to see where we've improved

73:39

So I mentioned before that we have twelve

73:40

different tools. You saw those on the screen

73:43

in the second slide that I shared

73:46

As a part of this inventory management agent

73:48

we have

73:49

we have tools that we've created for everything

73:51

So whenever Claude needs to retrieve data

73:53

we have a tool. Whenever Claude needs to analyze data

73:56

we have a tool for that

73:57

We have tools for everything

74:00

So I'm going to ask Claude to take

74:02

a look

74:03

at the tools that my agent has

74:06

and help me think through how

74:08

I can optimize here

74:11

So right now Claude is running an analysis

74:13

across the different tools that I have for my agent

74:17

and we're going to get to see

74:18

what some of the results for

74:21

Now, while this is working

74:23

I'll give you a

74:24

a tip

74:25

when it comes to building agents that we

74:27

carry with us at Anthropic for our agents

74:29

internally and the agents built with customers

74:31

Whenever we build agents

74:33

we lean into the same primitives

74:36

that we as humans have access to

74:38

So imagine yourself when you show up to work

74:40

right? You have a computer that's sitting in

74:42

front of you

74:43

You have the ability to navigate files on

74:45

a file system. You can type in the

74:47

browser and you can search the web

74:49

If you're an engineer

74:50

you have the ability to write and execute

74:52

code

74:54

When you think about Claude Code as an agent

74:57

we've effectively given Claude access to all of

75:00

the same primitives that you and I have

75:01

access to when we show up to work

75:03

every single day. Like Claude Code is a

75:05

great coding agent

75:07

because Claude is really good at code

75:09

But essentially what we've done with Claude Code

75:11

is we've just given Claude access

75:13

to a computer

75:15

Right? And this is really powerful because this

75:17

allows us to

75:19

drop in

75:20

better versions of Claude as we continue to release new models

75:24

and Claude just uses those primitives better than it did before

75:29

right? Like

75:29

imagine yourself after this conference compared to yourself

75:32

when you walked in

75:34

You're going to have the same tools at your fingertips

75:36

but you're, theoretically

75:38

your brain's going to be a little bit bigger

75:40

you're going to be smarter based on what you learned here

75:41

and you're going to be more effective

75:43

while using the same tools

75:45

Claude works the same exact way

75:47

right? And so whenever we build agents

75:50

we lean into human-like primitives first

75:54

These primitives are things like code execution

75:58

And the navigation of a file system

76:00

the keeping of a to-do list

76:02

the ability to search the web

76:04

these are foundational tools that we always start

76:07

with when we build agents

76:09

and we remove them as needed

76:10

An example that I like to give is with file

76:15

like document analysis

76:17

If you're building an agent that requires document analysis

76:20

maybe you have a lot of

76:23

CSVs or Excel sheets

76:25

that your agent is going to be looking over

76:27

code execution, so the ability to write and run code

76:30

is one of the best ways of

76:34

doing data analysis and working across lots of documents

76:39

right?

76:40

Like if you need Claude to look across a CSV

76:42

giving Claude a bash tool

76:44

so that Claude can

76:46

write a quick Python script

76:48

and reason across the results after running that

76:51

Python script

76:53

is much more effective than just uploading the

76:55

entire CSV into Claude's context window

76:59

right? So again, we lean into these

77:01

computer-like primitives first when building an agent

77:06

So if I scroll down here

77:07

that's exactly what we did here

77:09

You can see we took a lot of

77:10

steps and we actually removed most of the

77:13

tools that exist within our agent

77:16

and we replaced them with

77:19

some of the primitives that I talked through

77:21

previously. This is an inventory management agent that

77:24

leans really well to this

77:27

I have the ability to

77:28

consolidate and remove a lot of the tools

77:31

that I'm using to reason across

77:34

Excels and reason across forecasting data and just

77:37

give Claude access to the same tools that

77:40

Claude Code has in order to do that

77:42

What's cool about this is that when you build using

77:46

Claude managed agents

77:49

These tools are actually included by default

77:52

So if you want to give Claude access

77:55

to those same tools

77:57

that Claude Code has and use them to

78:01

drive powerful capability within your agent

78:03

you don't have to worry about writing a

78:05

tool that gives Claude the ability to write and run code

78:08

or you don't have to write a tool

78:10

that gives Claude the ability to use the

78:12

file system

78:13

You can just rely on those built-in tools

78:17

that we have built ourselves for Claude Code

78:20

that we just make available through

78:23

Claude Managed Agents

78:25

I'm going to ask Claude to rerun an

78:27

eval

78:29

The evals to see if we are getting better

78:36

Now, with your agent

78:37

there's always going to be the need to

78:39

add some custom tools as well

78:41

Like you're not

78:41

you're only going to get so far by

78:43

giving your agent the same

78:44

tools that we give Claude Code

78:47

so we always start with those

78:50

primitives like code execution and web search and to-do lists

78:54

et cetera

78:55

we always start there

78:57

And then

78:58

we either remove those tools as we don't need them

79:01

right? There might be some agents where we

79:03

just don't need web search

79:04

so we'll go ahead and remove that tool

79:07

and then we'll add custom tools whenever we need them

79:10

right? So again, when you think about tools

79:12

I encourage you to start with those Claude Code primitives

79:15

those human-like primitives

79:17

and then

79:18

add custom tools only as you need them

79:21

In the case of this specific inventory agent

79:25

we were able to

79:26

remove most of the tools and replace them

79:28

with Claude Code. So you can see right now

79:30

Claude is redeploying my agent to Claude Managed Agents

79:34

So again, I have my agent Locally

79:37

I am redeploying it based on some of

79:38

the changes that we've made

79:40

and now I can rerun some of my

79:41

evals to see the results

79:44

So you can see in that last command

79:46

I'm rerunning, the F1 eval

79:49

and we're gonna see what happens as a result

79:53

Now

79:53

we always get a lot of questions when

79:55

it comes to MCP

79:56

So in the case of CMA here

79:59

you have a couple different options when it

80:01

comes to tools

80:02

You can first lean on those Claude Code primitives

80:04

things like web search and code execution and file system

80:07

Again, that's what we start with

80:08

You can then create

80:10

just custom tools

80:11

so standalone tools

80:13

that only your agent has the ability to

80:15

use

80:16

Then you can connect your agent to MCP

80:19

We see a lot of folks run towards MCP first

80:23

and a lot of our customers end up

80:25

in this ecosystem where there's a lot of

80:27

kind of chaotic MCP servers that exist

80:30

A lot of times they have overlap

80:32

which can create some problems

80:34

So when we build agents

80:35

again, we start with those cloud code tools

80:38

We then create

80:40

local tools only for our agent

80:42

We don't run to MCP

80:44

And then only in the case where we

80:46

have

80:46

a common collection of tools that multiple clients

80:50

will benefit from accessing

80:53

do we go about the process of collecting

80:55

those and publishing them as an MCP server

80:57

So only when we have multiple agents

81:00

maybe multiple cloud code clients

81:03

that need to access the same set of

81:05

standardized and governed tools

81:08

we run towards MCP

81:10

Something else that's becoming increasingly common throughout the

81:13

industry

81:14

is

81:15

leaning on Claude's ability to effectively use code

81:18

execution as a means of executing tools

81:22

So we see a lot of capabilities coming

81:24

out around just giving Claude access to

81:28

use CLIs

81:30

and invoke APIs using code and actually run

81:33

tools using code instead of MCP

81:36

One of the drawbacks of MCP is that it does

81:39

cause some

81:40

it can cause some context issues just in

81:42

terms of polluting

81:43

context and taking up a lot of space

81:46

So there may be some cases where you

81:48

can just rely on code execution

81:50

either through CLIs or just by giving Claude

81:53

the ability to invoke APIs using code as a means of

81:58

creating

81:59

more flexibility for your agent where you do

82:01

not have to use MCP

82:04

So just something to keep in mind

82:06

as you're building

82:08

Great. Okay, so Claude just got done

82:12

looks like we have the before and after

82:14

from some of the changes that we've made

82:16

and I think that this is pretty compelling

82:18

right?

82:19

The first thing that jumps

82:21

Off the screen to me is the token

82:23

usage. So

82:24

before I was using over 200,000 tokens for

82:27

a particular task

82:29

After leaning in on some of those file system primitives

82:32

you can see that that went down dramatically

82:34

This is a direct result of giving my

82:37

agent code execution

82:38

So again

82:39

imagine instead of giving my agent a

82:41

full CSV that needs to be read in the context

82:44

I just give my agent the ability to

82:46

write

82:47

and run Python as a means of kind

82:49

of navigating across all of that information

82:51

The agent uses a lot less tokens when

82:54

it can write code and then run code

82:56

and then read the results

82:58

instead of having to consume all of that

83:00

data

83:01

in Claude's mind

83:02

and then use all of that kind of

83:04

collective brainpower to then make decisions based on

83:07

the results

83:08

A few other things

83:10

we can see that our costs went down

83:11

as well because we're just not using as many tokens

83:13

which makes sense

83:16

Our execution time went down as well

83:19

So

83:20

this was a pretty good Case where I

83:21

think we got better overall

83:23

but this is not something that will happen all the time

83:25

right? We, like, we might see some cases where we regress

83:29

but this was the case where

83:30

using some of those primitives as opposed to

83:32

some of our more stagnant tools was clearly the

83:36

the right decision

83:39

Great

83:40

Okay. We're going to jump back and we're

83:42

going to talk about sub-agents

83:43

for just a bit

83:45

I'm going to copy another prompt to Claude

83:47

and we are going to

83:49

investigate sub-agents

83:53

Now

83:53

I mentioned before that we had

83:55

twelve different tools

83:56

Three of them were effectively wrapping sub-agents

83:59

So if I'm Claude

84:00

I have the ability to call on a tool

84:02

that tool's a wrapper for a sub-agent

84:04

I can then go and invoke that sub-agent

84:07

See, Claude's doing a lot here

84:08

I'll scroll up just a bit and then

84:09

we'll talk through it

84:11

The two main use cases Where

84:15

or the two main instances where we see

84:17

sub-agents initially as being really effective

84:20

is first when you want to throw a

84:22

lot of clod at a problem

84:23

right? So let's say that you're trying to

84:24

do deep research or like web search

84:27

let's say that you're trying to do in

84:28

the case of clod code

84:30

code base exploration

84:31

That's a great case where like having many

84:33

different minds

84:35

running at the same problem makes sense

84:37

So sub-agents are a great way

84:39

to parallelize and throw a lot of clod

84:41

at a problem to get it done faster

84:43

and more effectively

84:45

The

84:46

second case where it's really common to use

84:48

sub-agents

84:49

is when you need a fresh mind to

84:52

look at a problem

84:53

So I'll use the Claude Code example first

84:55

If I'll

84:57

I'll use my example as a developer

84:58

If I am writing code

85:00

I do not want to be the same

85:01

person that is writing and also

85:04

reviewing my code. I'm going to have somebody

85:06

else review my code

85:08

So in the case of Claude Code

85:09

it makes a lot of sense

85:10

to have one instance of Claude doing the

85:12

writing of the code

85:15

and then another instance of Claude coming over

85:17

the top and reviewing that

85:18

That does not have context about the initial

85:22

instance of Claude. This is a great case for a sub-agent

85:25

using just a code review sub-agent and layering

85:27

it over the top is a great way

85:29

to do this

85:30

We also have a

85:31

a sub-agent within, our Our agent here

85:37

our inventory management agent

85:39

that we've actually kept as the result of

85:43

of some of the changes that we've made

85:44

and that's for forecasting specifically

85:46

So again

85:47

I have a forecasting capability that's within my

85:49

inventory management agent

85:51

I do want to keep my forecasting

85:54

separate from my main instance of Claude

85:57

I don't want anything in my initial context

85:59

window to distort the forecasting process

86:03

I do have a skill that kind of

86:05

walks through the step-by-step sequence

86:07

and the guidelines that I prefer Claude use

86:10

when writing and building forecasts

86:12

But again, I don't want the same Claude that I'm

86:14

that

86:15

my customer is talking with

86:16

to also be the Claude that writes the forecast

86:18

right? So I want to divide that

86:20

So I'm leaning on that second example of

86:22

when to use a sub-agent

86:24

as the place where we'd like to go

86:25

about doing this

86:28

So in this case

86:29

we've removed our other sub-agents and we've just

86:32

replaced them with primitive tools

86:34

but we are going to leave the forecasting sub-agent

86:38

Now

86:38

we're not going to expose our sub-agent as

86:42

a tool

86:44

Using cloud managed agents

86:45

there's a native capability for sub-agents

86:49

that allows the

86:52

logging and the observability of your sub-agents

86:55

to be really effective. One of the problems

86:58

with sub-agents is that

86:59

when you have multiple instances of cloud running

87:02

first off

87:02

it's difficult to make sure that the communication

87:05

between your orchestrators and your sub-agents is accurate and is seamless

87:10

right? There's a lot that can get lost

87:11

in translation

87:12

Just like when I'm talking to one of my colleagues

87:14

I might be thinking something

87:15

they might be interpreting it completely differently

87:17

The same thing happens with orchestrators and sub-agents

87:22

The

87:23

the other thing that can happen

87:25

is logging is really difficult in some cases

87:28

right? Because then you have to worry about

87:29

collecting the transcripts from multiple different agents

87:34

So within Cloud Managed Agents

87:36

we've added this native sub-agent capability

87:39

I saw it on here

87:40

let me scroll up just a bit

87:42

I think Cloud found it

87:43

Yes

87:44

So there's this callable agents capability that exists

87:47

within Cloud Managed Agents

87:48

which is essentially just like managed sub-agents

87:51

so that within your session information

87:53

you have

87:54

observability

87:55

and metrics about what exactly your sub-agents are

87:58

doing that is as accurate as your initial orchestrator

88:03

right?

88:04

this is, again

88:05

meant to solve one of the common problems of just having

88:08

a lot of information that is hard to

88:10

track with sub-agents

88:12

We just did some building

88:13

Again

88:14

I'm going to skip through these because we

88:15

spent some time talking about them

88:17

We just talked about sub-agents

88:19

Again

88:19

there's a few different cases where you can

88:21

use them. We just talked about callable agents

88:24

You can also just define your sub-agent as a tool

88:26

which is what we did previously

88:27

but we actually moved away from that and

88:29

we decided to use the CMA native capability

88:32

There are a lot of cases where you

88:34

can just now scrap the sub-agent entirely and

88:36

just give more flexibility

88:39

and capability to your main agent

88:41

So what we have a lot of customers

88:42

doing is actually just consuming capability into their main

88:47

in this case, orchestrator

88:48

because frontier models have gotten intelligent enough

88:52

to manage across more information where you just

88:54

don't need as many sub-agents

88:56

So again, when you're thinking sub-agent

88:58

I had a lot of claw

88:58

or I have a big problem that I

89:00

want to throw a lot of claude at

89:01

or I want a separate claude to kind

89:03

of look at

89:04

the work of either me

89:06

or of a different instance of claude

89:09

Two great times to use sub-agents

89:12

Okay

89:13

so let's look at the architecture that we ended with

89:16

Again, refreshing us, we started with an orchestrator

89:19

system prompt of about 400 lines long

89:21

we had 12 tools

89:23

three of them were sub-agents

89:24

What did we end with after this exercise?

89:26

We still have an orchestrator

89:29

but we deployed that on cloud-managed agents because

89:31

I didn't want to have to worry about infrastructure

89:33

scaling, security, etc.

89:35

I just wanted to worry about my agent

89:37

right? Like in, in Will's simple terms

89:39

like that is when I reach for cloud-managed

89:41

agents because I just want

89:43

to worry about building the best thing possible

89:45

and not all the messiness that comes with

89:47

it

89:48

We simplified our tools. We now have

89:51

we have right now three different tools

89:54

so we actually simplified everything to just use bash

89:57

read, and write

89:58

Now, when our agent starts executing

90:00

we sync some data

90:02

into the cloud-managed agents environment so that it

90:04

can reason across that data

90:07

We actually simplified our system prompt to fifteen lines long

90:10

and we replaced all of our business logic

90:13

with skills

90:14

So again

90:15

I was just stuffing requirement after requirement into

90:18

my system prompt. I decided to take that

90:21

package it up as skills

90:22

so that Claude could pull that information into

90:24

its brain

90:25

only when Claude realized that it needed it

90:27

in order to solve a problem

90:29

As a result of this

90:31

we showed how we can then start hill

90:32

climbing on evals to see improvements over time

90:36

So at the end of this

90:37

my eval score is about ninety-two percent

90:39

I've simplified my design

90:42

I'm leaning into some of the primitives

90:45

that make, Claude great

90:49

and I'm seeing the positive results after doing so

90:54

Again, some of the eval results

90:56

you see that here after running this

90:59

we're getting faster

91:00

we're using fewer tokens because we're leaning into

91:02

code execution

91:04

our turn count is remaining sort of the same

91:07

but again

91:08

because the token usage and the cost is going down

91:11

I'm actually okay with Claude taking more turns

91:14

There are some cases where we'll see the

91:16

latency not drop maybe as much as you would expect

91:19

but for some of these more sophisticated high-intelligence agents where

91:22

like, forecasting is at play

91:24

I'm willing to take a little bit higher latency

91:27

at the

91:28

expense of seeing my performance improve and my

91:31

costs go down

91:33

All right, let's wrap with some

91:35

some takeaways here in our last minute

91:40

When we build agents

91:42

we start with a single agent loop that

91:44

that is equipped with very simple primitives

91:48

that give Claude some of these human-like capabilities

91:52

like the ability to use the file system

91:54

like the one that you have on your computer

91:56

web search, code execution, sometimes a to-do list

92:01

Again

92:01

we start there and then we build accordingly

92:04

The next thing that we did is we

92:05

used progressive disclosure through skills

92:07

Instead of stuffing our system prompt with a lot of information

92:11

we made information accessible to Claude

92:14

whenever Claude realized that it needed that information

92:16

in order to solve a problem

92:17

This is great because we can run more efficiently

92:20

And, we're not polluting our context window

92:24

and we're giving Claude more flexibility to make

92:27

decisions

92:28

The last thing that I want you to walk away with

92:29

right, evals in general. This idea of hill climbing is

92:32

is a concept that we lean on really

92:35

heavily at Anthropic, right? You have evals

92:38

you establish a baseline

92:40

you then tweak your architecture and you rerun

92:42

evals and you get better over time

92:45

Now, as a result

92:45

it's important to make sure that your evals

92:47

are updated as your product capability expands

92:51

Always make sure that your evals are encompassing

92:53

the things that you care about

92:55

and that you're measuring within your agent

92:57

so that you can actually make sure that

92:58

your agent is accomplishing the thing that you

93:01

set out to accomplish

93:04

With that, folks

93:05

we're going to go ahead and wrap

93:07

I really appreciate your time today

93:09

I'll be in the back after the session

93:11

just outside of this room

93:13

in case you have any questions at all

93:16

thank you. Welcome all

93:18

Thank you for joining the workshop

93:20

We have a pretty

93:22

pretty full house here today

93:24

I'm very pleased to see that

93:27

quick show of hands

93:28

who here loves cloud code?

93:33

Okay, yeah. You're in the right place

93:36

Yeah, yeah

93:37

Lovely to meet you all

93:38

My name is Arnav

93:39

I'm a member of the Applied AI team

93:41

I'm an architect there

93:43

I'm here today to tell you

93:45

how we cloud code at Anthropic

93:47

Do a workshop, and, you can code along

93:51

You'll get some credits

93:53

I think there'll be a QR code in just a moment

93:55

so make sure to grab that

93:56

there'll be people floating around that might support you

93:59

so if you have technical difficulties

94:00

we can, we can help you out

94:03

Anybody here in this room who hasn't used

94:05

Cloud Code before?

94:08

Okay, great, good, good, because this is, there's more to do

94:12

there's more to find

94:12

more, more to find out

94:14

I'm happy you're here. Let's get started

94:16

So let me get my clicker

94:19

yes, excellent, yes. Please, please, please grab the QR code

94:24

set yourselves up

94:26

There'll be a repo here as well

94:27

that you can clone

94:29

there are three phases in it we're going

94:31

to work through

94:32

There's an interesting verification setup that you can

94:34

work through alongside me

94:36

and it's quite detailed

94:37

so

94:38

you probably will want to investigate it a

94:39

bit later yourselves

94:42

What we're covering today is based on

94:45

this version of the talk that Tarek gave

94:48

in San Francisco

94:49

just about a week and a half ago

94:51

Who here follows Tarek on

94:53

on Twitter?

94:55

Fantastic, great

94:55

yeah. Yeah. So

94:57

he published that

94:58

as a blog post called The Unreasonable Effectiveness of HTML Files

95:02

and, he's basically pitching that moving on from Markdown files

95:06

we're going to be using HTML instead

95:08

And we're showing some of that today

95:10

Like I said, there's a repo to go along

95:13

en-enjoy that along the way and later on

95:16

in the, in the week as well

95:18

Where are we now?

95:19

Where's this all going?

95:21

I think everybody probably notices that

95:23

agents are becoming more and more capable

95:26

Why is that happening? That's happening because the

95:28

models are becoming more and more capable

95:30

And so if the models become more capable

95:32

it means agents can run for longer and

95:34

you can give them more and more complex

95:35

tasks. But that also means

95:38

that we have to change our way of

95:39

working. We have to change our habits

95:41

That's part of what today is about

95:42

How can you

95:44

change the way that you're working with Cloud

95:45

Code to get more out of it?

95:50

If you're going to let your agent run

95:53

for a longer period of time

95:55

Then

95:57

you can burn through a lot of tokens

95:58

if it does the wrong thing

95:59

And you want to avoid that ideally at

96:01

the beginning

96:02

And so that's where the idea came from

96:04

to, to

96:05

to push more and front load more

96:07

of the

96:08

verification

96:09

that the human would do in the spec

96:11

into an HTML file because it's a more rich and more

96:16

more human ergonomic

96:17

way of engaging with the content that your

96:19

agent is going to be building for you

96:23

And we'll see how that works

96:27

There are a bunch of things

96:29

around agents that are worth reading

96:31

We have a lot of engineering blogs on

96:33

our webpage

96:34

we also have the

96:35

the standard blogs. You should check all of

96:37

those out. There'll be a lot of information

96:38

there

96:39

There's a lot of interesting information about harnesses

96:41

about long running agents. All of these things

96:43

are important

96:44

Today we'll focus on three levels

96:47

and there'll be some

96:48

something a bit more basic

96:49

something a bit more next level

96:50

and then something that's quite interesting

96:52

and that'll be a bit more in depth

96:54

The first thing is

96:56

the more capable the models get

96:59

the more you should try to resist constraining them

97:02

Who here, by the way

97:03

has heard of The Bitter Lesson by Richard Sutton?

97:07

Great, fantastic, this is, this is great

97:09

So I'll, I'll, I'll, for those who haven't raised their hands

97:12

Richard Sutton is basically the

97:14

the father of reinforcement learning

97:16

If you're reading a book about reinforcement learning

97:18

it's probably authored by him

97:20

And, his idea was basically that

97:23

you know, you could spend all your time trying to

97:25

with your human capabilities

97:27

hard code up front and constrain the system

97:30

but in the end

97:32

pouring more data and more compute at it

97:34

Ends up getting more capability

97:36

than anything that you could have come up

97:37

with. And there's a similar analogy here

97:39

right? The, the models are becoming more and more capable

97:42

and so

97:43

you should accept that

97:45

the model is probably better

97:46

at extracting requirements from you

97:49

than you are at defining your requirements

97:52

The, the requirements are latent within you

97:53

just like when you talk to your users

97:54

your users, they have an idea of

97:56

they know it when they see it

97:58

but they're often

97:59

not very good at articulating what they need

98:01

And likewise

98:01

you probably know what you want when you see it

98:04

but

98:04

Claude is likely better at extracting

98:07

what you want

98:08

and what you need from you than you

98:09

are in specifying it to Claude

98:11

That's another direction to take this in

98:14

So we'll talk about that

98:15

removing ambiguity

98:16

letting Claude

98:18

prompt you and interview you and prompt

98:21

Then how to then understand and plan

98:24

We used to do this all with

98:25

with Markdown files

98:26

we still do it with Markdown files

98:27

A colleague of mine once said

98:29

the Markdown file is the lingua franca

98:31

Of the AI native software development lifecycle

98:34

That was pretty poetic

98:37

but it seems like

98:39

that format is a bit constrained

98:40

It's getting too long. There's a lot of

98:42

lines of markdown file to read

98:44

We can condense more information into HTML files

98:47

and that's what we're going to do today

98:49

And then how to verify

98:51

not test actually, but just to verify in this context

98:55

how to make verification

98:58

native to the thing itself so that the

98:59

agent can drive it alongside a human

99:02

or eventually not, headlessly as well

99:04

That's where this is all going as well

99:05

right? The

99:05

the agents are going to be doing more

99:07

and more of this natively

99:08

and how can you set

99:09

the artifacts that you produce up to

99:12

natively be testable and

99:14

verifiable in the way that you need

99:17

So we'll do an example

99:19

By the way, has everybody had a chance to get

99:21

grab that QR code and

99:23

set themselves up? Very good

99:25

There'll be a link to the repo eventually

99:26

as well. I think that comes

99:28

in a bit

99:29

let's say we want to do a bill

99:30

splitting ad

99:33

very simple, you know, you'd want to

99:38

you go out with friends

99:38

you want to find out who owes what

99:40

yeah

99:42

let's let Claude interview us around doing this

99:46

so in this case

99:47

I'm actually going to

99:51

yeah, before I do that

99:53

give you an example of how you would

99:54

do this and how you could do this better

99:56

yeah

99:58

What's good prompting, what's bad prompting?

100:03

Bad prompting is when you say

100:04

just make it better

100:05

And a lot of people that I watch

100:07

using Claude Code just type

100:08

make it better

100:10

make no mistakes, yeah. Yeah

100:13

that's not good prompting

100:15

you want to encourage

100:16

you want to encourage Claude

100:18

to extract from you specific details

100:20

So give, give the domains, don't over-specify

100:24

The outcome, but specify the areas that you are interested in

100:26

right? That's what makes this good prompt on

100:28

the side different and better

100:31

we're making, you know, focusing on the audience

100:33

for example, or, suggesting an open-ended way to answer the question

100:39

as opposed to predefining it upfront

100:42

And then that will prompt Claude to iteratively

100:44

interview

100:46

All right. So, I've got a Claude open here

100:49

two different Claudes

100:51

Who here has used fast mode?

100:54

Okay, not that many people use fast mode

100:56

That's why I set it up here

100:59

and who here uses auto mode?

101:04

Oh

101:04

I'm very happy that you all use auto

101:05

mode. You need to be using auto mode

101:06

If you're not using auto mode

101:08

you need to be using auto mode

101:09

It makes it so much easier

101:11

yeah, use auto mode. Good

101:13

Who here is setting their effort parameter?

101:16

Good, good

101:17

Our recommendation is X high

101:19

but you can also set max effort

101:20

I mean, in my case

101:21

I think I kept it at X high for this

101:24

yeah, yeah, yeah

101:28

So, just to touch on how those look

101:30

right, like we've got forward slash effort

101:32

which is the effort parameter

101:33

forward slash fast, which is the fast mode

101:35

which is I've, I've turned it on

101:37

And then, we have, the auto mode

101:41

which we cycle into like this shift tab

101:42

yeah? Auto mode is the best

101:44

So, I'll copy paste my

101:46

my, you all, when you have the repo

101:49

will see, one of these pointers in there

101:52

so I'll just copy paste the prompt in here on the

101:54

the bill splitting app

101:56

So, okay, Claude's gonna ask me

101:59

you tab through these

102:00

right, like you'll tab through these like this

102:02

and in my case

102:04

I want to, I want it just for friends

102:10

and

102:13

I want it

102:16

is there a secondary audience?

102:17

No, no secondary audience

102:19

and we'll leave that as that

102:21

The key thing here in the prompt is

102:22

that you want to use

102:24

you want to prompt Claude to use the

102:25

ask user question tool

102:27

that you saw me use in the

102:29

prompt earlier before. Let me slip in those answers

102:32

and then it'll write that spec

102:33

right? So you

102:34

you saw me in my prompt

102:36

explicitly referring to the ask user question tool

102:40

That's what triggers this workflow

102:42

and, you know, depending on how well you specify that prompt

102:45

you'll get better outcomes

102:46

So it'll generate a spec

102:48

We could then turn that into

102:51

a bunch of different ways of

102:52

of, of building the app

102:55

It's going to take a while

102:55

so we'll go back to my slides

102:57

Could I go back to the slides

102:57

please?

102:59

Great

103:00

So let's say we have a plan

103:02

generates a plan, we've answered some questions

103:04

it, it

103:05

it's getting better at extracting turn by turn

103:07

from me what I actually want without me

103:09

having to upfront verbalize everything myself

103:17

Then I want to be able to check

103:18

is it what I want?

103:20

Well, an HTML file is more dense

103:24

much more information dense

103:25

much more ergonomic for you to understand

103:28

what the thing is going to look like

103:29

You can even use a screenshot with it

103:30

you can use the playwright MCP later on

103:32

for that as well

103:33

right? You can interact with it much more

103:35

richly than you could with a very long markdown file

103:39

and especially if the markdown files get

103:41

you know, they get more than about 200 lines long

103:45

unlikely you're going to read it

103:45

and certainly unlikely that your colleagues are going

103:47

to read them

103:49

Before we started here

103:50

I had Claude with Opus 4.7

103:52

generate a few examples

103:54

for me of what this could look like

103:55

with my bill splitting app

103:57

and we'll look at those now

103:59

We have them here

104:02

The prompt I used is in the repo

104:04

that you'll have access to

104:05

I asked prod, give me a few different directions

104:08

four different design directions

104:09

explore them, generate them as HTML

104:11

and let me explain

104:12

explore them across each other

104:14

And they came up with these different ones

104:16

So one sort of Brutalist

104:18

one Tokyo, FinTech

104:20

we'll click them all one by one

104:21

So this is what this one would look

104:22

like

104:25

and then this one

104:26

this one might look like this

104:28

Completely different aesthetic, right? Like stick around

104:32

And this is much better for me to

104:34

give feedback to Claude on

104:35

than it would be to just try to

104:37

infer from a markdown file what the thing

104:38

is going to look like

104:39

I could go in and

104:40

like I said, take screenshots

104:41

feed them back into Claude

104:42

Who here

104:43

regularly takes screenshots to give information to Claude?

104:46

Very good

104:47

You should be doing that

104:48

That's very good

104:49

especially when you're doing front-end

104:51

it's really hard to articulate

104:52

like, The thing is slightly off

104:55

or there's a misalignment here

104:56

and it's like

104:57

you'll find that you

104:58

run into the limits of what you can express

105:01

and it's actually easier for

105:02

especially Office 4.7

105:03

which has a much better vision model than before

105:06

to extract from you what the problem is

105:08

and proactively

105:12

Great

105:13

So

105:14

a few examples. Tarik has a lot more in his repo

105:17

which you'll find in the repo as well

105:18

online

105:19

So, so far

105:23

we've covered

105:25

letting Claude extract information from you interactively as an interviewer

105:30

because

105:32

the longer you let a

105:33

an agent run

105:35

the more important it is that the spec is comprehensive

105:37

the less likely it is that

105:39

you will be able to upfront to find everything

105:42

and it's better

105:43

for you to iterate with Claude in that manner

105:48

Then

105:48

what's a better and

105:49

more efficient and ergonomic form factor?

105:53

It would be the HTML file

105:56

And now

105:59

the more important part from all of this

106:01

is

106:01

how to verify what cloud is done

106:03

and how to make it agent-related

106:05

And that's what we're going to cover with

106:06

the repo. I

106:10

think we have

106:13

I think there'll be one slide where you'll

106:15

get to see the actual slides to

106:16

to engage with

106:17

the actual URL to engage with

106:19

So what you want to do is

106:23

make it part of the artifact

106:25

And that's what we're going to cover today

106:26

In fact

106:26

there's part of what we're doing today is we're going to

106:29

we're going to use

106:31

storybook fixtures, a testing library, sort of data emissions

106:36

and attributes, and playwright MCP for a React

106:41

for a React app

106:44

with

106:44

with components that you will have seen before

106:48

But remixed in a way

106:49

to make it easy for Claude

106:51

as an agent to

106:54

extract the data contracts in the DOM and then run verification

106:58

and then even record the verification

107:00

which is how the Claude code team

107:02

that Tarek is on runs this

107:04

and that's what the demo that he built refers to

107:07

right? Like

107:07

it'll be a generation of verification steps that get run through

107:11

they get a recording

107:13

and that's

107:13

then put on S3 or shared somewhere else with colleagues

107:16

and that's how

107:17

you turn these verification steps

107:20

into something that is generated by the agents

107:23

and then that you

107:25

have fewer and fewer touchpoints with

107:27

That's something that you can replicate

107:28

especially from that repo

107:31

The way to do that is to sort of sensibly modularize

107:34

but that becomes really clear when we look at the actual

107:37

the actual thing

107:39

So I encourage you

107:42

to go to that link

107:44

There'll be a repo

107:47

This is the repo

107:50

under CWC workshops, Cloud with Code workshops

107:54

and there's the one for today's session

107:55

which is how we cloud code

107:58

What's covered in there is phase one

108:00

which we just covered

108:01

right, the simple prompt to say

108:02

hey, I'm writing a bill splitting app

108:05

interrogate me

108:06

on the requirements

108:07

so that I don't have to

108:09

miss any in my upfront description to cloud

108:12

The second one is

108:14

generating four different HTML design directions that we could explore

108:18

for me to then feedback on or for

108:20

for me to decide

108:21

And then thirdly, we have a

108:22

a verification framework here

108:25

and it's in a different context

108:27

It's not to do with the

108:29

bill splitting app. It's regarding a to-do app

108:32

a small to-do app

108:34

written in React

108:36

And

108:38

we're going to show how we built verification

108:40

entirely into that along the way

108:44

That's described in detail

108:46

In the README for phase three

108:48

and then also in the verification detail as well

108:50

which is here. So

108:52

if you want a deep dive

108:54

on how this is set up

108:56

you can read this

108:59

and it's all provided in the repo

109:01

It's pretty cool

109:02

What is it focused on?

109:04

We've written a, a small to-do list

109:08

app here

109:12

I can add an item

109:14

test

109:17

It's hard to see. There we go

109:18

Very good

109:19

And then I could tick it off

109:20

and drop it as well

109:22

And I could clear the finished items

109:25

And so

109:26

many things are happening here in terms of the state

109:28

and we'd like to verify

109:30

that it's all working

109:31

alongside the tests that are already defined

109:33

but these verification steps

109:35

we'd like them to be driven by an

109:36

agent. There are three ways that we're going

109:37

to do this. So we're going to do

109:38

this in a human readable way

109:41

but then that same way of doing it

109:42

in a human readable dashboard

109:43

we're going to verify

109:46

in an agent first way

109:47

and let Claw do that separately

109:49

And then there's a separate way in the

109:51

in the repo as well

109:52

You can just run

109:54

run, run verify

109:55

and then it'll just run the test matrix

109:57

there. You'll get a slightly different

109:58

result if you do it from the repo versus here

110:00

but, the, the principle is the same

110:02

So human readable, agent driven, if you're doing it from

110:07

from Podcode or somewhere else

110:08

and then also just generally headless

110:10

like you might do it in

110:11

in CI

110:13

We'll have a look at it in a

110:13

bit more depth

110:14

We'll go here into the app

110:17

and we'll actually look at

110:23

There we go. We'll look at elements

110:27

We have here

110:28

The actual emitting, so the, the emitting different data

110:32

data aspects, the data verify unit

110:34

the total done and active

110:36

the component itself here

110:38

is publishing its state to the DOM

110:42

and so if I change my

110:43

if I change my state here

110:44

if I may, I might add test again

110:47

see that updates, and I drop it

110:49

it updates again

110:52

So this is what the agent can read later

110:56

as opposed to having to scrape the DOM

110:58

We can just

110:58

if you publish the state here separately from the React internals

111:02

you'd be able to run the verification independently of what

111:05

whatever the state of the app is

111:07

and that's

111:08

how this is set up to work

111:10

and then you can run this

111:11

further down the line as well

111:16

Each one of them gets this

111:17

right? If you look at the

111:18

if I, if I pull up the

111:24

That's a bit hard to read

111:25

let's do that here

111:26

every, every component gets one of these

111:28

right? There'll be schemas

111:30

there'll be fixtures, the known states, and then the invariants

111:34

there are particular ones here that

111:36

that always have to roll

111:37

always have to hold

111:38

you'll test them with probes

111:40

and we'll do one example

111:44

which we've hard-coded

111:45

whereas we've hard-coded an example that will fail the verification

111:48

and we'll let the human-verified dashboard catch it

111:52

but then we'll also let the

111:53

agentic way of doing it cache it as well

111:55

and we can let Cloud Code find it

111:57

and diagnose it for us

111:59

So that same approach also works here

112:03

So I've got the dashboard defined here

112:07

and what we see here is

112:10

like I said, we've got the different schemas

112:12

the invariants that we've defined

112:14

and we can run these

112:15

right? Individually

112:16

we can see how they would execute here

112:19

Got an example here

112:22

Or also, I could run all of them

112:25

In fact, I'm going to do that now

112:27

I run them all

112:28

One of them triggers artificially

112:30

because we planted it before

112:33

I'm going to scroll down and find out

112:34

what that is

112:36

And it's here

112:38

we will actually look at how

112:42

we can replicate this

112:44

for an agent as well

112:46

And we would do that like this

112:47

Go here

112:52

Select

112:58

And we could run

113:01

basically the manifest of all the different verification

113:04

steps that are defined here as states in

113:06

the DOM

113:07

We can get them in the same way

113:08

that we have them here

113:10

Expand that. It's getting a little bit small

113:18

As a new level like that

113:22

And then like that as well

113:27

Great

113:33

If we do that

113:34

then we can also actually run them

113:38

just specifically here

113:41

or do it manually first

113:43

and we reply, replay them all

113:47

What's that?

113:59

In this case, we're running this, not to perform the verification

114:02

but to provide the evidence of the verification

114:05

we can later on record this

114:07

we can record these as clips

114:08

which would just be videos that we capture

114:11

and then we would

114:14

Store them, share them with colleagues

114:15

put them on history or whatever it might

114:16

be

114:19

Let me pause that here

114:26

There we go. So here's our summary

114:28

We have one that's deliberately failed

114:30

I'll explain that in a second

114:32

But in each case

114:36

you can also see

114:38

the details of how this was done

114:41

The key thing here is that we want to

114:45

it's important to include the probes

114:46

to, to push off the happy path

114:48

And then

114:50

a lot of this will be generated by Claude for Claude

114:52

right? So there's

114:54

there's a way of scaling this further

114:56

in the end

114:57

we'll go to the one that didn't work

114:59

It's the one where we hard-coded

115:01

that the sums don't match

115:03

In this case, the, the stake doesn't match

115:07

three plus

115:10

three plus four does not equal ten

115:13

In this case

115:15

and

115:16

we'll get the same result if we run

115:24

verify all from

115:25

from the plot here as well

115:27

And we'll let Claude do that in a

115:28

moment as well

115:31

There we go

115:45

And it's a little bit hard to see if there's pass

115:48

pass, pass, pass, pass, and fail

115:51

So there we go. Okay

115:53

good

116:00

We can change this

116:00

right? I can, the, the point here is that I'm

116:03

I'm trying to show that the

116:05

the idea is to let

116:08

something agent native be read as

116:10

the DOM contract here

116:12

that's what we established earlier

116:13

that the state is managed here

116:15

or will not manage

116:15

but at least, viewable to an agent

116:18

that it can then run the verification end-to-end

116:20

itself

116:21

And we can let Claude do that

116:23

afterwards as well

116:26

In this case

116:27

we can also make a change

116:29

that will break more

116:29

Let me see if I can break the chain

116:32

or break the contract

116:33

but not the app

116:34

and I'll do that here

116:39

So I could change

116:40

for example, here

116:46

under, is that good? Under schedule apps

116:49

total stats, I'll delete that

116:51

I'll do, undo that afterwards as well

116:56

And if we now rerun

117:16

Then all of these ones at the bottom

117:17

here will fail

117:20

And we'll get the same

117:21

when we run this

117:22

again from here as well

117:39

There we go. All of these are failing

117:41

not because we broke the app

117:42

but because we broke the contract

117:44

That call can then natively verify

117:50

Excellent. Great

118:04

I want Claude to tell me

118:06

what's going on with these ones

118:08

at least about, not the ones that I broke

118:10

but I'll undo the ones that I broke

118:11

but I'd like it to tell me what's

118:12

going on with this one

118:14

and so let me correct

118:17

the change that I made before

118:20

Control Z, auto save, rerun

118:27

and there we go

118:38

So we've done it manually

118:39

and we've shown how we can

118:40

match what the agent would see versus what we would do

118:43

but we can let Claude run this headlessly

118:45

itself as well

118:46

So I'll do that

118:47

and I'll pull that open

118:47

just let it run

118:51

This one's broken, deliberately

118:53

And the rest works

119:05

Great

119:06

So we have cloud running here

119:10

I mentioned fast mode before

119:11

I mentioned auto mode

119:12

which is great. I mentioned forward slash goal

119:16

In this case

119:16

what we're going to do

119:18

is let Opus 4.7

119:20

help us find out

119:22

why that particular verification failed

119:29

I've already connected the playwright MCP for this

119:32

or, and it's going to use that

119:34

and it's going to run

119:36

Ski bug got rejected

119:37

4 plus 3 does not equal 10

119:40

yes

119:41

If you were to run bun verify

119:44

run bun verify

119:45

then you would actually pass the tests in this case

119:49

because

119:51

We deliberately put the verification in wrong

119:54

just to demonstrate that then the

119:55

the test matrix itself will actually pass

119:58

but the idea here is to separate out

120:00

what you could do as a human

120:02

versus then what you could do

120:04

via an agent directly from the browser using basically these

120:08

the, the commands that you saw

120:10

and then also what you could run headlessly

120:12

directly from the CLI

120:15

If you wanted to do it

120:17

like this as well

120:20

then you could record the outcome

120:22

that we talked about

120:26

Yes

120:26

So here, I've, in this particular setup

120:30

you've actually got the ability to

120:31

to show how you would record these

120:33

the same delayed running that we've shown

120:37

you could run and record as evidence

120:41

basically to show that it would work

120:43

and that you could store that

120:45

they could run like that

120:56

This is very common right now

120:59

the, the plot code team

121:02

records basically all the code changes that they do like this

121:05

all the front-end changes at least

121:07

especially on the run

121:08

the

121:09

at the pace of the shipping that we

121:10

have at the moment

121:13

Are people able to pull up the

121:14

the repo and get the verification set up working?

121:16

Yeah, very good, very good

121:27

Yes, you can store them

121:28

I mean, like

121:28

you can just put them in S3 or whatever

121:30

or share them with colleagues

121:31

or, in our case, we have a version

121:33

we have a, we have a internal

121:35

this is more of an automated than what

121:37

I'm showing here

121:38

in our case, we do record them

121:39

not certain for how long or what context

121:42

but, it's part of a regular cadence

121:47

And then you get the

121:48

in fact, you get the

121:51

I should have triggered that

121:53

I don't see that. There we go

121:55

So you can, you have each clip

121:57

you could download all of them or download them individually

121:59

and then that's basically the bundle that

122:02

that, that proves the verification worked

122:07

so I guess to

122:12

bring it around to what we've covered so far and

122:15

and where it's going

122:19

Three different surfaces, the, the human surface, like the

122:24

And then the, the agent first from the browser

122:27

mapping on the same way

122:29

And then

122:30

you can also run it in CI with just run

122:32

done, verify

122:34

The

122:35

objective here at the end is to figure

122:36

out how do you embed

122:40

the verification into the artifact itself

122:45

There's more detail

122:46

I encourage you to check out the repo itself

122:49

and actually run a few examples yourself and

122:51

run a few tests where you can change the code

122:54

see what breaks, rerun it yourselves

122:58

it's quite detailed, and it's what Tarek and

123:00

and team use in the Cloud Code team

123:03

for their work already. This is a

123:05

this is, this came, pretty quickly

123:08

just

123:09

a week and a half ago

123:10

into this demo

123:12

so I encourage you to check this out

123:14

What's new really is the remixing and the

123:17

the new arrangement of

123:18

of compo, of, of Or primitives that you're already familiar with

123:23

that you're already using

123:23

just to make it available to the agent first

123:26

That concludes, really. Everyone

123:30

thank you for being

123:31

for being here today

123:32

I'm pretty much excited to have

123:34

this session with you

123:35

I know we are almost at the end of the events

123:38

so thank you for bear with us

123:40

So just to start

123:41

let me introduce myself

123:43

I'm, Ivan Nardini. I'm a developer advocate at Google Cloud

123:46

working on building content in partnership with Anthropic

123:50

And, in order to start this presentation today

123:52

I just want to ask a very simple

123:55

a very simple question

123:57

So

123:57

how many of you in the last week

123:59

use any AI tool

124:01

to code or build application?

124:04

Okay, the majority

124:06

how many of you

124:07

use the same AI tool to build

124:09

and deploy application on Google Cloud?

124:13

Yeah, just a few of them

124:14

So the

124:15

the goal here today is just try to

124:17

make it better. So in this

124:19

in this demo, what I'm gonna show you is

124:22

how you can use Cloud on Google Cloud to build and

124:26

deploy application and to end

124:29

And, in order to do that

124:30

I'm gonna wear five different

124:33

hats

124:34

And, we will start from

124:37

imagine the use case

124:38

right? Imagine the scenario

124:40

You are an enterprise context

124:42

In an enterprise context

124:43

you probably have a team

124:45

like the one that we are

124:46

visualizing here that

124:48

it's engaged to build a new feature

124:51

or a new product

124:52

So starting from the left

124:54

you probably have a PM

124:56

some of you in this room are PMs

124:58

and

124:59

the PM might have an idea

125:01

of how to improve

125:02

a particular product or how to implement a feature

125:05

and starting from this idea

125:07

he shared this idea with a UI UX design

125:11

which allows him to design the idea

125:13

to visualize the idea

125:15

And, after the idea start getting a shape

125:19

then

125:20

the idea

125:21

is sent to

125:22

a software engineer that essentially start developing the core logic

125:26

behind this idea in order to ship

125:28

and make the application accessible or the feature

125:31

the future, feature accessible to everyone

125:35

And, but before to ship it

125:37

in order to be confident on what you are releasing

125:40

you probably have to pass

125:42

a security review. So that's why you have a

125:44

a

125:44

a rule like the

125:46

security engineer

125:47

that will allows you to be confident

125:49

in the release

125:50

And finally, once the application gets

125:53

released, you probably have a data persona

125:58

like a grow marketer or a data analyst that analyze

126:02

the data that are collect through the app

126:04

in order to generate insight and produce feedback to the PM

126:09

in a way that the product or the feature itself gets

126:13

improved

126:14

So these are kind of

126:16

you know

126:16

all the personas that

126:18

in a simple way we can imagine are involved in

126:21

the software development lifecycle

126:23

Now, with respect to these personas

126:26

Cloud Code, the Anthropics coding agent

126:29

augment all of them

126:31

providing several components that we are gonna use

126:35

today

126:36

So as I said

126:37

in the, for the remaining part of this presentation

126:40

I'm gonna put the head of all these personas

126:42

and gonna show you how you can use these

126:45

Cloud core components together with

126:47

Cloud on Google Cloud in order to ship

126:51

build and ship a very simple

126:53

feedback application

126:54

that

126:55

at the end of the presentation we are

126:56

gonna use to rate

126:58

my, my performance here

127:01

But Before to start building

127:04

of course you need to

127:06

set up and the

127:08

code that we are gonna use

127:10

And

127:11

I'm so

127:12

excited and so proud that we work

127:14

together with Anthropic to make this process of

127:16

setting up

127:17

code in order to use

127:19

model on Google Cloud in a very simple way

127:22

in a very straightforward way

127:24

So, you have multiple methods to use

127:27

code models on Google Cloud in code

127:30

but the simplest one

127:31

the faster one, is using, the application default credential

127:35

which, automatically finds, finds, your credential, for example, the user one

127:41

and based on the environment that you are gonna use

127:44

And, as you can see in this representation

127:46

so recently, code also introduced this

127:49

wizard that will simply allows you to

127:52

detect your project and your region where the models are served

127:57

and, check which models are available in your project

128:01

and, like, let

128:03

you to pin them in order to start

128:06

building your application

128:07

At this point, like, probably you're familiar with this

128:11

and you're wondering, okay, but what's it

128:13

what is different to use just cloud code with

128:16

with, cloud models while using cloud on GCP

128:20

on Google Cloud?

128:21

So there are many reason

128:23

why you want to do

128:24

why you want to do this

128:25

So first of all

128:27

because you pay for what you use

128:29

So the con, the usage of

128:31

cloud models on Google Cloud is per token

128:33

so you don't receive a message

128:36

message cap, and also, if you're building

128:40

enterprise application that needs to go

128:43

to production, you can always

128:45

access to what is called provision in throughput

128:48

which essentially will reserve some

128:51

throughput for you

128:52

in order to build this kind of application

128:55

the other important reason why you want to consider

128:58

like, Cloud on Google Cloud is

129:01

as I said, the setup is pretty straightforward using the

129:04

the ADC

129:05

You don't have API to rotate or

129:08

you know

129:09

environment variable to set in some sense

129:12

So it's a, it's a happy journey in

129:16

in, with respect to this aspect

129:18

you can access model in your project

129:21

with your own

129:23

you know, policies, set, and

129:26

also like the data stays

129:27

in your project while you are interacting with

129:29

the clock code

129:31

And model are served

129:33

in multiple region, so you have global endpoint

129:36

you have a regional endpoint

129:37

depending on, you know, the availability that you

129:39

that you need

129:40

And as a Google Cloud

129:41

talking about availability, we have very great

129:45

availability service that, standards that will allows you to

129:50

use Cloud

129:52

and

129:52

on one of the most performing infrastructure that you can find

129:56

in the, in the market

129:57

So these are all

129:59

some of the main reason why you want

130:01

to consider Cloud on a Google Cloud

130:03

especially in an enterprise context

130:05

So now that you have

130:06

like few reasons of why using Cloud on Google Cloud

130:11

we are ready to

130:12

build

130:13

And so as I said

130:14

I will start wearing the head of a

130:15

PM

130:17

So I imagine that you just joined

130:19

the company, or maybe you're already part of the company

130:22

you have, you, you want to improve our services

130:24

you want to implement a new

130:26

features, with respect to a particular product

130:29

What was happening in the past is that

130:32

you have the idea

130:33

you go to a UI UX designer

130:35

and, you ask him to prototype and visualize the idea

130:39

Now, with the, with the cloud and cloud code

130:43

all you, you, what you can do is just

130:46

drawing a picture like the one that you see

130:48

here, while you're drinking a coffee

130:50

maybe in, San Francisco, and, then let, cloud doing

130:56

implementing the idea for you

130:58

So let's see this

131:00

in action

131:04

So this is, the Cocoa UI

131:08

like you will probably familiar with that

131:10

and you are familiar with the CloudMD

131:11

which is essentially give some instruction

131:14

Here we just say

131:16

that we are a PM

131:17

we want us, we want to have a

131:19

starting from the picture

131:20

we want to render

131:23

a prototype of, the app

131:26

the Y frame

131:27

that we are gonna then

131:28

use and pass to the UX designer

131:31

and in few minutes

131:32

you can see how Cloud was capable of rendering it

131:36

and, just starting from a very simple

131:40

picture or drawing, that you, you

131:43

did while you were drinking your coffee

131:45

So, pretty, pretty straightforward

131:48

But imagine, how much time you save in doing this

131:51

because

131:52

compared to what you were doing in the

131:53

past with the back and forth

131:56

to

131:56

in order to get this first prototype of your idea

132:01

Okay, so at this point

132:04

the PM gives, like, creates a prototypes and

132:08

pass these prototypes to the UI/UX

132:11

developer, and, at this point

132:14

it needs to implement a more solid

132:16

interface in order to use it in

132:19

in production

132:20

So in, in this particular use case

132:23

what we want to create

132:24

it's at least three

132:26

like, pages, from the landing to the thanking

132:30

message, like message page, and a dashboard that

132:35

will allow me

132:36

to show you in real time

132:38

what can be the feedback that I will

132:39

receive from the room

132:41

So in this case

132:43

There are many ways you can

132:44

you can implement this

132:46

but in this case

132:47

I want to use an additional capabilities of the code

132:50

which is, the plan mode

132:52

So with the plan mode

132:53

what we do

132:54

we put Claude in a mode where it thinks

132:57

before to

132:59

like it thinks and propose

133:00

what it's gonna do before to implement

133:03

any code

133:04

And this is very important because it gives me

133:06

like

133:07

a degree of freedom

133:08

of deciding to change something

133:10

based according to my preference or according to

133:13

some standard

133:14

that probably I will get access through an MCP server using

133:18

Figma, for example

133:20

So now that we have

133:22

in mind what we are gonna build

133:23

let's see this in action as well

133:25

So we started from

133:27

the wireframe from the PM

133:30

similar prompt, I enabled the plan

133:32

the plan mode, and so as you can see

133:35

compared to before, in this case, I'm simulating the, receiving some instruction from Figma using

133:42

design doc, but as you can see

133:44

compared to before

133:45

it creates a plan

133:46

of what it's gonna do with respect to

133:48

all the components that are defined in the

133:50

slide

133:51

We look at them

133:52

we are happy, we accept

133:55

and Code will implement all of them

133:58

And at the end

133:59

what we get is

134:01

this, optimized version. So as you can see

134:04

we start from here

134:05

and we get this

134:08

Very, very straightforward

134:10

but you can see how

134:12

we are

134:13

shifting from a prototype to something that can be used

134:17

in this session in a very simple way

134:22

Okay, so this is the part that probably every view

134:25

or every, or view in this room

134:27

like you do every day

134:28

right?

134:30

let's, let's wear the third hat

134:32

which is the one of the software engineer

134:35

And, the software engineer, you receive this, front end, like

134:41

all the components that I was sharing you before

134:44

and, maybe it doesn't know anything

134:47

as probably, some of you in this room

134:50

it doesn't know how to deploy this application on Google Cloud

134:54

right?

134:55

So

134:57

how you can, how you can do that

135:01

how you can, you know

135:03

and to have

135:04

there's a clear picture of what are the

135:06

components

135:07

on Google Cloud

135:08

that you need to use in order to

135:10

deploy a very simple application like the one that

135:12

I show you today

135:14

Luckily, it's not a problem because as a Google Cloud

135:18

we invest a lot of time to integrate with this

135:21

large, by coding ecosystem that now is

135:26

growing around, models

135:29

And, we have, we, in the last few months

135:32

we released two, important components

135:36

So the first one is

135:38

the Developer Knowledge API with this

135:40

with, the associated MCP server

135:42

And the second one is the

135:44

Google Cloud Skills

135:45

So with the knowledge

135:47

with the developer knowledge API

135:49

you get access to the

135:51

a fresh

135:52

documentation from Google Cloud that can be directly consuming Cloud Code

135:57

through the MCP server

135:59

and, it will have Cloud Code to figure it out what

136:02

what is the best

136:03

architecture, what is the best implementation

136:06

to deploy a certain application on Google Cloud

136:09

This is very important because

136:10

again, what

136:11

what we are saying here is that

136:13

you don't need to know

136:14

like how to deploy the application on Google Cloud

136:17

you can just leverage

136:18

Cloud Code and this MCP server that we expose now on

136:22

on Google Cloud

136:23

side to build application like this one

136:26

So in this case

136:28

probably what we wanna

136:29

we are gonna do is that we are

136:30

gonna deploy

136:31

the feedback API on a serverless function like a Cloud Run

136:35

where we connect with a web

136:37

web

136:38

oriented DB, like a Firestore, to correlate

136:41

to collect the raw responses that we

136:44

we will give through the feedback Backup

136:46

And then, because we want to have that

136:48

that analytics part, we will, build

136:52

implementation in a way that we can store

136:54

those raw response in an analytical data warehouse like BigQuery

136:59

and we, in BigQuery

137:01

we will post-process and will consume this information in a dashboard

137:07

in a, in a

137:08

in a dashboard like the one that you can find in

137:11

in Looker

137:12

But again, it, you, you can build this using

137:18

a code in combination with the MCP server without no

137:21

without you having a prior knowledge

137:23

of how to do that

137:26

This, like the MCP server and the

137:28

developer knowledge API that I just mentioned

137:30

it got paired with also the skill part

137:33

And the skill part is

137:35

if with the, MCP server

137:37

you will be able to design the architecture

137:39

with the skills

137:40

you will be able to cover the single

137:42

blocks of this architecture. For example

137:44

we release a, a sim, a simple skills that

137:48

enable Cloud to deploy

137:50

on Cloud Run, to deploy an API on Cloud Run

137:53

or, to connect, like, Cloud Run with Firestore

137:58

So it's more about

138:00

the implementation itself rather than

138:03

you know, giving, the overall picture

138:07

So once you get Cloud Code enabled with

138:11

the MCP server, with

138:14

the documentation and the skills that I just mentioned

138:17

you can just directly implement

138:19

the Architecture that I was showing you before

138:22

and you can do this in parallel

138:24

So

138:25

another components that you can use in CodeCode

138:27

is a sub-agents

138:29

And as you can see here

138:30

we are gonna spin up three different sub-agents

138:33

one for the API

138:34

one for the ingestion pipeline

138:36

and the other one for the dashboard

138:38

and you can parallelize the implementation of each of these components

138:42

just like, you run a team

138:44

a team sprint, in the, in your normal, like, usual development, lifecycle

138:51

So let's see this in action as well

138:56

So here we are again

138:57

first of all

138:58

I just want to show you that we

139:00

have enabled the MCP server of the documentation

139:03

and we have the skills

139:04

some of the skills that we pre-built

139:07

One, one time more

139:09

I provide a very simple prompt

139:11

The first step is designing the

139:13

cloud native backend

139:14

So it will, start, like, it will provide me

139:20

draft of the architecture. In this case

139:22

I could use, again, the plan mode, but for simplicity, I didn't

139:26

And then you use the skill

139:27

one of the skill that I provide in

139:29

order to implement the API

139:31

Let's say that we are

139:32

happy with the API spec

139:34

then we have the architecture

139:37

we have the API spec

139:38

the next step is running

139:41

multiple, agents in parallel in order to implement the three

139:45

the three components of the app

139:47

that I was showing you

139:49

So it's, it's pretty quick

139:51

As you notice, Cocode, like

139:54

it also managed the testing part

139:56

one after it finished the implementation

139:58

and at the end

139:59

you will get your

140:01

your app, which is now, ready to

140:05

deploy on Google Cloud

140:10

Okay

140:11

So at this point

140:14

we have, we have the code of the app

140:17

that is, ready to be deployed

140:19

but because we are deploying on

140:20

on cloud and we want to open this

140:22

up to a larger audience

140:24

we want to deploy it in our confid- confidently

140:27

like, so this is

140:30

this is when you want to consider

140:32

to run a security review

140:34

Now, depending on, your company, you can

140:38

you can have different security requirements

140:40

So, for example, you may want to check if your

140:44

application is solid with respect to the most

140:47

common OWASP issue

140:52

or because you are deploying this application on cloud

140:55

and one

140:56

one thing that you need to consider is

140:58

probably you will use what is called a service account

141:01

and you want to limit

141:04

the, the service account when it calls a particular API

141:07

like the one related to reading and writing ADB

141:11

with respect to certain role

141:12

So you are, you, you're sure that you

141:15

you are limiting what it can happen when

141:18

when the application runs some operation on

141:20

on the cloud itself

141:22

So, again, these are just a couple of examples of

141:27

what you want to consider in a

141:29

phase like this one. And of course

141:31

this representation is a

141:33

strong simplification of what can happen in the

141:36

in the real life

141:38

it's just, one of the possible scenario

141:40

that you can have

141:41

And you can see why it's also a simplification

141:44

because

141:44

we are letting the security engineer

141:47

not only to ap- approve if the app is

141:49

secure enough to be deployed

141:51

but it will also deploy the app

141:53

in this case

141:54

But again, this is just a demo

141:56

so we have this degree of freedom

141:59

With that being said

142:00

let's run the final demo

142:01

and let's, get the app, running

142:05

So in this case

142:07

as you can see

142:07

I use a pre-built security review that you can find in

142:10

Cockroach

142:12

si- very simple, very simple prompt also in

142:15

in this case

142:17

And, what is happening is

142:18

Cockroach run the first test

142:20

essentially

142:22

double check, That, everything is aligned, it found, like, a possible issue, and

142:28

it automatically fix it

142:30

So in a way that the app now is secure

142:33

and once it is secure

142:34

it deploy the back

142:35

the, the backend API

142:37

and it deploy the API itself

142:40

So what we get at the end of this

142:42

it's an endpoint with our app

142:44

The app is live

142:45

so I will, quickly unlock my laptop

142:51

and I will ask

142:54

the backstage to share on my laptop

143:00

So the, the app, as you can see

143:02

is, up and running on GCP

143:04

for people that doesn't know GCP

143:06

this is Cloud Run. It's a serverless

143:07

ser

143:09

service that you can use in order to deploy app

143:11

And

143:13

the app

143:14

at the end looks like this one

143:16

So if you remember

143:17

I show you the original

143:20

I show you the feedback

143:23

frame at the beginning

143:24

So what we can do now live

143:27

I can just give me a score

143:28

What do you think the session is going

143:30

so far?

143:31

Give me five? Oh

143:34

okay. Thank you, man

143:35

I really appreciate that

143:37

Okay, cool session. Let's be

143:40

let's be simple

143:41

I submit, and then, in real time

143:44

it updates the number of response

143:46

the score

143:47

and, you know, the visualization, and also, just for fun

143:51

I build a feedback analyzer

143:53

So once I click this

143:55

it will, run, it will call, Cloud, Cloud on Google Cloud

144:00

and, based on the feedback and the comments that you send

144:03

it will generate this

144:05

this summary

144:06

So pretty, pretty straightforward

144:12

I will last to go back on the

144:14

presentation

144:15

Thank you

144:17

Okay

144:18

So at this point

144:19

we have the app

144:20

we got, we got, good feedback

144:22

Thank you again, man

144:24

but the development lifecycle is still there

144:28

we have the last step to cover

144:30

which is essentially people there start

144:33

using our app

144:34

If you saw one of the KPI that

144:36

I had on the dashboard was the time

144:38

the response time, so how long is

144:41

was taking you to just in

144:44

like providing a feedback

144:46

So these are the kind of information that

144:48

you can use in order

144:50

to, like, through this data, you can collect this data

144:54

analyze them, and generate

144:56

insights in a way that

144:58

they can be used in order to improve

145:00

the, application

145:02

Now, Running, if you're new on

145:05

Google Cloud

145:06

there are several services that you can use

145:08

in order to analyze this data that comes

145:10

from the app

145:11

So a couple of them

145:13

one, as I said at the beginning

145:15

is an analytical data warehouse

145:16

which is BigQuery, and, for the reporting part

145:20

you have a tool like Looker

145:22

But again, as we said before

145:25

you don't need to know how to use

145:26

BigQuery in order to analyze those data

145:29

as well as how to build a dashboard in Looker

145:32

because we also provide a

145:34

MCP server for doing this

145:36

Now, for the sake of time

145:38

I'm not gonna demo how to

145:39

use an MCP server to query BigQuery or building a

145:43

a dashboard, but, I want to quickly share with you

145:48

where you can find

145:49

this information in order to do that after this session

145:52

because we are gonna release the code

145:54

So I will ask

145:55

to shift back on my laptop

145:58

Thank you

145:59

So

146:03

Okay

146:04

Loading time

146:05

So

146:06

with respect to all the MCP server that are available on

146:09

on Google Cloud, we, recently announced the

146:13

agent platform

146:14

So in the agent platform

146:16

you have, services that, is represented by the agent registry

146:20

and in the agent registry

146:22

you have the list

146:23

of all the MCP server that we natively

146:26

support on Google Cloud

146:27

So for example

146:28

we have the developer knowledge service that I just show you

146:32

and we also have the BigQuery

146:35

the BigQuery MCP server that you can use in order to

146:41

Query

146:42

the data that we just collected from the app

146:45

it's very, like

146:46

this register is relevant because it tells you how to set

146:50

how to set the MCP server on your side

146:53

as well as it gives you some observability

146:56

feature and

146:57

the description of all the tools that you will find

147:00

for the MCP server. So you know

147:02

how Clockwork will be able to use

147:05

this, this, this server to, query your data

147:09

With respect to the

147:12

Looker part, also, like

147:16

we released the MCP toolbox of DB

147:19

This is a open source

147:21

like a model context protocol server

147:23

which include an integration with Looker

147:26

and, it's very well, We have a very

147:30

very

147:30

very well-documented quick start

147:32

on how to set up

147:33

with the

147:34

clock code

147:35

and start using it in order to consume

147:37

that data from BigQuery and build your dashboard

147:40

So I leave you this as an exercise

147:42

like, we, we are going to release the code

147:45

so you can go home

147:46

and integrate these two parts

147:47

It's pretty straightforward, but the dashboard that you

147:51

you can create, they are pretty, they are pretty powerful

147:53

and you will see

147:55

how nice they can be

147:57

Okay, back to the presentation

148:01

I think I'm just in time

148:02

so time to, time to wrap up

148:04

What I try to explain you today Is essentially

148:09

two things

148:10

So first of all

148:11

I was trying to

148:11

and I hope I did

148:13

I did good enough

148:14

I was trying to show you how

148:16

like, all the components of

148:18

Cloud Code, including skills, MCP server, sub-agents, they can really, like

148:24

speed up the process of software

148:26

development, as well as how you can use Cloud Code with

148:32

cloud models on GCP in a very seamless way

148:35

Like, if you saw, we run several session across

148:38

like, multiple, personas, and, like, it was, the experience was just

148:44

straightforward. It was just

148:46

incredible

148:47

So this is, what you can get if you combine

148:52

you know, Cloud Code with

148:53

cloud models on, on GCP

148:56

As I said, the read, the code

149:00

is gonna be available right after the session

149:02

We have a great quick start

149:04

and we have, a very well

149:06

maintained documentation, both on Google Cloud side and Anthropic side

149:10

so I highly recommend

149:12

to just go there and check out

149:13

And then I hope I cover everything

149:17

but

149:17

if you still have questions or you want

149:19

to provide additional feedback

149:21

just feel free to reach out

149:22

these are, Good. Okay, amazing

149:25

Welcome to the last workshop session of the

149:27

day

149:28

I hope you have all enjoyed the very

149:30

first day of Code with Claude

149:32

my name is Maya

149:33

I'm a member of our Applied AI team

149:35

here at Anthropic

149:37

what that means is I spend about half

149:38

my time developing our own first-party products and features

149:42

and the other half of my time helping

149:44

customers develop their very own products

149:47

features, agents on top of our models

149:50

Today I'm here to talk to you about

149:52

how to build

149:53

a proactive agent workflow with Claude Code

149:57

can I get a show of hands

149:58

who has used our routines feature

150:01

inside of Claude Code?

150:03

All right, some folks over here

150:04

Awesome, awesome

150:06

that's what I'm going to be talking about today

150:08

Okay, so first off

150:10

a question for the group here

150:12

Who

150:12

has tried to run Claude Code on a

150:15

cron?

150:16

Can I get a show of hands?

150:18

Folks, put your hands up high

150:19

Awesome, awesome

150:21

now keep your hands up if you've enjoyed

150:23

building all of that infra and maintaining that job

150:27

All right, we have one guy back there

150:29

We have one guy. Thank you

150:30

Thank you for your effort

150:30

Thank you for your work

150:32

we felt similar pain internally at

150:34

at Anthropic as we tried to develop

150:37

proactive agents, that run on Claude Code

150:41

and we decided to do something about it

150:45

So we believe that coding agents

150:47

shouldn't wait for you to press enter to

150:49

get started

150:51

Right now, Claude Code is a really powerful coding tool

150:55

but we want to take Claude Code and

150:57

turn it into a really powerful coding teammate

151:01

A teammate notices when something breaks

151:04

and does something about it

151:06

Right now

151:07

a tool waits for you to enter your

151:09

prompt and actually press enter

151:12

So

151:12

the goal of today's presentation is to talk

151:14

about

151:15

how we

151:16

have created this feature called routines

151:18

to take Claude Code from

151:21

a tool today into the teammate of tomorrow

151:27

So today we'll be talking about four things

151:30

I'll go

151:30

through some of the challenges that you folks

151:32

have felt building proactive agents today

151:36

I'll go through this new feature inside of

151:38

Claude Code called routines

151:41

We'll go through a real example about how

151:43

we use routines internally at Anthropic to automate

151:46

documentation creation

151:48

And then finally

151:49

we'll talk about applying routines to your own

151:51

workflows

151:55

So

151:56

I want to talk first through the challenges

151:59

with building proactive agents today

152:02

We all know it's doable

152:04

but I want to talk about

152:05

what's a little bit cumbersome with this

152:08

The first thing that's a little bit difficult

152:10

with building proactive agents today

152:12

is deciding where these agents should run

152:15

You probably don't want them running on your

152:17

local machine because if you close your laptop

152:19

or your laptop dies

152:21

your agent session is done

152:23

What that means is you'll need to manage things like hosting

152:27

data persistence, and authentication

152:30

Basically

152:31

you'll need to build a whole infrastructure outside of your prompts

152:35

which is doable

152:36

but

152:36

it's a lot of work and there's a

152:37

lot of boilerplate code there

152:40

The next thing you'll need to do

152:42

is figure out when to actually kick off

152:44

these sessions and trigger these agents

152:47

Again, you can build things and build on top of cron

152:51

or you can do things like

152:53

post to endpoints that you have to spin up

152:56

but again, there's

152:57

there's a lot of infra that you need

152:58

to build yourself here

153:01

Finally

153:01

the challenge with building proactive agents today

153:03

is sometimes you want to be a human in the loop

153:06

but other times you want to be a

153:07

human out of the loop with these agents

153:10

right now when you kick off a headless Claude code session

153:14

it's often hard to figure out what your

153:15

agent is actually doing in real time

153:18

There's no way to watch

153:20

steer, bound, or even resume your agent session

153:23

It's, it's difficult to do that

153:25

so we wanted to address each of these three issues

153:28

and build routines

153:30

Routines is a brand new feature

153:32

inside of Claude Code

153:34

It's an automation where you can kick off

153:37

a remote Cloud Code session by only defining the prompt

153:41

what repos you want to connect it to

153:43

what connectors it has available to work with

153:46

and a trigger

153:48

Cloud Code handles the rest

153:53

So there were three

153:54

kind of

153:55

main things we were thinking about

153:57

as we went ahead and developed

153:59

this routines feature inside of Cloud Code

154:03

The first thing is that we wanted these

154:04

agents to be always available

154:07

These agents

154:08

these routines run on Cloud Code's managed infrastructure

154:13

And

154:13

what's nice with that is that we deal with the hosting

154:16

the session state

154:18

and the connector all for you

154:20

Nothing depends on your laptop being opened

154:23

and we deal with all of the cloud stuff for you

154:26

which I think is quite nice

154:28

The next thing

154:29

is we want these agents to be able

154:31

to work proactively with customizable triggers

154:35

You might want to kick them off on a time-based schedule

154:38

or you might want to work

154:41

event-based

154:42

we have the ability to work natively with GitHub events

154:46

as well as your own custom events that

154:47

you can post to

154:49

webhooks and endpoints with the event payload as context

154:55

Finally, and the last point that I think is really nice

154:57

is that these Claude Code sessions that get

155:00

get launched with these routines

155:02

are interactive and steerable

155:04

as if you were launching Claude Code in

155:06

the terminal

155:08

Every routine is really just a Claude Code

155:10

session under the hood that you can open

155:12

you can watch, follow up on, steer, and resume

155:17

from web, CLI, and desktop

155:21

And so

155:22

I want to walk you guys through

155:24

a real use case that

155:26

we use here at Anthropic internally

155:28

So

155:28

the question for us and for a member

155:31

of our engineering team is

155:32

how can we automate docs creation with routines?

155:39

So just to add a little bit of data behind this

155:41

weekly PRs for Claude Code have gone up

155:44

200% since the beginning of the new year

155:48

This

155:48

has been super awesome for

155:50

our Claude Code engineering team

155:54

their productivity is insane

155:56

This has been really awesome for you folks

155:58

because you get new features inside of Claude Code very

156:00

very quickly

156:02

The one person that this has not been

156:04

so awesome for

156:06

is the one engineer responsible for maintaining our

156:08

documentation across Cloud Code and the Agent SDK

156:12

And so when routines launched

156:14

she was a super big fan and early

156:15

adopter

156:16

And I want to walk you through how

156:18

she set up

156:18

a couple routines to help

156:20

automate documentation creation

156:22

for Cloud Code and our Cloud Agent SDK

156:26

So on my side here

156:27

I have the terminal open

156:29

and I encourage you all to open your

156:31

terminal and launch Cloud Code

156:34

And inside of the terminal here

156:36

I'm able to type what we see on the screen here

156:39

slash schedule, and actually type in something that Sarah

156:43

our documentation queen

156:46

has done to actually set up this routine

156:48

So she went and typed in

156:50

once a week

156:51

please review all the new changes merged to

156:53

main against our documentation repo

156:56

and create a PR to update docs if

156:58

you see any changes

157:00

I encourage you folks to think right now

157:02

What are some tasks that you do every

157:04

day that would help if they could run on a schedule

157:07

or if Claude Code could actually initiate these

157:10

sessions for you?

157:11

I encourage you to think about that

157:14

After kicking this off inside of Claude Code

157:17

Claude comes back and prompts me with a

157:19

couple questions

157:20

It might ask, hey Maya

157:21

what time every week do you want me

157:23

to actually kick this off?

157:25

Or once I create a PR

157:27

do you want me to notify you in any way

157:28

maybe ping you on Slack?

157:30

And once I answer these questions

157:32

Claude actually goes ahead and creates a routine

157:35

that we'll view

157:36

in a little bit inside of Claude Code

157:38

on the web

157:42

But first

157:42

I want to walk through the three main

157:44

decisions you'll need to make

157:46

as you create any routine

157:47

The first decision

157:48

is you'll need to figure out

157:50

when your routine should trigger

157:53

What's actually the event

157:54

or

157:54

is there a certain cadence that you want

157:56

this to run?

157:58

The second decision is

158:00

what context or what information does Claude need

158:03

to have to actually be successful here?

158:06

Do you need access to certain docs

158:08

or does Claude need access to certain tools

158:11

to ping you?

158:13

Finally, the last thing to think about is

158:15

how do you actually steer Claude in the

158:16

session to keep it honest?

158:18

How do you guide

158:20

Claude to the output that you want?

158:22

And so we'll dive into each one of these

158:24

and I'll talk through how Sarah

158:26

a member of our team

158:28

actually does this to automate documentation creation

158:33

So the first one is the trigger

158:36

When should this event actually run?

158:39

So inside of routines

158:40

there's basically two ways to do this

158:43

You can have things kick off on a

158:45

schedule on a time-based trigger

158:48

For that earlier example I showed you

158:50

this is how we do

158:52

a weekly review

158:53

of differences between our source code for Claude Code

158:57

as well as our documentation repo

159:00

You can also have routines kick off

159:02

on an event-based cadence

159:04

So maybe every time a release is cut

159:07

you can diff the release branch

159:09

against the docs and see

159:10

if there's any new features that you'll need

159:12

to spin up PRs for in our documentation

159:14

repo

159:16

Or maybe your engineer is actually deploying

159:19

changes and creating PRs might tag their changes

159:21

maybe this is like a new feature

159:23

they could tag it with a label that says need docs

159:26

and you could actually kick off Claude Code

159:28

sessions anytime one of these

159:31

labeled PRs get merged

159:36

The next thing you'll need to think about

159:38

is context

159:40

What does your agent actually need to know

159:42

to be successful?

159:45

Likely

159:46

you'll need to give it access to

159:48

either one or more code-based repos

159:51

So for this docs example

159:53

we need to give Claude access to

159:55

not only our Claude code source code to

159:56

figure out what new changes exist there

159:59

but also our docs repo

160:00

right, for Claude to actually create

160:03

new PRs there

160:06

Next

160:06

you might want to provide additional context to

160:08

these sessions

160:10

maybe for this one example

160:12

I want Claude to have access to

160:14

all of our existing marketing briefs

160:16

Maybe I want Claude to use

160:18

similar language and verbiage that we use

160:21

in other marketing materials externally at Anthropic

160:24

So maybe all of this lives inside of Google Drive

160:27

and I'll want to give Claude access to

160:29

these files during the session

160:30

so I'll hook up the drive connector

160:33

Or maybe anytime Claude creates a PR

160:36

I'll actually want it to ping me on Slack

160:39

so I'll give it access to the Slack

160:40

connector

160:42

It's important to think about this

160:44

as you're setting up the routine

160:45

because whatever context Claude has

160:48

that's, the ceiling of how successful Claude will be

160:55

Finally

160:55

the last thing to think about here is

160:57

steerability

160:59

How do we actually ensure the quality

161:02

of Claude's outputs?

161:05

There are a couple ways to do this

161:08

one thing that I think is quite interesting

161:10

is to actually invest in agent-on-agent review

161:14

If folks have actually designed multi-agent systems here

161:17

and have heard of the generator

161:19

critiquer pattern

161:21

this is something that we borrowed here

161:23

You can actually set up

161:25

one routine to go ahead and create

161:27

docked PRs

161:28

and you can set up another routine that

161:30

maybe triggers on that PR's creation

161:33

to actually go ahead and leave comments on

161:35

the PR

161:36

before a human actually gets to it

161:40

Another option we had emphasized

161:42

it's nice to have a human out of the loop

161:44

but sometimes you actually do need to monitor

161:46

these Claude code sessions

161:47

and maybe nudge Claude in a different direction

161:50

What's really nice, and what I'll show you in a second

161:52

is that you can actually open Claude code on the web

161:55

and you can view what's happening inside of

161:57

a live session

161:58

as if you would working with Claude in

162:00

the terminal

162:01

You can ask it questions mid-session

162:03

you can push it in another direction

162:06

You can also resume a past routine or

162:09

a past session and continue the conversation

162:14

Finally, the last thing we do

162:15

and what I think is quite obvious

162:16

is that we verify Claude's outputs

162:19

For this documentation example

162:21

we actually render the page that

162:24

or the documentation page that Claude has changed and created

162:27

and we confirm those outputs are what we

162:28

expect

162:31

And so now I'll jump back into that

162:33

initial routine that we've kicked off here in this demo

162:37

and we'll jump to the demo slides

162:39

Awesome

162:40

And so we can see right now I'm in Claude.ai

162:44

and I can go over to this left

162:46

side panel and actually click on this code button

162:52

and I can jump into routines on this left-hand side

162:56

and

162:57

I can actually click on this routine that

162:59

I had created earlier

163:02

On the left-hand side here

163:04

you can see that it's connected to two repositories

163:07

our mocked-up Claude code source code

163:11

as well as our Claude code documentation

163:14

You can see that this runs every Monday at 10 a.m

163:17

and it's connected to

163:18

GitHub as well as Slack

163:22

These instructions here on the right-hand side

163:25

Claude generated for me

163:27

based on the initial prompt that I pasted

163:29

in and the questions that I answered

163:32

We can see here that

163:34

this is a, a weekly documentation sync

163:36

for our Claude code fork

163:40

And I can actually go ahead and click

163:43

on a session here

163:45

and I can see that these initial instructions are what

163:48

gets pasted in for the very beginning of

163:50

this Claude code session

163:53

We can see that Claude has read these

163:55

instructions

163:56

and started by looking at the source code repository to see

164:01

any changes, any recent PRs that have been merged

164:04

looked at our change log

164:05

and compared that to what's inside of the

164:07

documentation repo

164:09

We can see Claude's actually found some changes

164:11

here and gone ahead

164:13

and opened a PR for me

164:16

This is one example where you can kick

164:18

off a routine on a schedule

164:21

Now I want to show you another example

164:23

where we can actually kick off a routine

164:26

based on a GitHub event

164:29

So here I've actually created a new routine

164:32

and I've already filled some of this in

164:34

I want to make

164:35

another documentation automation, routine, and

164:40

but this time I want this to actually

164:41

trigger every time I create a new GitHub

164:43

issue

164:45

So I pasted in some instructions here

164:47

basically to investigate the issue that this session triggers on

164:52

figure out if it's related to a documentation gap

164:55

and then if it is

164:56

and if you believe that this is a gap

164:58

go ahead, open a PR

165:00

and actually ping me in this channel

165:02

So I've gone ahead and connected

165:04

our Cloud Code documentation repo as well as

165:06

our source code again

165:08

and I want to show you how I

165:09

actually set up this trigger

165:11

Like I had mentioned before

165:12

there's two different types of triggers

165:14

schedule-based and event-based

165:16

and within event-based triggers we have native GitHub events supported here

165:21

as well as the ability to

165:22

trigger from your own code by sending a

165:24

post request

165:26

So here I'll create a

165:29

GitHub event trigger

165:30

and trigger on issue opened

165:33

anytime I open an issue inside of this

165:35

Cloud Code documentation repo

165:37

and I want this connected to

165:39

Slack so I can send me a ping

165:40

anytime I make a PR

165:42

as well as our GitHub NCP

165:46

So I will go ahead and create this here

165:49

and now let's make sure that this is

165:51

working. So

165:52

I have a new

165:53

issue open here that I'm about to create

165:55

inside of our Cloud Code docs repo

165:58

so I happen to know that there are a few

166:01

few tools missing from docs in this new version

166:04

so I'm going to go ahead and actually

166:06

create this

166:09

So I can see here

166:11

that I've gone ahead and created that

166:13

and now let's refresh this page

166:16

And we can see actually that

166:18

a new run has gotten picked up here

166:21

We can see that these initial instructions are

166:23

the very first prompt

166:25

and we can see that

166:26

this additional context from this issue were passed

166:28

in as well

166:30

I happen to know that I actually already

166:32

have another PR open for this

166:34

so let me just guide Claude to

166:35

stop this session

166:38

I've already made these changes

166:47

And we can see here

166:49

the ability to actually steer Claude in real

166:51

time after a routine gets kicked off

166:55

Awesome

166:57

now let's come back and talk about the different ways

167:01

that we can use routines to actually automate your challenges as

167:05

developers

167:06

I want to talk through

167:09

a couple ways that we could turn common

167:11

software engineering or developer challenges

167:14

into routines with Claude Code

167:17

The first one I want to talk through

167:18

is this deploy verifier

167:22

maybe you have recently

167:25

just deployed changes to an

167:27

to a service

167:28

and you want to make sure that this

167:29

service is healthy and you shouldn't roll back

167:31

these changes

167:33

I want to think about this

167:35

in three ways

167:36

First, what should my trigger be here?

167:39

Second

167:40

what context is important to provide to Claude

167:42

in this routine?

167:43

And lastly

167:44

how do I plan to actually interact or

167:46

steer this routine to keep Claude honest?

167:50

I happen to know that my CD pipeline

167:52

can post after every deploy

167:54

so this seems like a pretty good trigger

167:56

to actually kick off this routine on

167:58

I can actually post to this

168:01

webhook that we support inside of routines

168:03

and this can actually kick off my Claude

168:05

code session

168:07

In terms of what context I think is

168:09

important to provide Claude here

168:11

I can think of a couple things

168:13

One

168:13

it's probably nice to give Claude access to

168:16

the source code for the service that we

168:18

recently deployed on

168:20

Next

168:21

it's probably important to give Claude access to

168:23

monitoring tools

168:24

Maybe that's Datadog, maybe that's Grafana

168:27

I don't know what you folks use

168:28

but

168:29

likely helpful to give Claude access to these

168:32

monitoring tools

168:34

And maybe if something goes down

168:36

I want Claude to alert me

168:38

I want Claude to ping me on Slack

168:39

or send me an email

168:41

or

168:41

maybe even send me

168:42

a text using Twilio or something like that

168:45

These are some of the connectors or the

168:47

tools that I would give Claude access to

168:49

inside of this session

168:51

In terms of keeping Claude honest or actually steering my session

168:56

maybe I'd start by having Claude run an

168:58

investigation for me and giving me an eventual no

169:02

go

169:02

or no-go decision to actually roll back this

169:04

change

169:05

I could jump into Claude Code on the

169:07

web and actually view

169:09

and read Claude's analysis for this session

169:12

Then maybe I can continue to work with

169:14

Claude if I actually think that this should

169:16

be rolled back

169:17

and actually

169:17

use Claude to help me roll back a change

169:21

Maybe eventually

169:22

as I watch Claude work more and more

169:24

and trust its decisions

169:26

I can let Claude roll back the change

169:27

itself if based on the monitoring

169:30

monitoring data it has access to

169:33

if I deem that that's the right decision

169:36

There are other challenges

169:37

like maybe you want to build an on-call investigator

169:40

or maybe you're actually a PM and

169:42

your job is to go through

169:45

A lot, a lot of issues inside of your backlog

169:47

maybe that's GitHub issues

169:49

maybe that's posts inside of a Slack channel

169:51

and maybe you want to kick off

169:52

a weekly job that actually reads through all

169:54

of these issues

169:56

maybe it's kicked off on a time-based trigger

169:58

you give it access to

170:00

GitHub and Slack and wherever your issues live

170:03

and you use Cloud to actually help you

170:05

prioritize and maybe open PRs for the most important issues

170:09

Okay, my final takeaways here

170:13

proactive agents, beat reactive agents

170:15

We want Cloud to go from a tool

170:17

to a teammate

170:19

you can move from

170:20

an agent that is waiting for you

170:22

to actually press enter

170:24

and create a PR to an agent that reacts to problems

170:28

and opens a PR itself

170:31

We built routines so

170:33

you don't have to focus on maintaining all of this infra

170:36

but instead you can actually concentrate

170:38

on your domain and process expertise

170:41

This is what routines handle for you

170:44

And finally

170:45

I encourage you to get started with routines

170:47

today

170:48

you're a single slash schedule command

170:52

inside of Claude code

170:54

away from creating your very first routine

170:56

Okay

170:58

my name is Kevin

171:00

I'm an engineer here at Anthropic

171:02

and today we'll be

171:03

learning about how to build agents that remember

171:06

so

171:07

today

171:08

we're going to talk a little bit about

171:09

the base case with agents today

171:12

which is that they're isolated

171:13

And this kind of limits their usefulness in

171:15

a lot of real world workflows

171:17

we'll look at how we can actually solve this problem

171:20

with our new agent memory stores feature that

171:22

we've launched

171:23

this will give agents access to a live

171:25

memory store that they can read and write

171:27

to over multiple sessions

171:29

And then we'll look at how we can

171:31

improve these memory stores over time

171:34

using a new feature that we call dreaming

171:36

and then finally

171:37

we'll get to see how all this ties

171:38

together with both our CLI and also

171:41

our awesome console interface

171:47

so in the previous

171:48

workshops

171:49

I think we've learned a little bit about

171:50

how Cloud Manager Agents has these concepts called an agent

171:54

environment, and a session

171:56

in this workshop

171:56

we're gonna add two additional concepts here

171:59

The first one here is a memory store

172:02

so a memory store is a persistent

172:04

file system-like store that gives

172:06

that attaches as a resource to sessions that you create

172:10

and it gives agents access to

172:12

it gives agents the ability to read and

172:14

write information across sessions

172:17

A dream, is what

172:19

is a asynchronous

172:20

job that runs in the background

172:22

it looks over

172:23

an input memory store

172:25

and a bunch of your previous

172:26

sessions that are represented as transcripts

172:29

and then we run a harness over them

172:31

to

172:32

distill new information that

172:34

maybe the original agents missed

172:35

We do things like fact-checking

172:38

we also organize and consolidate and to duplicate

172:40

information so that your memory stores don't grow

172:42

unbounded over time

172:47

So, also in case anyone in this room has not already

172:51

downloaded the workshop repository you'll need for this

172:54

here is the

172:55

URL. I'll give a few seconds for

172:57

folks to just grab that

172:58

Okay

173:00

so

173:01

let's

173:01

kind of like talk a little bit about

173:02

the problem today. So when you

173:04

when you create agents on cloud manage agents today and sessions

173:07

most of the time you're creating one session

173:09

at a time and these are isolated

173:10

right? The agent doesn't remember information from the

173:12

past

173:13

and it doesn't transfer information to future sessions

173:16

so I want to kind of like take us through this

173:18

this base case today

173:20

so if I switch over to my computer

173:25

I'm in our workshop repository here

173:27

hopefully everyone can see this

173:29

in the back

173:29

but I've just run the bootstrap script that we've

173:33

included in the repository

173:34

And what this has done is basically created

173:36

some seed information for us

173:38

So we have an agent

173:39

we have an environment

173:40

and we have a few

173:41

like, previous sessions that talk about things like the keynotes

173:45

from day one, as well as, you know

173:47

a previous workshop

173:49

And so

173:50

what I'm going to do from here

173:52

is

173:53

actually go ahead

173:54

and start

173:55

walking through this guide with you guys

173:57

Feel free to follow along on your computers

173:59

I'll be using the CLI

174:01

and then also showing kind of the console interface

174:04

for each of these steps

174:06

So first, we're going to create

174:07

a session that basically tells it some

174:09

some new information

174:10

so I'm going to copy this command here

174:14

And as you can see

174:15

we are

174:16

creating a session with the agents that we've created before

174:19

with the environment ID

174:21

and we've given it a title here of just

174:22

like, write test with no memory

174:26

Great

174:27

so now that we create this session

174:28

I'm going to quickly switch over to my console here

174:32

and you'll see that

174:34

let me just move this over

174:35

You'll see that the session shows up in console

174:37

it doesn't, it has a status of idle

174:39

nothing running yet

174:41

so then we're going to switch back

174:42

and then I'm going to send

174:43

this session some new information

174:47

so let's copy this

174:53

And what I'm doing here is sending a

174:55

first user message here with information about the CMA talk yesterday

175:00

naming a few keywords like multi-agent

175:02

orchestration, multi-agent orchestration outcomes in memory

175:06

And I've also just given it this URL

175:08

you know

175:08

example that I took notes and uploaded them

175:10

here

175:13

So if I switch back over to my console

175:16

we should see this event pop up

175:20

And what we sort of expect the

175:22

the agent or the model will do here

175:24

is it'll just respond and say like

175:25

great, thanks for the information

175:27

Not sure what else you want me to

175:28

do here

175:29

so we'll give it a sec

175:31

And maybe focus to catch up

175:36

Go refresh

175:39

sorry

175:40

Yeah

175:40

so it looks like the model responded here

175:42

Let's collapse this

175:44

Yeah, it's basically saying, okay, great

175:46

thanks for telling me this information

175:48

so then if we do

175:49

Sorry

175:51

this is using, I believe, Sonnet

175:54

the agent that was created at the beginning

175:57

Yeah

176:00

cool. So if I go back here

176:02

to my workshop here

176:04

we're going to create basically a second section

176:05

that's kind of the retest

176:07

So I'll go through these steps a little

176:08

bit faster

176:14

and then again

176:15

I'll send it

176:15

a message here that's this time I'm going

176:17

to ask it for information about

176:20

the stuff that I just told it

176:23

And if I go back to our console

176:24

here and check out

176:26

the other session that should have been created

176:30

We would expect the agent to basically

176:34

say something like, yeah

176:35

I don't really have access to this information

176:36

I can help you in these various ways

176:39

Great. So that's effectively the base case today

176:42

so if I go back to the slides real quick

176:46

this is a quick recap of what we

176:47

did

176:48

We told it something

176:49

asked another session about it later

176:51

no information is transferred between the sessions

176:55

So how do we solve this problem

176:55

right?

176:57

just like in humans

176:58

we've introduced the sort of concept of memory

177:00

And again

177:01

a memory store here in

177:03

the

177:04

Cloud Manage Agents platform

177:05

is a file system-like store

177:07

under the hood, you can

177:09

you can create as many memory stores as you like here

177:11

so you

177:11

don't have to necessarily restrict a memory store

177:13

to one organization. You could create it per user

177:17

per workspace

177:17

et cetera. It's up to you to kind

177:19

of define

177:19

what the boundaries of the memory store are

177:22

and then under the hood

177:23

this memory store gets attached as a file system to the

177:26

the session container

177:27

and the model has tools to read and

177:29

write to it

177:30

the actual interesting thing here is that we've

177:31

actually u-

177:32

mounted it as a file system because it's

177:34

such a powerful interface for the model

177:36

you can use things like Bash to

177:38

like, like explore the file system

177:41

It can

177:42

use grep to kind of search for keywords

177:45

it can also read files and do a bunch of

177:46

like, really powerful things

177:48

that

177:49

make it much more useful for the agent

177:53

so I'll switch back to my computer here

177:56

and we'll walk through kind of how to create a

177:58

a memory store

178:01

so first thing first is actually creating it

178:02

So I'm going to copy this

178:04

command here

178:05

Feel free, again

178:06

to follow along on your computers

178:09

and the kind of parameters that you need

178:10

here are mainly just like a name

178:12

so I'm calling mine CWC memory

178:15

You can give it a quick description

178:16

and then

178:17

I'll show you

178:18

also in console later that there are two

178:19

additional parameters that you can set here

178:20

but let me just follow through with a

178:21

simple example

178:25

once I create this memory store

178:26

you can actually see it in console under manage agents

178:30

memory stores

178:34

And you'll see that it's active

178:35

you can actually click into it and view

178:39

essentially a file system viewer of what's currently

178:41

in it

178:42

Of course, there's nothing in it right now

178:44

Additionally, we offer functionality

178:47

we offer the ability for you to like add

178:49

manually add memories. So you can create

178:51

like a file under a specific path

178:53

add some content, et cetera

178:56

If I go back just a brief second and talk about

178:59

creating a new memory store

179:00

there's actually two sort of additional parameters that you can

179:05

Set on the memory store

179:07

when we actually mount it

179:08

So if I go back to the repository here

179:14

the next step, once you create a memory store

179:15

is to actually use it with your sessions

179:18

So I'm gonna, again

179:19

copy a few of these commands

179:21

So

179:21

the first one here is basically just

179:24

giving us the shape that we need to

179:26

pass the sessions API request

179:28

I'll paste it in

179:29

our terminal here so we can see it better

179:33

so again, the memory store here

179:34

you just pass in a memory store ID

179:37

and you can also give it a prompt around

179:39

that will steer the agent to

179:41

read and write specific information

179:42

So you might want it to focus on maybe

179:45

like, a specific, link or a specific

179:48

like, area of focus on

179:49

Let's say you're making

179:50

making, like, a investment agent, right?

179:53

you want to focus on specific things to

179:54

remember for the future

179:55

so you can do that with the prompt parameter

179:58

additionally, there is an access field

180:00

that defaults to read or write

180:03

you can change that to read only

180:05

which will make it so that the session

180:07

and the agent

180:08

will only be able to read from that memory store

180:09

it cannot update it

180:12

So once I run this

180:14

you'll see

180:14

I have created a new session in console

180:20

and this time it'll have a memory store

180:21

attached

180:24

And then if I send an event here this time

180:26

we'll just basically repeat the test that we

180:28

did just before. So let me grab this

180:37

Again, same text as before

180:38

we're telling it new information

180:41

and hopefully we'll be able to observe a

180:42

different behavior this time

180:50

Yeah. So clicking into

180:51

if you click into the session details

180:53

you'll see the

180:53

the information that I just gave it

180:56

And now the model is like first looking

180:58

at memory to see

180:59

okay

180:59

was there anything that like I need to

181:01

remember for this conversation?

181:04

and now it's going to actually like

181:06

of course, there's nothing in our memory store

181:07

so

181:07

now what it's going to do is actually

181:09

save

181:10

the content that I told it to that memory store directly

181:13

and it saved it under this like sessions.md file

181:17

and it's great telling me that

181:18

what it did

181:24

so then

181:25

if I go back and do that same

181:26

test that we did before

181:27

this time I'll just copy both

181:32

sorry

181:37

Again

181:38

or this time we're going to create a

181:39

new session

181:40

with the same memory store that we were

181:41

just using

181:45

and we will send it

181:47

an event here

181:49

asking it what are the things that it found

181:51

or learned from the CMA talk

181:57

Once again, going back to our console UI

181:59

we can see the recall test running

182:02

And as we would expect

182:04

the model is now first

182:06

looking at its memory store to see if

182:08

there's any information

182:10

And again, it's now using grep to find

182:13

any sort of keywords here

182:14

It's looking for CMA

182:16

And great

182:16

it found like a lot of information that

182:17

we just told it from the

182:19

from a previous session

182:20

And now it's able to answer my question

182:22

right?

182:23

and this is like

182:24

of course, a very simple example

182:25

but this illustrates kind of the power of

182:26

of memory

182:27

and this is like something that

182:28

was kind of difficult to do before

182:30

right?

182:37

I'll give quick pause in case anyone's straggling

182:44

okay, great. So what, what else can

182:46

can you do with a memory store here?

182:47

Well, we actually offer

182:49

additional sort of endpoints that allow you to

182:51

manually inspect the store itself

182:54

so I'll use the CLI here

182:55

but for instance

182:56

you can list all of the memory files

182:58

that are in the memory store

183:01

We can, like, do that

183:03

there's also each, memory, a memory store

183:07

memory files in a memory store are also

183:08

versioned

183:09

so anytime you make a change to a file

183:12

et cetera, there's a new version that's created

183:14

and we offer a set of endpoints for

183:15

that

183:19

And then I can also kind of take

183:20

you through the memory store UI

183:28

So each memory store you create

183:29

again, is here. And if we go back to our

183:32

like, kind of file system view

183:33

you can actually see

183:34

the files that it created

183:36

There will be, like

183:36

a directory structure here if Claude is creating

183:38

kind of, subdirectories to organize memory files

183:41

You can actually edit these memory files directly

183:44

if you wanted to. So

183:44

for instance, if Claude wrote something that was incorrect

183:48

or maybe you just wanted to add more information

183:50

you can do that

183:51

And again, as we saw before

183:52

you can add additional memories

183:53

to a memory store

184:02

okay. So

184:03

I'm going to go back to the slides

184:04

real quick

184:08

And as we just talked about

184:10

this is how you create a memory store

184:11

and then mount it on a session that

184:13

you want to use it on

184:14

Again

184:14

it's up to you to kind of decide

184:15

which of your sessions will use memory

184:17

which ones will not

184:23

we also saw how you can like list

184:24

memories and see what's currently in a memory

184:26

store

184:30

And let's move on to talk a little

184:31

bit about dreaming now

184:33

so

184:34

when you have

184:35

agents that are reading and writing to this

184:37

memory store over time

184:38

we've noticed that

184:39

oftentimes they can start just kind of dumping

184:41

information to that memory store

184:43

So it'll start writing maybe every

184:45

every task that you ask it to do

184:46

it'll maybe record some information

184:47

right? And over time

184:48

your memory store is going to grow

184:50

right? And there's no real process

184:52

before that would allow you to sort of organize that memory

184:55

maybe check to see if anything was stale and

184:58

and consolidate any duplicates

184:59

right? And so this is where dreaming comes

185:01

in

185:02

dreaming is a, is a batch process that runs

185:05

again, asynchronously

185:06

You launch it, using, our API or through console

185:10

and it'll run a

185:11

a

185:12

new dreaming harness that we built that is

185:14

a multi-agent setup

185:15

It will look over each of the

185:17

input sessions that you've given it

185:19

So you specify like

185:20

an input memory store that you wanted to dream over

185:23

along with like a group of transcripts that

185:25

you think might

185:25

help or enrich that memory store

185:27

It'll look through each one

185:29

do again fact checking

185:31

enriching with additional details

185:33

maybe dates, specific identifiers

185:35

and then it will

185:36

also organize those memory files and see if there's any duplicates

185:39

anything that can help

185:40

fix, so that when you

185:43

and it'll produce an output memory store such

185:44

that in the future

185:45

when you attach that output to additional sessions

185:48

it will hopefully increase efficiency

185:51

efficiency of like information retrieval

185:53

it also hopefully increase the intelligence of the

185:55

agent

185:58

So let's take a look at how this works

186:00

Again

186:01

switch back to my computer and we'll walk

186:03

through it together

186:12

so what you... To

186:13

to get, get started with dreaming

186:14

basically, you'll need to actually create the dream job

186:19

And I'll

186:19

I'll go through a little bit of the

186:20

parameters here. So the model

186:23

here that we're choosing is Cloud Opus 4.7

186:25

You can choose between Opus 4.7 or Sonnet 4.6

186:27

depending on kind of the level of quality you want

186:29

as well as maybe like token costs

186:32

it takes in two inputs

186:34

so you'll need the memory store that you

186:36

want it to dream over

186:38

as well as a list of session IDs

186:40

so this is up to you to decide

186:41

You can, you know

186:42

maybe dream over daily and

186:44

dream over maybe like 10 sessions at a time or 20

186:46

right? It could go

186:47

you know

186:47

all the way up to like 100

186:49

We're also looking to scale it further

186:51

beyond that

186:52

optionally

186:53

you can also provide the dream job some

186:55

additional instructions that you might add

186:56

So we provide it with a default prompt

186:58

that does a bunch of things

187:00

If you wanted dreaming to

187:01

for instance, specifically fix a few things

187:04

like maybe you're working in a domain that requires

187:06

like, very specific details, right?

187:08

You might ask a dream job to really focus on

187:11

hey

187:11

make sure you backfill these details so I

187:13

remember for the future

187:14

right?

187:14

You can also get it to

187:16

you can also steer it to maybe

187:17

like, organize files a bit more

187:18

like, I want this specific structure in my memory store

187:21

please do that

187:23

so let's actually go and run this command here

187:26

Great, we'll get, we'll get back a dream ID

187:29

and then once again

187:30

I'll go back to our console here

187:32

And so dreams are under

187:33

again, manage agents, dreams

187:37

And once I create the dream job

187:38

it'll start, it'll start with pending

187:40

but it'll start running pretty shortly after

187:42

And you can actually see the status of

187:44

the job both in console and through the API

187:45

I'll show both

187:47

But in console here

187:47

you'll see the input memory store as well

187:49

as the token count

187:51

And generally, like, a dream job can take

187:54

you know

187:54

depending on the size or the number of

187:56

transcripts that you give it

187:57

it could take anywhere from

187:58

you know

187:59

a couple minutes to hours at a time

188:02

And that's really the benefit of doing it asynchronously

188:04

right? Like

188:04

this is not something that you want to

188:05

do live while your agents are working

188:08

And we'll just give it a

188:09

a minute here to

188:10

to run. And you can see

188:12

as it's running, as agents

188:13

or as the harness itself is running

188:15

we're updating the token count for you

188:16

so you can track its progress over time

188:18

The other really cool thing here is that

188:20

Dreaming is actually built

188:21

directly on top of Cloud Managed Agents primitives

188:24

so you can actually see

188:26

that we are creating a session for the Dream job itself

188:30

and you can actually click into it to

188:30

see exactly what the Dream is doing

188:32

this offers a really nice amount of

188:33

like, observability, and so you can

188:35

like, diagnose issues potentially

188:38

and so

188:38

you can see the prompt that we give

188:39

it

188:40

there's a lot of details here that I'll kind of

188:42

that you can explore on your own time

188:44

but the under the hood

188:46

the

188:46

the

188:47

Dreaming harness itself is launching sub-agents to

188:50

look over

188:51

all of the transits that you've given it

188:54

And each sub-agent essentially has a system prompt

188:56

that tells it what to do

188:58

look over it, and the orchestrator is responsible for just

189:00

like

189:00

making sure all the agents are running and

189:02

kicking them off as they go

189:05

So we'll give it another minute for

189:07

to let this run

189:32

another thing to call out here is that while

189:36

we don't actually touch the input memory store

189:38

at all that you create

189:39

So this is a non-destructive process

189:41

What we actually do is we will clone

189:42

your input memory store

189:44

into what's called an output memory store

189:46

and the dream job will be writing basically

189:48

to a new memory store such that

189:50

any edits that are made are non-destructive

189:52

and then we'll see down the line how

189:53

you can utilize this output memory store in

189:56

your future sessions

190:00

Generally, this takes about a minute or so

190:02

depending on, how fast the job runs

190:22

one other thing is that you'll see that

190:23

I'm sort of checking on the job periodically

190:25

as it's running in console

190:27

when you do this programmatically

190:28

we offer an API that allows you to just essentially query

190:32

for the dream job

190:33

and it'll have a status

190:34

so you can pull for the status

190:37

Great. So it looks like it just completed

190:39

and the cool thing here is that in console

190:41

we actually show you a diff

190:43

of what it did

190:44

so you can see dreaming here

190:46

it created an index file

190:48

this index file has

190:49

sort of these slugs that reference the various

190:52

like, memory files that

190:55

a future agent might need

190:56

And the main goal of this is really

190:58

just that future agents

190:59

it's a lot more efficient to kind of

191:01

look at an index file and quickly grok

191:02

like

191:03

what it needs to go look for instead

191:04

of maybe doing a

191:05

a wider grep

191:07

Additionally, it's actually adding, like

191:08

additional information that was not present in the

191:11

in the

191:11

first couple of sessions that I created

191:13

So

191:13

it's creating this event logistics file that gives

191:16

the whole schedule of code with Claude

191:19

a bunch of names as well

191:22

and again, schedule for day two

191:26

and you also see that it actually kind

191:27

of reformatted the memory file that I created

191:29

in a previous session. So this time it is adding

191:32

again, a slug, a description of the event

191:34

some additional metadata

191:36

And again, adding more details

191:38

And generally we find that like

191:41

more information actually really does help future sessions

191:43

And if you think about it intuitively

191:46

while an agent's working on a task currently

191:48

it's kind of hard to predict down the

191:49

line what it might need

191:51

right? That's just generally a harder prediction problem

191:53

So it's actually good to kind of write

191:54

additional details down that a future agent might remember

191:57

and dreaming can always like

191:58

go back and like remove stuff that is

192:00

no longer needed

192:03

and if I go to the output memory store here

192:06

I'll just click on it

192:10

again

192:10

you can see

192:11

all the files that I created

192:12

It's another good way to sort of see what's going

192:15

what dreaming, dreaming did

192:17

and if you wanted like a human in

192:18

the loop kind of review process

192:19

this is kind of where

192:21

this is super helpful

192:22

A human can kind of go in and

192:23

see if the dreaming harness made any mistakes

192:30

Okay, great

192:31

just gonna switch back to the slides real quick

192:33

cause I wanna show a diagram

192:39

yeah

192:40

So again, under the hood, this is sort of what

192:42

how dreaming works

192:43

right? This is a multi-agent harness

192:46

we have an orchestrator that is

192:47

mainly responsible for spinning up sub-agents

192:50

And again

192:50

we spawn one sub-agent per

192:52

input session that you give it

192:55

and kind of the

192:55

the reasons behind this are we

192:57

actually kind of want dreaming to be exhaustive

192:59

by design

193:01

if you give it a hundred chances

193:02

we wanna make sure

193:02

like

193:03

Claude is looking over all the information to

193:04

make sure it's not missing anything

193:05

right?

193:08

great. So now let me

193:09

let me switch back to my computer again

193:12

and this time I'm going to actually

193:13

like

193:14

walk through how we might use this in

193:15

a future session

193:20

So, once the dream, is done

193:22

you can actually go and grab

193:24

the output memory store using this

193:26

So again, we're

193:26

we're just retrieving the dream resource

193:29

with the dream ID that we created and then

193:31

querying or describing the

193:33

the JSON memory store ID

193:36

Great. So this is the memory store

193:38

And again

193:39

you can look through what memories it created

193:50

and now we'll do this

193:51

sort of test again that we did before

193:52

with the two sessions. So

193:53

I will create a new session here

193:57

And you'll notice that this is now using

193:59

the output memory store from dreaming

194:06

Once again, we'll send it an event here

194:15

We're just asking it what sessions I attended

194:18

what resources do I have links for

194:20

and what follow-ups did I flag

194:27

Once again, going back to the console

194:30

it's a really great way to

194:31

you know, visualize what's going on here

194:36

And you'll see that this time

194:37

when it's reading from memory

194:39

you'll see all the stuff that dreaming did

194:41

right? So the index

194:43

the event logistics

194:47

and it's now reading kind of

194:49

it's starting to read the index first

194:53

Okay

194:54

great. Now it knows I'm going to go

194:55

straight to the sessions file

195:01

It's being a little exhaustive here

195:02

just checking for, you know, event logistics

195:18

Okay

195:19

great. Let's see what it came up with

195:20

So I think as you can see

195:22

I I'd have to go back and show you the

195:24

the previous session

195:25

but this time I think there's a lot actually

195:26

a lot more information here

195:27

So it gave me a recap of all the sessions I

195:29

I attended

195:31

it's giving me timestamps now about like all

195:33

the sessions that were

195:34

that were planned for day two

195:35

as well as like the resource links

195:37

right? I think this kind of showcases again

195:39

like how

195:40

dreaming can really enrich

195:41

information that is transferred between sessions

195:48

And then optionally

195:49

at the end, you know

195:50

if you're really happy with the output memory store here

195:52

you can go ahead and actually retire the

195:53

old memory store

195:55

so this won't affect like your previous sessions

195:57

It just means that you know

195:58

it will

195:59

it will keep you the number of memory

196:00

stores in your organization

196:02

down to a reasonable level

196:07

great. So

196:08

I'm going to

196:09

kind of switch back to the slides here

196:16

And talk a little bit how you can

196:16

kind of view sessions

196:19

memory stores, and dreaming as three composable layers here

196:22

right?

196:22

If you think of a session as a

196:24

as an isolated instance of an agent running

196:27

it's one, usually typically one conversation thread

196:29

typically ephemeral, right?

196:31

A memory store augments that

196:33

So now you can connect information between your sessions

196:36

across multiple sessions, right?

196:38

And then finally, with dreaming, you're now organizing, enriching

196:43

and improving your memory stores over time

196:45

so that as you scale up the number

196:47

of sessions you have

196:48

and as you scale the information that it's

196:49

being processed through those sessions

196:52

your memory stays at a reasonable level

196:55

it's manageable, it doesn't blow up

196:57

Also

196:58

it

196:58

checks for things like staleness to make sure

197:00

all the information is most

197:01

is up to date

197:02

right?

197:03

I wanted to highlight some of the maybe

197:04

good questions that I got from the audience

197:06

here. So one of them was around

197:08

generally like sort of what is the like

197:09

token usage of this feature like?

197:12

so as I said before

197:12

like I think by design

197:14

we actually do want it to be exhaustive

197:15

so we do expect it to use a

197:16

lot of tokens

197:18

the

197:19

the nice thing here is that because most

197:21

of the processing is agentic

197:23

most of the tokens are actually cash

197:25

So we're expecting like about a 95% cash rate

197:28

cash hit rate on

197:31

like most dream sessions

197:32

right?

197:33

And additionally

197:34

we are exploring like other ways of offering

197:37

this at lower costs to you

197:39

So

197:40

example of this would be like similar to our batch API

197:42

we could offer things at a 50% discount

197:44

by scheduling it at different times

197:47

other additional like token usage controls include like switching the model

197:51

steering the prompt a little bit more

197:52

also providing more like just general budgeting of

197:55

tokens

198:03

Great

198:04

so I think we're just about winding down

198:06

so I'm gonna

198:08

quickly

198:09

kind of

198:09

go over again for the folks remaining in the room

198:12

like what we went over today

198:14

so again

198:15

we talked about the problem that most agents face today

198:18

which is like

198:18

how do you remember information across sessions?

198:20

I think this is a pretty

198:22

like, well-established kind of problem now

198:25

And we talked about how

198:26

memory is kind of the first step to addressing this

198:27

right? You give

198:28

agents access to something where they can dump their information

198:31

read from it, et cetera

198:33

but

198:34

we also saw how this creates a problem where memory stores

198:36

can grow unbounded over time

198:38

They can grow disorganized

198:39

information can grow stale

198:41

et cetera

198:42

And then we saw how dreaming can be

198:43

used as a way to mitigate this problem

198:46

right? So we

198:46

we take another

198:48

set

198:48

of agents that

198:49

their entire job is to improve that memory

198:51

store for future use

198:57

Yeah

198:57

is

198:58

what are some of the lessons I learned

199:00

and helping Claw code and Cowork kind of

199:02

like grow and as we're building up this

199:04

team

199:05

and kind of like what things I

199:07

and it's it's interesting it's lessons I learned

199:08

even if I think about my time at

199:10

Meta

199:11

or the Microsoft but even at Tropic

199:13

like it's funny I did this slide deck

199:14

maybe like a month ago

199:16

and

199:16

already

199:17

I've had to change some of the content

199:19

cause

199:20

for example when I started this deck there

199:21

were no routines and that even like that

199:23

way of working was different for me

199:25

and so

199:26

yeah like we really wanna like I wanna

199:27

kind of cover

199:28

five themes that I've noticed

199:30

one which is

199:31

the bottlenecks have moved they've shifted

199:34

and so my bottleneck shift

199:36

What are then some of the team norms

199:38

that we had to rewrite within the Cloud

199:40

Co team?

199:41

I want Solana to share a little bit

199:42

about all these team norms we had to rewrite

199:44

how we rolled them out

199:46

and also what are some of the proof

199:48

some of the, you know, like some

199:50

some signals that I got of

199:51

oh yeah, we're trending in the right direction

199:53

and

199:53

it's always going to be

199:54

important for me to

199:55

have a look at to see is it still serving us

199:57

trending in the right direction

199:59

And then I'll end it with

200:01

a few kind of like questions that I

200:02

still have for myself

200:04

and then also some suggestions

200:05

for you to maybe take an action item back

200:08

to your teams to have conversations together

200:12

So with that, the first section

200:14

the bottlenecks have moved

200:16

I call it the shift

200:17

But you'll probably hear me repeat this kind

200:19

of subtitle text a lot

200:20

which is

200:21

what served you prior

200:23

may not serve you any longer

200:25

And when actually, even when I think about my experience

200:27

whether it was like an athletic or meta or Microsoft

200:29

the constant growth mindset

200:31

is just a

200:32

muscle that has served me really well

200:34

And especially right now

200:35

when the rate of

200:36

I don't know if you all are feeling it

200:37

the rate of change is just a little bit crazy

200:39

right? Like I remember the first time I

200:41

started doing some vibe coding was last year

200:43

and it was still making some

200:45

some, you know, bugs. And I'm like

200:47

why, why are you using constants everywhere?

200:48

That's not good at engineering practice

200:51

And now it's, it's, you know

200:52

just become so much more capable

200:54

But that's

200:54

that's what I am seeing as this interesting

200:56

shift of when the bottlenecks moved

200:59

how do you think about adapting

201:01

in terms of everything else around that bottleneck?

201:04

So maybe y'all are feeling this too

201:06

but like for years

201:07

engineering bandwidth was the expensive thing

201:10

Like coding throughput was really expensive

201:12

And when you think about

201:13

all the processes we have of shipping software

201:17

a lot of it was around

201:18

hi, welcome, there's a chair

201:19

Take away those reserve tags

201:21

there's no one sitting

201:24

And there's another one right here

201:25

sir

201:26

Welcome

201:28

but yeah, like all of that

201:29

like even when you think about how we

201:30

used to do planning

201:31

like remember we used to do like waterfall and then agile

201:34

everything was used to be because engineering bandwidth

201:36

was really expensive

201:37

Actually, I'll take a little segue here

201:40

This is not the first time our ind

201:41

like when you think about our industry

201:43

we've always had to adapt

201:45

Like I'm gonna put you all in a time machine

201:47

come back with me

201:48

all the way to the year 2000s

201:50

That was when I started my career

201:52

I worked on Visual Studio

201:53

and we were shipping Visual Studio 2005

201:57

And I kid you not

201:57

in those days, if folks remember

201:59

we used to ship software on like CD-ROMs

202:02

before CD-ROMs was actually floppy disk

202:04

And so I still remember at VS 2005

202:06

there were really hard deadlines

202:08

We had to hit those deadlines

202:09

cause we had to get the software to the manufacturing lab

202:12

to print on the CDs

202:13

to put in the boxes

202:14

to ship in the stores

202:16

And so when you

202:17

when you even think about that

202:17

when we were able to distribute software online

202:20

that also changed how we ship software

202:23

And so that's what I'm finding really interesting of this

202:25

this new, you know

202:26

shift that I'm seeing is engineering bandwidth

202:29

It's no longer the expensive thing

202:32

So for example, on the cloud coding

202:34

for sure coding is rarely

202:36

the slow part anymore

202:39

and I would say it's not even that

202:40

it's not the slow part

202:41

it's just also the throughput has really

202:43

really increased. So it's not only like

202:45

yay, we're all getting to build more

202:47

it's just the amount that we're generating

202:49

has also changed a lot

202:52

And so what we saw was

202:54

you know, when your

202:54

your bottlenecks shift

202:56

from kind of the coding and the actual act of typing

202:58

like if you remember

202:59

it used to be

203:00

writing code was expensive or writing tests

203:03

or refactoring. I remember all of these conversations of

203:06

we have to schedule some time to do refactoring

203:08

Oh

203:08

but we have to do product work and

203:09

this is expensive. When are you going to

203:11

find that time to do it?

203:12

All of that has shifted now

203:13

That is no longer the bottleneck

203:15

And so when that happened

203:17

sometimes I noticed the bottlenecks end up shifting

203:20

towards other areas

203:21

And so what happened

203:22

like for example, verification

203:25

review, cross-functional partners, security, because coding is no longer the bottleneck

203:30

and also we're doing so much more of it

203:33

these are some of the new bottlenecks that

203:34

we're seeing. So it's really always us asking

203:36

is this code correct?

203:38

Who reviews this code? That's like probably one

203:40

of the top questions I get from all fellow eng leaders

203:43

like how are humans keeping up

203:45

with how you guys are doing like code reviews?

203:48

And interestingly, also how is it maintained?

203:51

Because now it is also a lot easier

203:53

for us all to generate a lot of code

203:55

so also thinking about that maintenance cost too

203:59

So with that

204:00

these were some of the processes

204:02

that I noticed quietly stops working

204:05

And I love that phrase

204:06

quietly stops working, because I don't know if you all

204:08

like, a lot of times we all put in processes

204:10

hopefully for a reason

204:11

right? Like we're thinking

204:13

hey, there was a gap here

204:14

we want to improve

204:15

But what I found over the years is

204:17

rarely do processes kill themselves

204:19

We tend to just

204:21

layer more and more and more processes on

204:23

Like I remember it on one team

204:24

we had so many SLAs

204:26

Like there was a P0 bug SLA

204:28

a high priority, like server view incident

204:31

Anyways, there's so many SLAs

204:32

After a while, I'm like, oh no

204:33

we need to stack rank priorities

204:35

so that all the engineers knew which

204:37

SLA was going to be even more important

204:39

And I remember even at that time

204:40

I thought, hey

204:41

we should

204:41

start thinking about what things we should defrag

204:44

a little bit

204:45

But, yeah

204:46

so these are the processes that might have

204:48

served you. If you remember again

204:49

it's that line of

204:50

what may have served you may not serve

204:51

you any longer. But like the planning norms

204:54

we used to spend a lot more time

204:55

you know, pre-planning because coding time was expensive

204:58

code ownership

204:59

there used to also be a lot of questions of who

205:01

who, who wrote this code?

205:03

Who owns it?

205:03

That's a little bit of an other question now

205:06

code reviews

205:07

what we'll get into a little bit

205:09

Team makeup is interesting too

205:10

It's not only like roles are blurring

205:13

right? Like there'll be

205:14

like engineers can also now

205:16

now have AI to help augment non-engineering roles

205:19

My non-engineering partners are also all shipping code

205:22

So what happens

205:23

when roles start blurring and you don't have those

205:25

you know, like silos anymore?

205:29

And then that also goes to knowledge share

205:30

Knowledge sharing and onboarding and everything is another

205:33

signal that we're noticing at Cloud Code

205:36

how we used to do things change a

205:37

little bit too

205:40

And so, you know

205:40

in the first section we talked about the

205:42

shift

205:43

So within the Cloud Code team

205:45

what are some of the norms that we

205:46

have to rewrite?

205:47

So I want to share some of those

205:48

with you and then

205:49

you know

205:49

hopefully some of them will resonate with you

205:51

or you might find helpful

205:54

And so number one is code review

205:57

like

205:58

human judgment of like who actually needs it

206:00

and we kind of go through all of these

206:02

but you know, like the onboarding has also changed

206:04

how we do planning

206:05

you heard him let me talk about like planning a lot

206:08

hiring, especially with roles blurring and kind of team makeup

206:11

how that has changed for us too

206:13

and also

206:13

org shape. That's kind of one of my

206:15

favorite spicy topics. I'll

206:17

I'll share that story with you all when

206:18

I start proposing that in Anthropic

206:20

I love my recruiting partners

206:22

they're awesome, but I remembered, you know

206:24

one recruiting partner really did think I was

206:25

crazy

206:26

And so I want to share that with

206:27

you all too

206:29

So how have planning changed

206:30

but also technical debates?

206:32

Planning, we do a lot less of it

206:34

Like I, I would also, and also the timing

206:37

I call it like JIT planning

206:38

almost like JIT compiling

206:39

because even when I first joined

206:41

I'm like, don't we need a six month roadmap?

206:44

And we, you know, we put some effort in

206:45

we wrote it, it was pretty good for three months

206:47

and then I came back over the new year

206:49

and so many things had changed already

206:51

So I realized, wow

206:53

six month roadmap just seems like a little

206:55

bit too long. So again

206:56

it's

206:57

how do you make sure you kind of

206:58

like do just the right amount and the right time

207:00

because again

207:01

prototyping

207:02

and code generation is just not the bottleneck

207:04

that it used to be

207:06

The technical debate one is a fun one

207:08

too

207:09

So technical debates code wins

207:12

I'll share with y'all when I first joined

207:13

the cloud code team

207:14

I wanted to do a refactoring

207:16

I wanted to like get to learn about

207:17

the code base

207:18

And me and Boris had a healthy technical debate of

207:21

you know, which, which way to go

207:23

And I almost leaned into my old toolbox

207:26

I almost tapped him on the shoulders

207:28

go

207:28

let's go to that room and

207:29

have a whiteboard and we can

207:31

and I'm like, wait, wait, wait, wait a minute

207:33

Nowadays

207:34

I can just generate all the different options

207:36

we've been discussing

207:37

I generated three PRs

207:39

And the cool part about that for the

207:41

technical debate is

207:42

I really cared not only about the implementation of the API

207:46

but also the impact

207:48

to all the callers into the API

207:50

And so when I can have cloud help

207:51

me generate the three different versions

207:53

it allowed us to have a debate of not only implementation

207:56

but also impact to the colleagues

207:58

So I think that's another really interesting kind

208:00

of pivotal change

208:03

so yeah, like, you know, when building is cheap

208:05

arguing expensive, again

208:06

how does that shift your team norms a

208:08

bit?

208:09

I do want Call out

208:10

this makes it even more important

208:12

to make sure you set up team culture

208:14

for how you think about alignment

208:16

For example, what totally will fly is because code is

208:19

you know, so much faster for us to generate

208:21

it shouldn't be like the last person who check in wins

208:23

you know

208:24

like I'm going to stay up at

208:25

3 PM to submit this PR

208:27

set up a routine so that

208:28

I get the last word in

208:29

So definitely a no-no

208:31

That makes it even more important to make

208:32

sure you have good team culture that can have open

208:35

honest, technical debate

208:36

but also a good team alignment

208:42

so I talked a lot about kind of

208:44

what we reduce in

208:44

in planning

208:45

On Cloud Code

208:47

we definitely have reduced the

208:49

design doc before every code ritual

208:51

I would say certain teams and for definitely certain scenarios

208:54

it's still really important to think about design docs

208:57

especially as we're doing like kind of like

208:58

async discussions

209:00

But on Cloud Code

209:01

most of our discussions is like instead of a doc

209:04

a PR

209:04

That's kind of like one of the word we have of

209:06

hey, we found an idea

209:07

go prototype. That's the other thing

209:09

We don't really do a lot of

209:10

product reviews because the landscape is changing fast

209:13

So let's prototype

209:14

let's actually get a lot of internal ants using it

209:16

and

209:17

I'm really a big fan of shipping it

209:18

out to all of you and then hearing

209:20

all the excellent feedback so that we can really

209:22

and, and again, like that

209:23

that has changed our

209:24

planning ritual to be a lot less design

209:26

docs

209:27

and mostly discussions are in PRs or prototypes

209:32

so we reduced that

209:32

but what did we double down on?

209:34

And I think this is an area where

209:35

we actually need to do more and continue

209:37

being better

209:38

It's the verification

209:40

Cause again, it's like the throughput is different

209:43

and there are new ways to break

209:44

So how can you scale out and I

209:46

call it kind of shift left

209:47

Like in the old days

209:48

you would get code out and

209:50

and I would love to

209:51

for me to find bugs before any of

209:52

you find it

209:53

And what's better than me finding a bug

209:55

is actually what I call shift left

209:57

like more

209:57

more automation. So we catch it earlier to

210:00

the source

210:01

So that is something that I think we

210:02

need to continue to double down on

210:04

And the other thing that's interesting is also

210:06

because roles are blurring

210:07

For example, my designers

210:08

like

210:09

I would love to have more confidence that

210:11

when I checked in this code

210:13

I don't break something

210:14

Like I remembered I fixed a

210:16

I fixed a bug in resume or something

210:17

And the next day I was catching up on Boris's

210:20

you know, threads. And I saw someone tag him of

210:22

I'm noticing a bug

210:23

I remembered having that sinking feeling

210:25

I don't know if y'all have like

210:27

Did I just cut you off?

210:29

Like, so because of the throughput

210:31

like, I just really want to make sure everybody

210:32

regardless of roles, just has a lot higher confidence

210:36

for the change that they're putting in

210:39

Who made this change?

210:41

And so my advice here is

210:42

again, because all our PRs are assisted by Claude

210:45

it's a little bit of an audit question

210:47

And so for us

210:48

what is more helpful than just that question

210:50

is what I call like double-clicking into it

210:52

Like, so in the old days

210:54

when you used to ask

210:54

who made this change?

210:56

What question are you really trying to answer?

210:58

Are you looking for who caused this regression?

211:00

Definitely don't want to blame

211:01

but just want to know

211:03

who was the last person that touches code

211:04

that might have caused this break?

211:06

Or are you looking for an expert to

211:08

answer a customer question?

211:09

Or are you looking to gain contacts?

211:11

So whatever is that double-click question

211:13

also think about

211:14

is there a way you can automate?

211:16

It's funny

211:16

I mentioned I had to change a slide

211:17

deck. When I started this talk about a month ago

211:20

part of my every morning is I would

211:21

you know, bring up my desktop code and go into

211:24

here's a customer feedback channel

211:25

And I kind of want to ask it to summarize

211:28

and that was just a morning ritual I

211:29

would do with my morning cup of coffee

211:31

Now it's routines, right? Like I can set that up

211:33

and then even better than what I had done before

211:35

which was I would kick it off

211:36

I'd be like, with my morning cup of Coffee

211:39

I always get to kind of

211:40

a pretty good overview

211:42

So that's my point of like

211:43

code ownership is a little bit more fuzzy

211:45

but on the flip side

211:47

double click to what question you're really trying

211:48

to answer

211:49

and seeing how Claude can actually help you

211:51

with those

211:54

How do you keep up with code reviews?

211:56

So I'm really glad if y'all saw

211:58

the keynote this morning

211:59

Kat talked about it

212:00

We definitely leverage heavily Claude code review

212:04

Now what's interesting is going to be

212:06

where do you trust Claude

212:08

like a lot

212:09

but then where do you still want a

212:10

human?

212:12

And so for sure

212:12

like, you know, and actually Claude also

212:14

as Kat showed you all

212:15

the, the, Claude also does a great job babysitting PRs

212:19

and so we definitely have Claude handle

212:21

all the style and lint and PR feedback requests

212:24

even like maybe catching some bugs and fixing

212:26

them before it even

212:27

does the full commit

212:29

and also

212:30

adding tests

212:31

That's

212:31

like really what we leaned heavily into Claude

212:34

for

212:35

But where I still definitely want to human

212:36

is that expertise. It's all about trust but verify

212:40

So, for example, legal review

212:41

I always definitely want to make sure I'm

212:43

getting my legal partner still

212:45

it's, you know, like risk tolerance

212:47

So, for example, trust boundaries and security sensitive code

212:50

I still want to make sure

212:51

I'm pulling in the experts

212:53

the other area which

212:55

which is kind of fun is also that

212:56

product sense and taste

212:58

Like I remember one of the hobby

212:59

like one of the fun things I like to

213:01

to use Claude for is I like to

213:03

decorate Claude for the holidays or the seasons

213:06

and I remembered last

213:08

holiday I wanted to

213:09

update Claude in the terminal to You know

213:11

give them a little holiday theme

213:13

and

213:14

I asked Claude Claude to

213:15

turn Claude into a snowman

213:17

Claude wasn't that good at asking art in those days

213:19

and I remember asking

213:21

that's where product sense really comes in

213:22

I asked my design partner

213:24

hey, can you review this for me?

213:25

And she gave me such good feedback

213:26

She's like, you turn Claude into like the Mr.

213:29

Peanut character

213:31

because I was trying to make him snowman

213:33

She said, okay

213:33

I'm going to do something more simple

213:34

So Claude was like ice blue with snowflake

213:36

but

213:37

keep in mind kind of that product sense

213:38

as well

213:41

Which leads me to

213:42

what should my team makeup be?

213:44

Because roles are blurring

213:45

Claude is augmenting

213:47

I'll share with you on Cloud Code

213:49

there's two profiles for engineers that

213:51

I've really heavily indexed on

213:53

One are like creative builders with product sense

213:56

Usually you'll see these are

213:57

you know

213:57

the dreamers. There'll be a big sense of

213:59

curiosity

214:00

You're really passionate about

214:01

oh, here's a problem. Here's a

214:03

maybe I can, you know

214:03

ship a product that solves that problem

214:05

But then there'll be a lot of iteration

214:07

to make sure you're delivering a delightful experience

214:10

the other one is deep systems expertise

214:13

So when I first joined Cloud Code team

214:15

I noticed actually we were pretty good with

214:17

product generalists and creative folks

214:19

but we were missing folks with like distributed

214:21

systems expertise

214:22

I mean

214:22

you're building things like Cloud Code remote to

214:24

ensure we can run pods everywhere

214:26

You really still need

214:27

those expertise there. So I would say

214:29

you know, for every software engineering lead you're part of or

214:32

or supporting

214:33

think about those hard parts

214:35

of where you might want to continue to

214:36

still double down

214:38

But for sure, what I index less

214:39

less on is raw throughput because thanks to the models

214:43

we're just a lot more

214:44

efficient

214:46

The cross-functional gaps is another interesting one

214:48

So for example

214:50

I remember that I wanted to

214:51

do an update to how we do survey responses

214:54

but I didn't have a dedicated content designer

214:56

to work with me

214:57

And I'm an engineer

214:59

my writing skills are quite terrible

215:01

I

215:01

I struggle to write things in a short

215:03

and succinct form

215:04

I don't want the surveys to like

215:06

you know

215:06

overload you all in the terminal because every

215:08

line space is really important

215:10

But that's where Claude

215:11

like in the old days

215:12

I would be

215:13

who's a group of content designer I can work with

215:15

and I could kind of give changes back and forth

215:18

but now Claude has really helped me to

215:19

augment that role and was a really good

215:21

content design partner

215:23

for me to kind of like make sure the verbiage is

215:25

is good and succinct

215:27

And on the flip side

215:28

like I say, a PM, our PM code a lot

215:30

which is really fun to see

215:32

So again, like I, I would say with Claude

215:34

like you have non-traditional coders now being able

215:37

to do more engineering

215:39

but you also have engineers that we can

215:41

also now lean in to do things that was traditionally

215:43

you know, like not on the technical side

215:45

but more where the content or design

215:48

So it's very interesting. I found that Claude

215:49

and AI has really

215:51

augmented roles kind of like all around

215:55

This was a spicy one

215:58

so I'm def

215:58

like I remember when I first joined Cloud Code

216:00

everyone's like, okay

216:01

you're gonna grow the team by a certain amount

216:03

and I could tell recruiters were still using the typical

216:06

you know, ten ICs to one manager

216:08

and then how do you start

216:09

thinking about nesting?

216:11

I really have leaned into keeping it really scrappy

216:14

and actually maybe I would step back to

216:15

whether it's a Anthropic

216:17

or Meta, or Microsoft, whether it was like Visual Studio

216:20

or Facebook Marketplace, or AR VR devices

216:23

or Quad

216:24

like what I have found to

216:26

really help me ship great product is heavy

216:28

heavy, heavy dog feeding, especially for leaders

216:31

where nowadays it's actually fun

216:32

I can still be in the code

216:33

but for a while it wasn't

216:34

and when you can't get your hands on the code

216:36

I would always make sure I make your time

216:39

so that I'm actually using my product days

216:40

in and days out

216:41

And so

216:42

that's why I wanted every manager in Cloud

216:44

Code to start out as an IC first

216:46

and also like earn some street cred with the team

216:48

and, and really learn how to be an effective engineer

216:52

and

216:52

and then I really structure the org

216:54

to be as flat as possible cause I

216:56

want us to be super agile

216:57

This was just, my

216:59

my recruiters had some concerns cause I remember they said

217:01

do you want to

217:02

hire managers and

217:04

they will start as an IC first?

217:05

No manager would be interested in that

217:07

I'm like, well

217:08

this is what dogfooding on the cloud code

217:09

team's about

217:10

this is what I expect and if someone's not interested

217:13

it's better for us to do

217:15

earlier separation

217:16

But also, again, this is like

217:18

there's no way I would have been able

217:19

to ramp

217:20

or be able to do code cause my time is

217:23

you know

217:23

there's just a lot of context switching

217:25

without plot

217:26

And so for those of you in the

217:27

room who are managers

217:28

I really, you know

217:29

like encourage y'all to kind of like lean in

217:32

It's, I'll, I'll be honest, for

217:33

for a long time at Meta

217:34

I would still try every year to do one PR

217:37

a year just to

217:38

but the, the code, the

217:39

the workflows would always change

217:41

Like I, all the internal tools

217:42

like by the time I learned one command

217:44

it would have changed

217:45

Nowadays, I don't even remember git commands

217:48

I just always ask Claude to

217:49

to help me out

217:50

out with all of that

217:53

Knowledge sharing

217:54

what becomes your new source of truth?

217:57

So for example, on our team on Cloud Code

217:59

the code is a source of truth

218:00

That's why I would go back to when

218:02

I'm like answering customer requests

218:04

I just have my desktop Cloud

218:06

put desktop Cloud Code and I have it on my

218:09

you know, like, local repository

218:11

so actually answer a lot of customer questions

218:13

So for us

218:14

just having that code base be the source

218:15

of truth

218:16

also like prevent some of the lag that

218:18

you might have had before

218:19

how to keep up the documentation

218:21

you know, correct with the code

218:22

But I would say this is where it's

218:24

like do what makes sense for your team

218:25

So for example

218:26

if you still have a lot of really good specs

218:28

check those into the repositories and then have

218:30

Cloud lean in to help

218:32

you know, like, hey, you know

218:33

like take a look at how verify my

218:35

code execution

218:36

so matches what I expect on the spec

218:41

And so how do we roll it out?

218:43

Cause these were like I had just gone

218:44

through some of the norms that we had changed

218:47

and I think it's interesting

218:48

there's a blend of

218:49

what do we kind of like mandate as team norms

218:52

like make sure you're gaining alignment with the team

218:55

and then where do I really enable each

218:57

you know, like we have pods within cloud code

218:59

wanna make sure I enable each pod to

219:01

do what makes sense for them as well

219:03

So in terms of the balance

219:05

in terms of forcing function

219:07

it's aligned with the teams on the must-dos

219:09

So these are a few of the core

219:10

cloud code team principles

219:12

that we really live and breathe

219:13

And by the way

219:14

if you remember, again, I'll

219:15

I'll go back to that growth mindset and

219:17

always think about if something's still serving you

219:19

We keep these up to date

219:20

so every few months we'll be like

219:22

hey, is this still having

219:23

you know

219:23

the same effect or

219:24

serving the purpose that we wanted when we

219:26

started it?

219:27

So for example, and I should replace this slide

219:30

it's not every engineer uses cloud code

219:32

you know, that's obvious, it's actually every cloud code team member

219:34

including cross-functional partners

219:36

And actually

219:37

we all use cowork quite a bit too

219:40

quadify everything you can

219:42

This is one thing where we're like

219:43

you know what's better than one of us

219:44

doing it? Having quad

219:46

So always think about how

219:47

is there some way for you to automate when there's verification

219:50

more shift left

219:51

but always be thinking what you're doing something right now

219:54

is there some way that quad could actually

219:56

help you do it?

219:57

And the last one's my favorite

219:59

explicit permission to kill those processes

220:01

because again, processes will kill themselves

220:04

but as you're supporting teams

220:05

it's really important to always get that fast

220:07

feedback for your team of whether the things that

220:10

you know, like are we spend

220:11

we're spending a lot

220:12

I'm actually, I remember when I first run quad

220:13

how we used to do standups

220:15

and then the team got a little bit big

220:17

so then we had a spreadsheet

220:18

we would all put up our

220:20

you know, like weekly process, progress, and then I was like

220:23

oh wait, we should just do a

220:25

a, a skill, right, like a, a standup script

220:27

so then we can just run quad

220:28

and all of us can always be very

220:30

be

220:31

much more kept aware of what everybody else

220:32

is doing. So that's just something

220:34

an example of

220:35

one day I remember looking at the spreadsheet going

220:37

wait, this is, doesn't make sense anymore

220:39

Always question

220:41

and always look to defrag and kill old

220:43

processes

220:44

What I want to make sure that I

220:46

leave a lot of room for pods to adapt

220:48

it's

220:49

for each team really has a lot of

220:50

high agency for how

220:52

you know, they do triage or leverage cloud to do triage

220:55

any planning rituals or stand-ups

220:57

how they think about on calls

220:59

and also which workflows to qualify first

221:01

So we don't usually mandate thou shalt

221:04

automate this. We have some suggestions and learnings

221:07

but always give room to your team

221:08

especially they may be

221:09

you know, touching on different, problem areas

221:14

So if I zoom out

221:15

what were the three things I prioritize on

221:18

you know, I, I, when I joined Cloud Code

221:20

but I feel has to make the biggest difference

221:22

keeping the team as flat as possible

221:24

like managers release the pod pods at work

221:27

but really keep it agile

221:28

So for example, on, on Cloud Code and Cowork

221:31

we have one overall team mission

221:32

cause sometimes when you start creating pods

221:34

each pod then wants maybe to set up their own mission

221:37

and then anytime you have to shift

221:39

it might be, take a lot of time to

221:41

to walk people through that

221:42

but it's really as flat as you can

221:44

I felt that served us really well

221:46

The cloudify everything better than you doing it

221:49

see how cloud can help you

221:51

it really frees us up to

221:52

do more of the harder work

221:54

and again, the processes, they do pile on

221:56

so I encourage you to kind of like

221:58

work with your team to see

221:59

what are the processes that actually you should

222:01

be able to let go

222:06

So does it actually work?

222:08

I can't go into the explicit numbers

222:10

but I think these are three general

222:12

kind of metrics that you can look at

222:14

as you're rolling out

222:15

you know

222:15

changes on your team that might start steering you to

222:17

yeah, this seems like it's kind of

222:19

starting to be successful. The onboarding ramp-up time

222:22

that has dramatically reduced

222:24

so that's an interesting thing of

222:25

like, how soon an engineer or a designer or

222:29

you know

222:29

like a PM

222:30

can start being effective on your team

222:33

the PR cycle time shortening

222:35

I think this one's interesting to double-click into a bit

222:37

because it's

222:38

it might actually help you identify a gap

222:40

that's not just in terms of lack of AI adoption

222:43

but where the rest of the pipeline might

222:44

be struggling to scale

222:46

So, for example, again, as, as we're now, like, you know

222:49

just putting in through so much more code

222:51

sometimes a product infrastructure can build on CI

222:54

still keep up with the amount

222:56

that engineers are checking in

222:58

so both of those should go down

222:59

but then what should

223:01

go up a little bit is cloud-assisted commits

223:03

For example, for us, it's by default

223:05

every commit is cloud-assisted. I don't think I've

223:07

seen a

223:08

non-cloud-assisted commit probably in the last

223:11

four months or so

223:12

but hopefully these three things are

223:14

are kind of like metrics that you can

223:15

kind of take a look at

223:16

at your team and see how it resonates

223:19

I would also say

223:20

though, outside of just number of commit

223:23

also think about the end goal

223:25

Like if you zoom out

223:26

what is the product that you're trying to

223:27

make more delightful for users

223:29

or

223:29

what is the problem you're trying to solve?

223:31

Because sometimes you see it on all the headlines of

223:33

this company said

223:35

X percent of code is not generated by

223:37

AI

223:37

I think throughput is great

223:39

but really think about how you measure

223:41

what it is that you're actually really trying to solve

223:43

Like, for example

223:43

we really want to

223:44

make sure we're keeping an eye on quality and reliability

223:47

so those are

223:47

some of the things that we're paying more

223:49

attention to as well

223:51

Okay

223:52

so my last section

223:54

is to audit your own efforts

223:56

I'll share with you transparently

223:57

actually

223:58

I still have a couple of questions that

224:00

I am still asking myself

224:02

the iOS and Android org is a very interesting one

224:04

Like

224:05

when

224:05

engineers can now more efficiently kind of flex

224:08

across different mobile platforms

224:10

there's a more traditional way of

224:12

you know

224:13

having an iOS team and an Android team

224:15

still make sense

224:16

That's something that I'm still kind of thinking

224:18

through

224:19

How much do you push that fully automated review?

224:21

Again, like

224:22

when do you

224:23

kind of strike that balance between fast enough

224:25

and we lost something important?

224:27

It goes back to the trust but verify

224:29

And what's interesting is you even might have

224:31

heard today and earlier

224:32

Darren Danella's talk, the model capabilities do keep improving

224:36

So also

224:36

even if you might need to do more verify than

224:40

than trust for a certain section of work

224:42

that might also change with the next model

224:43

So that's why it's always important to reevaluate

224:46

And roles are blurring

224:47

so how do you make sure that

224:49

everybody feels equally Productive

224:53

So if I were to leave you with one

224:55

you know, thing to take back of is

224:57

you know, pick your noisiest workflow

224:59

And by noisiest workflow

225:00

it could be most expensive

225:02

with something that, you know, what you yourself might be dreading

225:06

or even

225:06

your team might not really look forward to it

225:08

And ask

225:09

is this still really serving what's the purpose of there?

225:11

Like, actually, I'll share another funny story

225:13

There was a team I was on

225:15

where we used to have this weekly review

225:17

It was a very expensive review

225:18

like 50 people, you know, in this large room

225:20

and then I noticed everybody's on their laptops

225:23

except for when it's their time to give status report

225:25

and then they pop their head up

225:27

say the status, and then go back down

225:28

And I'm like

225:29

this is a very expensive meeting

225:31

And I just asked a simple question of

225:32

why are we having it?

225:33

And just that one question

225:35

everybody's like, yeah, it's true

225:37

And so we canceled it

225:38

So that's why it's always important to

225:40

to think. And

225:41

and so yeah, like I

225:41

I figured this might be something

225:43

fun for you to

225:44

take back and look and see what's one

225:46

piece of workflow that you might consider

225:48

can you either automate

225:50

or maybe even is it Still

225:51

you know, proving

225:53

is still serving its intended purpose

225:57

So with that, that's the end of our talk

225:58

I'm like, thank you. Thank you

226:00

Thanks

226:05

thanks a lot for

226:07

thanks a lot for attending

226:08

I really

226:08

really for sure thought this room was going to be empty

226:11

so thanks for not having by myself

226:13

And I'm around here today and tomorrow

226:15

so if you have any questions or want to chat more

226:17

feel free to introduce yourself

226:19

and happy to chat