Anthropic 工程师刚刚发布了有史以来我见过的最好的 90 分钟课程。 Claude Opus

Transcript

Let's get going

0:01

so I'll start with the basics

0:02

So we'll, we'll start with what

0:04

what is an agent or what

0:06

makes an agent work?

0:08

so agent is basically

0:11

a model in a loop with tools

0:14

So you'll see on the left you have

0:15

the model that's deciding and taking the next

0:17

action

0:18

and calling tools in a loop until your

0:20

task is done. So that's like the very basic what

0:23

what an agent is

0:27

Yep

0:39

Oh, a link to a slide deck

0:42

we can try to

0:44

We, we, we can, we can share it after

0:46

Yeah, we'll, we'll share it after

0:47

It'll be hard for us to get it

0:48

out right now

0:50

but happy to answer questions

0:52

and, and share the slide deck at the end

0:54

Are you, is it hard for you to see

0:56

see the slides from the back?

0:57

Okay

0:57

Sorry about that

0:59

all right

1:00

the next thing I wanted to chat about

1:02

is what a harness is

1:03

Harness is typically like a layer that

1:05

wraps around the model that helps you make

1:07

these

1:08

tool calls in a loop

1:10

to get you the outcome for the task that you're

1:12

the task at hand

1:13

And it's making assumptions about what the model

1:16

cannot do and trying to sort of fill

1:18

those gaps

1:19

So it's running in a loop

1:20

making tool calls, managing your context

1:23

encapsulating everything that the model cannot do

1:26

A good example

1:27

is our Sonnet 4.5 model had something what we called

1:31

context anxiety

1:32

So whenever Sonnet 4.5 felt like you were

1:35

getting to the end of the context limit

1:37

it would try to wrap up the task

1:38

really quickly

1:39

So when you're building a harness around Sonnet 4.5

1:43

you would add something like a context reset

1:45

to get around the limitation of the anxiety

1:48

But with the next model

1:49

like Opus 4.5, Opus 4.5 did not have this limitation

1:52

and you did not need to do this

1:53

like, context reset for Opus 4.5

1:56

So when you're building a harness

1:57

you wanna

1:59

Encapsulate all these

2:00

assumptions

2:01

for different models and keep up with it

2:03

with every new model that comes up

2:05

which is why building a good harness is really

2:07

really important

2:07

and that's something we've tried to do with

2:09

cloud managed agents

2:11

And then the last bit

2:13

you want your agent to have access to

2:14

a sandbox or a computer

2:16

essentially

2:17

a place where it's gonna run

2:19

the code it's writing to execute the task

2:21

so typically you're giving cloud access to like

2:23

a sandbox or a container where it's executing

2:25

the code it's writing

2:27

it's able to edit files

2:28

install dependencies, et cetera

2:30

Cloud code is a really good example of this

2:32

It's running, it's a harness essentially running locally on your computer

2:35

has access to your files

2:37

and it's executing

2:37

updating your files as it goes

2:40

it's also writing, code that's untrusted, it might be buggy

2:44

so you wanna make sure When you're deploying production agents

2:48

your

2:48

the code your agents are writing is sandboxed

2:50

and contained within a container

2:53

and then you want

2:54

your sandbox to be isolated and the blast

2:56

radius to be limited. So

2:57

typically you want to execute every task in

2:59

a separate container so you're not mixing up and you're not

3:04

increasing the blast radius for

3:05

for your tasks

3:08

Next up, I'll walk you through

3:11

the first architecture that we actually built

3:13

managed agents with. This was in

3:15

December of last year where we

3:17

shipped our first architecture to our early access customers

3:20

got a bunch of feedback

3:21

and then actually pivoted to a new architecture

3:23

So I'm going to walk you through some

3:24

of like the pitfall

3:26

pitfalls and like issues that we saw with

3:28

the first one

3:30

So with the first architecture

3:31

what we did was we put everything in

3:33

the container

3:34

So we had the session log of everything

3:36

that's happened within that session

3:37

So that session is essentially your

3:40

unit of work

3:41

the harness was also running inside the container

3:45

and then the sandbox execution was also happening

3:47

inside the container

3:48

The benefit of that was

3:50

like, because everything was in the container

3:51

there was no service boundaries to design for

3:54

Everything was local, easy to edit

3:57

file edits were essentially just like syscalls

4:02

But what happened was we realized we adopted

4:05

a pet

4:06

so if

4:07

if you've not heard about the pet and cattle analogy from

4:10

like, the old infrastructure days

4:13

pet is when you essentially have a named

4:15

and hand-tended entity

4:17

When it gets sick

4:18

you have to nurture it back to life

4:20

Otherwise, you

4:21

you basically cannot afford to lose it

4:23

And cattle is more interchangeable

4:25

more disposable

4:26

So when one fails

4:27

you can replace it with another one and

4:29

move on

4:30

So a good example is imagine you had

4:31

like, a single database shard

4:33

It goes down, your product goes down

4:35

you have a pet

4:36

But imagine you have a master replica set up

4:38

your master goes down

4:39

you can bring up a replica

4:40

That's a cattle, right? So you have many instances

4:43

one goes down, that's not a problem

4:44

you can bring it back up

4:45

So what we had with our initial setup

4:47

was essentially a pet

4:51

so few, few issues with this

4:53

Debugging was really hard because everything was running

4:55

in the container

4:57

We didn't know what was causing the issue

4:59

So a, a drop packet

5:00

a dead container would show up very similarly

5:03

and we had to essentially like WebSocket stream

5:05

into the container to debug

5:07

the other issue was investigating meant we had to

5:12

essentially investigating was hard because customer data would

5:15

live on these containers

5:16

so we didn't have a way in

5:18

essentially, to go in and debug

5:19

So that was like the other issue we

5:20

saw in our early access

5:22

And then the last one was

5:23

if a customer wanted us to connect to

5:26

let's say, an M, MCP in their VPC

5:28

it meant peering networks

5:30

or shipping them our harness and having them

5:32

run it

5:33

So there were like a bunch of challenges

5:34

with that architecture that

5:35

that we

5:36

weren't able to really

5:37

have a good solution to overcome

5:39

So what we decided that

5:41

we wanted to essentially decouple the brains from the hands

5:44

and I'll dive into what that means

5:48

so the agent essentially became three separate interfaces

5:52

On the left, you have the session

5:54

which is essentially

5:55

an append-only log of everything that has happened

5:58

in the session so far

5:59

In the middle, you have the brain

6:01

that's your harness or the agent loop along

6:02

with Claude

6:04

and then on the right are the hands

6:05

so sandboxes, your MCP tool calls

6:07

et cetera

6:08

Provisioned on demand, gone when the work is done

6:12

and if you need

6:13

if you lose a sandbox

6:14

you just spin it back up

6:17

So every component essentially became a cattle and

6:20

interchangeable. So when the sandbox died

6:23

the harness or Claude could choose to just

6:25

spin it back up

6:26

right?

6:26

The arrow essentially went back to Claude

6:28

and

6:28

the sandbox to Claude was essentially like any

6:30

other tool call. So the tool call fails

6:32

sandbox fails, it decides what to do

6:34

and it might decide to

6:35

spin it up again or choose to continue

6:37

to move on

6:39

when the harness died

6:40

you could bring up a new instance of

6:41

the harness

6:42

and continue where you left off by essentially

6:44

just

6:45

reading your event log from the session and

6:47

continue from there

6:48

So work resumes from the last event

6:50

as if nothing went wrong

6:52

nothing happened

6:54

so the only thing worth protecting here was

6:55

like your session log

6:57

Everything else was interchangeable. It could go down

6:59

you could bring it back up

7:02

The other thing that we did was credentials

7:04

never entered our sandbox. With our previous setup

7:08

your credential would be in your sandbox

7:10

so you'd be more susceptible to prompt injection

7:13

The change we made was

7:14

if the sandbox ever needed to call out

7:17

to, Something that needed a token or access control

7:22

it would be routed through a proxy that

7:24

would inject your token

7:26

before it called the external endpoint or external

7:28

MCP

7:29

That way the model never sees your credentials or your tokens

7:33

which means you can never prompt inject Claude to say like

7:36

hey

7:36

give me your tokens or print out the

7:38

tokens or write them to a file

7:39

You could never do that because Claude actually never sees your

7:42

your tokens

7:45

The other thing we did was because we

7:47

had this like append only event log in the session

7:51

our compaction strategy could stay flexible

7:53

So nothing is ever thrown away

7:55

We're not compacting and changing your event stream

7:57

We're always storing

7:58

each and every event that happened and then

8:00

the compaction strategy is the one that's deciding

8:03

oh, do I want to look at my entire event log

8:06

summarize it and put it in the context?

8:07

Do I want to only look at like

8:09

the last N events and put those into context?

8:11

So we

8:12

we could have a more flexible

8:14

context strategy because of this like interface design

8:16

that we chose

8:19

And then the other benefit we saw

8:21

is because we could immediately start our inference calls

8:24

we were able to bring down the Tamdeforce

8:26

token by about ninety percent

8:28

So the cloud would only spawn the sandbox when it needed

8:31

and

8:31

it would not wait for the container or

8:33

sandbox to spin up

8:34

for you to get your first token back

8:36

So we were able to get like a

8:37

much snappier and faster experience with this

8:40

architectural change

8:45

and then

8:46

here's sort of like the illustration of like

8:47

one brain can reach many hands

8:49

So your agent looper harness can now call a sandbox container

8:53

it could call into a customer's VPC

8:56

it could call into an MCP server

8:58

So all these like hands are essentially interchangeable

9:00

and you can call and plug in many of these

9:02

and

9:03

it made our architecture a lot more flexible

9:08

so if you take one thing home from this

9:10

it's that harnesses encode assumptions

9:14

And assumptions can go stale

9:16

So we design

9:17

interfaces to outlast these assumptions and made the

9:20

architecture more flexible. So this allowed us to

9:23

like I said, change our compaction strategy

9:25

change out the harness completely

9:27

but the interface and the architecture remains

9:30

the same

9:31

and we can

9:32

really have a more pluggable way of operating this

9:38

All right

9:38

so

9:38

next we're gonna get into a little bit

9:40

of a hands-on portion

9:43

we got you some complimentary

9:45

API credits to help you build with us today

9:48

so please scan the QR code

9:51

either sign in to your Cloud Platform account

9:53

or create a new one

9:55

and you should be able to

9:57

get in there and get those credits so

9:58

that you can build with me and

10:00

and then with Gabe

10:02

And I'll, I'll, I'll leave this for a second

10:04

Hi

10:07

Sorry, what?

10:10

Last slide

10:20

It's basically what we talked about here

10:22

right? This is the

10:25

this is sort of like the three interfaces we designed for

10:27

right? So the first one is the

10:29

the, the question was like

10:30

what were the interfaces that we designed that make

10:32

make, make things outlast, changes?

10:35

So we have the append only session log

10:37

you have the harness that's again interchangeable

10:39

and then the hands where you can plug in a sandbox

10:42

tool calls, or whatever the model might need access to

10:45

and that

10:46

that's what kind of makes it more generic

10:47

and extendable

10:50

All right, going back to

10:55

the credits

10:56

does anyone else need to

10:58

scan this before we jump in?

11:00

Okay, still seeing a couple phones

11:03

Gonna give everyone about thirty seconds

11:04

Thanks for your patience

11:11

Oh, there are problems with the internet?

11:12

Oh, no

11:21

Gabe, do we have a mic?

11:22

Cause I, I can't hear some of the questions

11:32

Yeah, the, the, the credits should arrive once you

11:34

once you sign up

11:35

and, and, hopefully in a minute or two

11:38

yeah, we're

11:38

we're getting a mic so that we can

11:40

hopefully walk around and

11:42

get, get some of these questions

11:43

But yeah, I, I

11:44

I'm gonna walk you through

11:45

building a really simple

11:47

agent that tells you jokes

11:52

and I'm gonna cover some of our core APIs

11:54

our concepts first

11:55

and then we'll

11:56

hop over to platform.anthropic.com to

11:59

build your first agent

12:01

so our first step is gonna be creating our

12:03

our environments API

12:04

which is v1 environments that helps you

12:06

create the environment or template for your sandbox

12:10

So this is gonna have

12:12

your networking setup, so do you want

12:14

unrestricted networking

12:15

or you wanna restrict it to specific domains or MCPs

12:19

you can define what packages you want installed on your

12:22

container or sandbox

12:25

so this is gonna be like the first

12:26

step we're gonna take to set up our environment

12:30

Next, you're gonna create the agent

12:31

So agent is essentially

12:34

you're, you can give your agent a name

12:36

So in this case

12:37

we have, we're gonna call it joke teller

12:39

we're gonna

12:40

choose which model

12:41

you wanna use for this agent

12:42

So you can use Opus 4a

12:44

you can use Haiku

12:46

or Sonnet, your, your choice

12:49

and then we're gonna provide the agent with

12:51

a system prompt. In this case

12:52

we're gonna say, you're a comedian

12:54

find a fresh joke on the topic

12:56

deliver it well

12:58

and then you can provide this agent with

13:00

tools

13:00

In this case, we're gonna provide it with agent tool set

13:03

which is sort of like our

13:04

basic tool set. This is like your bash tool set

13:06

read, write

13:07

grep, et cetera, and you, you, almost all agents will need

13:10

need this

13:12

so that's gonna be your step two

13:14

Step three is gonna be

13:15

finally creating the session

13:17

so session takes an agent ID

13:19

and environment ID and creates a session

13:21

This is your task

13:23

unit of work or your task

13:25

and when the session is provisioned

13:27

it's idle

13:28

No work has been done

13:29

right, until you send an event

13:33

And then this is sort of what an

13:35

event stream on your session would look like

13:38

you're gonna have events that you send and

13:40

the events that the agent will come back

13:41

to you with

13:42

On the left side

13:43

you'll see you have user dot message

13:44

You can say, oh, tell me a joke about distributed systems

13:47

and then the agent will come back to

13:48

you with a funny joke

13:50

you can use something called a custom tool

13:52

Custom tool essentially lets you define

13:54

a custom tool that you wanna handle

13:56

how you, how you respond to it

13:57

and Claude will essentially put

13:59

a

14:00

custom tool use event on your event stream

14:02

that you can listen to

14:03

and then you can process that event and

14:05

then return the result

14:07

to Claude as custom tool result

14:09

And then there are some other examples in

14:11

there for what type of events you would

14:13

see

14:13

So you'll see session dot error if your

14:15

session ever ends up in an error or

14:16

season error

14:17

Status idle is when your session is idle

14:20

not working

14:20

and this is typically when you're not

14:22

really have a sandbox that you're using

14:24

and then Interrupt on the left

14:26

is when you want to stop

14:27

or kill the session and don't want to

14:28

do the work anymore

14:31

this is sort of like a full feature

14:33

list. Maybe if we have time at the end

14:34

Gabe can kind of come back to this and dive in

14:36

but

14:37

Gabe will be kind of working with you

14:39

on his part of the workshop

14:40

going deeper into some of like these

14:41

concepts like multi-agent and memory that we also provide

14:47

All right, before we go to that one

14:49

we're going to hop over now to platform.claw.com

14:55

and hopefully you can use your API credits

14:57

and see those now if you don't

14:59

have Travis walking around with a mic in

15:01

case you have questions or you need to

15:02

just like flag him down for

15:04

for anything

15:06

but if you go to your console

15:07

you'll see in this manage agents section

15:09

there is something called as a quick start

15:12

This is sort of like our

15:14

right, thank you

15:15

so it sounds like maybe everyone might not

15:17

be able to follow along

15:18

but hopefully

15:20

we can find a way to

15:21

share out these slides and you can try

15:22

it out at home. Or if you have your phone

15:24

maybe you can try to pair it with

15:25

your hotspot and see if you can follow along

15:29

all right

15:29

so let's try this one more time

15:31

We're gonna say, build me

15:34

an agent that tells me jokes every

15:39

time given a topic

15:43

And then you just hit enter

15:48

And you can see Claude is coming up

15:51

With, system prompt, it came up with a name

15:54

a description, chose a model

15:56

You can, like, go change this model if you want to

15:59

and then

16:00

give the agent access to the

16:02

agent tool set

16:03

And then you will hit

16:04

create agent

16:06

That should basically call the

16:08

V1 agents endpoint to create the agent

16:10

Okay, that, that got created

16:12

Next, we're gonna configure the environment

16:15

which is the template for the sandbox

16:21

we're just gonna create a new one

16:28

And then we'll just give it unrestricted access for this one

16:31

but you can, obviously, when you're p-building a production agent

16:34

you, you wanna tighten the access

16:36

control which domains it can

16:38

reach out to, or specific MCP servers

16:40

So we'll do unrestricted

16:44

And then here you can see you can

16:45

like choose packages you want to install

16:48

et cetera

16:49

And then next we'll just go and start

16:50

the session

16:54

All right

16:54

now we're going to go test run our session

16:58

wait, tell me jokes about cats

17:04

And you'll see the model started thinking

17:11

and it came up with a cat joke

17:15

Yeah, that, that was like a

17:17

that

17:17

that's how easy it is for you to

17:18

go build your agent. Now this

17:20

agent is up and running in the cloud

17:22

All you have to do is start a

17:23

session with the agent

17:25

send events, get back events

17:27

Everything else is managed by us

17:29

So we're managing all the infrastructure

17:30

the hosting, the harness, the sandbox management, the scaling, the storage of your event log

17:36

and then you can focus on building your

17:38

product around this

17:40

thank you

17:41

I'll now hand it over to Gabe to

17:42

do the more hands-on bit of the workshop

17:45

and hopefully, hopefully

17:46

we have better luck with our Wi-Fi

17:49

Thank you

17:56

Can you hear me?

17:57

Great

17:58

I see there's a bunch of people that

17:59

are kind of squished in the back

18:00

There's room here and here if you want to come in

18:03

so we can fit everybody in

18:06

before we start

18:07

here's a

18:09

repo that we're going to be using

18:11

so if you want to

18:12

take a picture of the QR code

18:14

git clone that repo

18:17

cd into the research desk

18:19

directory, and then run, npm install or boni

18:24

I'll let you do that

18:26

then I'll walk you through

18:28

through what we're going to be building today

18:30

together

18:33

still see some phones out

18:35

I know you're, there's Wi-Fi issues

18:37

Hopefully you can, you can get

18:39

you can get to the repo

18:42

okay. Oh, oh, I see you

18:43

Okay

18:46

Cool. All

18:49

right

18:50

So what are, what are we gonna be building today?

18:52

We're, I'm gonna show you how you can use

18:54

cloud managed agents to build a variety of

18:56

different applications

18:57

We're gonna start with

18:58

building a custom chat-like interface so that you

19:00

can talk to an agent

19:02

we can say hello

19:04

then we're gonna augment that agent into a multi-agent

19:08

ecosystem so that it can delegate tasks

19:11

to sub-agents. We're gonna give it skills

19:13

specifically a skill to be able to call

19:16

the SEC website and get SEC filings for individual tickers

19:21

and analyze specific forms

19:24

then we're gonna be able to use the

19:26

these new enhanced agents to run more asynchronous

19:30

agentic tasks that can take a long time

19:32

to run

19:34

and then

19:34

we're gonna bring it all together

19:36

by

19:36

enabling this sort of MapReduce interface where

19:39

we can talk to an agent

19:41

and that agent can then delegate to many agents

19:44

and do sort of like a MapReduce thing

19:47

showing you how you can build a custom

19:49

harness on top of CMA to orchestrate

19:52

you know, your own work with custom tools

19:54

and manage multi-session, kind of interactions

19:59

and then we'll finish up by

20:00

I'll show you

20:01

our deployments API that allows you to

20:03

set up recurring work

20:05

on a schedule so that you can run

20:07

sessions, on a

20:09

on a given cadence to run

20:10

kind of longer term analysis that keeps updating

20:12

itself

20:14

Sound good?

20:16

Quiet

20:17

Okay, great

20:19

all right, so let's switch over to

20:22

a coding interface. I'm gonna be using

20:24

Cloud Code for this. You're

20:25

free to use whatever coding agent you prefer

20:29

I don't think anybody's writing code by hand

20:30

so I'm not going to do that either

20:34

there is a skill

20:35

that we've loaded up

20:36

in your repository

20:37

that's gonna walk you through the workshop step

20:40

by step

20:41

so if you wanna rerun the workshop after I'm done

20:44

later on when you have better Wi-Fi and better internet access

20:48

you're able to do so

20:49

at home

20:51

so the first thing you're gonna wanna do is

20:53

I assume you've checked out the repository

20:55

you've seeded into the research desk repo

20:58

you've run bun I

20:59

we're gonna spin up our

21:01

server with, bun run dev

21:03

I'm gonna make this bigger

21:04

I'm assuming nobody can see

21:07

Okay, is that better?

21:09

Yeah, and if you have questions while Gabe

21:11

Gabe is presenting

21:12

feel free to raise your hand and I

21:13

can run to you with the mic

21:14

We, we have helpers, so if you run into issues

21:17

raise your hand

21:18

People will come, help you, help you out

21:22

cool

21:22

So if you've gone this far

21:24

you'll be

21:25

basically be able to load this website

21:27

and the first thing you're gonna wanna do

21:29

is go to the setup page

21:33

everybody with me so far?

21:35

Yeah? Okay

21:37

now we're gonna

21:42

okay, once, once, he takes a picture

21:45

we're, we're gonna start up

21:47

our workshop skill

21:48

So in Cloud Code

21:49

you can type in

21:51

workshop start

21:54

and it's gonna start running

21:55

through the workshop for you

21:56

We're gonna do it in six acts

21:59

I apologize, there's no intermission in between

22:02

I hope that's okay with y'all

22:04

but we're gonna go through it

22:05

Cloud will

22:09

sure

22:10

We're gonna go into auto mode

22:12

Claude

22:12

Claude is gonna keep track of each of

22:14

the acts so that

22:15

you can continue on

22:16

you can keep typing workshop next so that it

22:19

you know, is gonna

22:20

keep, progressing through the, through the thing

22:23

I'm gonna ask Claude to do it for me

22:25

but handhold

22:27

handhold me and teach me what it's doing

22:30

and so the first thing

22:31

it's gonna basically make sure that our prerequisites

22:33

are set

22:35

it's gonna copy

22:36

the dot env dot example file into a dot

22:39

env dot local

22:42

oh, no, stop

22:44

We're not there yet. Okay

22:46

It's a little eager. Okay

22:46

it's gonna copy that file

22:48

that file

22:49

has a few values for you to fill

22:51

in. You're gonna wanna paste in your API

22:52

key

22:54

you're gonna wanna paste in

22:55

the SEC

22:56

wants to know who you are when you're calling their API

22:58

so you're gonna have to put in your

22:59

name and email

23:01

this is just so that

23:02

they know who's calling them

23:05

and then that's basically it

23:06

Once, once you've done all that

23:10

it's going to, well, we're going to move on to the

23:14

to the next thing. So

23:15

everybody with me so far?

23:16

You've put in your API key

23:18

into your dot m dot local?

23:20

You put in your Edgar identifier?

23:23

Show of hands. Be a quick show of hands

23:25

How

23:25

how many of you are able to follow

23:27

along who might have Wi-Fi or other way?

23:29

Okay, we got one, one person at least

23:35

MVP

23:35

Pardon? oh, oh, for the repo

23:41

This one?

23:46

Rohan, you want to hand the

23:47

the mic?

23:49

Yeah, the slide with the credential

23:50

That's it. This one. Thank you

23:52

Oh, okay, sure

23:53

And then, and then given, given people don't have Wi-Fi

23:55

please

23:55

please like stop us and ask questions so

23:57

we can

23:57

at least make this more interactive and help

23:59

you understand concepts so that you can

24:01

build with your own time after the workshop

24:07

Okay

24:09

I'm, yeah, I have

24:14

a. Yeah, this is the app spinning up

24:16

when you clone and CD into research desk and run

24:20

run, run dev, this is a website you will

24:23

you will get

24:24

Can I ask a question?

24:26

Yeah, I don't see you

24:27

but

24:27

Oh, yeah

24:29

I'm wondering like, you know

24:31

so when you say cloud handle and manage the agents

24:34

where is this running? Is this running on AWS?

24:37

Yeah, that's a good question

24:39

So we, we

24:39

we're gonna run it in our own infrastructure

24:42

today we, we have like a one P offering

24:45

that we run on

24:47

but we also have cloud platform on AWS

24:50

if you wanna run on AWS using your

24:52

AWS credits

24:53

What we also offer

24:54

and maybe we can talk about this after this demo

24:56

is self-hosted sandboxes. So if you wanna bring

24:58

your own sandbox through

25:01

different providers like Vercel or Modal

25:04

or Daytona or a bunch of others

25:06

you can do that

25:07

even, you know, through Google Cloud and

25:09

and AWS. If you wanna run your sandbox in that

25:11

environment, you can

25:14

okay, I'm gonna continue with the demo

25:16

the, the first thing you want

25:18

the next thing you wanna do is create

25:20

push this button, which is gonna create

25:22

your agent. So it's gonna create two things for us

25:26

an environment, which Priyanka talked about

25:28

is a template of what the sandbox is

25:30

gonna look like

25:32

for this demo

25:33

we're gonna

25:34

give it access to a few

25:36

websites that it can reach out to

25:37

mainly the SEC

25:39

We're gonna add

25:40

some Python packages so that it can reach out to

25:42

to

25:43

the SEC with, you know, some predefined

25:46

Python packages

25:48

it's also gonna create

25:49

an agent. So

25:50

it's gonna create this very basic agent

25:53

Right now it has no tools

25:55

it has, just a prompt

25:58

well, it has the basic tools

25:59

but there's no skills

26:00

there's no sub-agent coordination

26:02

nothing like that

26:04

I'm gonna tell Claude to

26:08

continue

26:09

and implement the interface with which we can

26:12

talk to this agent in our application

26:15

like Bianca mentioned, the main

26:17

way you communicate with a session is through

26:19

an event stream

26:20

So you listen

26:21

onto that stream for all the events that

26:23

the agent is pushing back down to you

26:25

and you can use that

26:26

an endpoint to push events back up

26:28

So really

26:29

everything that happens in the communication for

26:31

these more chat-like applications happens on that stream

26:35

So we can see here what Cloud is doing is it's

26:39

implementing

26:40

the section so that when

26:42

I press enter in my chat box

26:44

it's gonna send a user dot message event

26:46

with the content that I'm sending

26:48

to that session

26:49

And then it's gonna

26:50

route the

26:51

SSE stream back onto my browser from my

26:53

backend

26:54

So it's gonna listen onto

26:57

the events that are coming in the

26:59

stream connection to our API

27:01

and then restream them directly back onto

27:04

the frontend so that I can render them

27:05

in a nice way

27:07

and once Cloud has done that

27:09

we should be able to go

27:10

to this desk

27:12

Tab and just say hi to our agent

27:15

and if everything worked

27:20

hopefully, great

27:22

so we can see here now we're

27:23

we're talking to our agent

27:24

If you wanted to build a chat-like application

27:26

this is where you would stop

27:29

which is great, but we wanna do more

27:31

The whole point of cloud managed agents is

27:33

that you can do long running asynchronous tasks

27:36

not just for minutes

27:38

but hours, days, weeks, and even months

27:41

so let's see what we

27:42

we have to do next

27:43

right?

27:44

The next thing that we're gonna wanna do is

27:48

oh, Claude is very eager today

27:52

okay, the next thing is we

27:54

we wanna augment our agents

27:55

right? We wanna

27:56

give our agent a skill so it can

27:58

call the SEC

27:59

We wanna use multi-agent coordination so that

28:02

our session can delegate work to sub-agents that

28:04

are more specialized and

28:06

capable of doing some tasks

28:08

and we wanna

28:09

basically move up the ladder of intelligence

28:13

so let's do that

28:15

by saying, workshop next

28:18

now what Cloud is gonna do is it's

28:20

gonna implement the provisioning scheme to

28:22

upgrade our agent so that it uses

28:25

more subagents or any subagents

28:29

is we're gonna give it a skill

28:31

a skill that lets it

28:35

talk to the SEC

28:36

We've described how to use the Python package

28:38

we've described that it has to use the

28:40

Edgar identity to call

28:43

we've, we've described all of that

28:44

and we're gonna give it a custom tool

28:47

for later in the workshop

28:49

that custom tool is what we're gonna be

28:50

able to use to tell our

28:52

head of research agent to spin off

28:54

multiple sessions that we will coordinate

28:57

for a, a sort of custom harness

28:59

So this is what Claude is doing here

29:01

it's updating its progress

29:03

creating this custom tool called dispatch analysis

29:05

We're gonna come back to this

29:06

in a second

29:08

we give it a schema

29:10

it's going to update the

29:13

the

29:13

description of the main agent so that it

29:15

creates a new version of that agent

29:18

this new version will have

29:21

oh, pager duty

29:24

this new version will have

29:26

a few different sub-agents

29:28

that it can delegate work to

29:29

as well as a reference to itself

29:31

So if

29:32

the main agent ever wants to

29:34

spin off a copy of itself with its same prompt

29:36

it can do that

29:38

this is important because

29:40

sometimes you don't necessarily know

29:42

what sort of tasks your session's gonna wanna do

29:45

but you have a general

29:47

idea that you wanna have sub-agents that

29:49

you know, spin up and do work

29:51

so you can define a self-reference to

29:53

you know, yourself

29:54

The cloud can then spin up a sub-agent for

29:58

important

29:59

this question has come up in the past

30:00

It doesn't copy the transcript

30:01

of that session into the sub-agent

30:03

It just copies over the tool description and the schemas

30:07

and the prompt, into, into that sub-agent

30:12

cool

30:13

So now, if we go into here

30:18

we should see that there's a new agent

30:21

Oh, I think I need to provision them

30:22

So there should be a button in our setup

30:25

that says staff to desk

30:26

When I do that

30:27

it's actually gonna run

30:28

oh

30:31

okay

30:32

I ran this before. Let me delete the

30:34

skill so that

30:35

I can recreate it for you

30:37

I was trying to do the magic

30:39

cook show thing, but that's fine

30:44

okay, so we'll delete the skill

30:45

We'll ask Cloud to reprovision it

30:49

it's gonna create all those resources

30:50

and then we'll be able to see here all the sub-agents

30:53

the skill that we created

30:55

a memory store. So for these sessions

30:57

we, we, we also support

31:00

long running memory store that can be shared

31:02

across sessions

31:03

and we're gonna use that so that

31:05

different sessions can learn about different financial instruments

31:08

and they can share that knowledge

31:09

with each other

31:10

And at the end of this workshop

31:11

I'll show you how you can use that memory

31:13

long running memory

31:14

to keep improving your agents long term over many

31:17

many, sessions, days, months, years

31:22

cool

31:22

So we should see here that the agent now

31:26

here

31:30

has the skill that we described

31:32

It has sub-agents

31:34

with their own prompts and tools and permissions

31:37

it has a prompt

31:38

and it has a model

31:41

and I think we're

31:42

we're ready to

31:43

use this new dedicated sub-agent to do our

31:46

first asynchronous long running task

31:49

So let's

31:50

tell Claude to move on to the next

31:52

thing. The next thing that we want to do is

31:55

we want to tell Claude

31:56

not in a chat-like application

31:58

but a more asynchronous thing

32:00

hey

32:00

research this financial instrument and give me a

32:02

report

32:03

in a format that

32:05

I define and I care about

32:07

And we're going to use something called outcomes

32:08

Outcomes is how we've created self-supervision

32:12

So in an outcome

32:13

you tell Claude, this is your task

32:15

this is your rubric to evaluate your task

32:18

it's going to go off and do that work

32:20

and then when it thinks it's completed

32:21

it's going to ask a sub-agent to

32:24

look at the rubric and the work that Claude did

32:27

and evaluate

32:27

did it do it correctly and did it

32:29

do it to completion

32:31

if it does, then great, and if it doesn't

32:33

it will try again

32:35

so here what

32:36

Claude is implementing for us is basically this user-defined outcome event

32:41

so that anytime we start a session

32:43

in our UI that I'll show you for a specific ticker

32:46

it's gonna send this user-defined outcome with a specific rubric

32:51

max iterations

32:53

and a description of the task at hand

32:56

And so then Claude will go run

32:58

that task for us

33:00

and so

33:00

what does that look like in the UI?

33:02

If you look to this scorecard

33:05

I can tell Claude

33:06

hey, I, I wanna analyze NVIDIA

33:08

Oh

33:11

Please

33:15

Okay

33:15

great

33:16

So it's gonna kick off a session to

33:17

do the analysis on NVIDIA

33:20

this is gonna take a while

33:21

I'm gonna

33:22

do the cook show thing and show you a previous session

33:24

of what, of what happened

33:26

it's gonna take about ten-ish minutes

33:29

but here's something I ran yesterday

33:31

the first thing we're telling Claude is

33:33

here

33:34

is an outcome. I want you to analyze NVIDIA's most recent

33:38

SEC, you know, quarterly filings

33:41

you're gonna use Edgar identities

33:43

this is where your Edgar identities are

33:45

you have memory, so check if you've already done analysis previously

33:49

and make sure that you use that instead

33:51

of

33:51

refiling and researching things that you already know

33:56

I want you to find the most

33:58

recent form that, that I care about

34:00

and run any analysis over that

34:03

and then I want two deliverables

34:04

I want you to write a note in

34:06

in my memory, for that specific company

34:10

and I want you to write a scorecard

34:11

in a well-defined structured output

34:14

and I want you to save that

34:15

in a specific file location

34:18

so then Claude went off to work

34:21

started calling tools

34:22

it didn't find any notes in its memory store

34:26

it, looked in a few different places

34:29

and then it decided

34:30

okay, I know what to do

34:31

I'm gonna leverage my sub-agents to do

34:34

the bulk of this work

34:36

I'm gonna spin up a financial

34:39

expert analysis agent, or extractor

34:42

and then a risk evaluator analysis

34:44

And you can see all of that in the

34:47

console here for full observability

34:49

Each of these lines here represents a thread

34:52

and a thread represents the work done by

34:53

a sub-agent

34:55

And so each of the sub-agents went off to work

34:57

and then when they were done

34:58

they let the main agent know

34:59

hey, this is what you asked of me

35:01

and then the main agent woke up

35:04

got that information from the sub-agents and then

35:07

filed everything it needed to do

35:09

and created the deliverable. It created the JSON file that

35:12

you know, we specified

35:14

and it made sure to update the memory

35:16

And then the cell supervision started

35:19

So it kicked off a sub-agent to

35:21

verify that the outcome that we define

35:23

is being met

35:25

it looked at the rubric

35:26

it analyzed all the work that was done

35:29

in the session and analyzed the artifacts that it created

35:33

and it, decided in the end that this was satisfied

35:36

that the, you know, scorecard was delivered

35:38

that the numbers are well defined

35:40

that the memory update

35:42

you know, was correct, and everything looked good

35:46

I don't, I think this is still running

35:48

but Once that completes

35:50

you would see a table here of each of those tickers

35:53

because we can extract the reports

35:55

the scorecard from the session and download it

35:58

You put in a database

35:59

For this demo, we put it locally

36:01

but I think you get the idea

36:05

okay, this is great

36:06

but what if you wanna do a hundred or

36:08

or ten tickers at a time?

36:10

Wouldn't it be great if I could chat

36:12

with an agent

36:13

to build a custom harness that could then delegate work

36:16

for each of the tickers into a different session

36:20

and then when it's all done

36:21

aggregate the results and give me

36:23

a bunch of scores?

36:25

so that's, that's what we're gonna do next

36:27

we're gonna leverage that tool

36:29

the custom tool that we created

36:30

to build our custom orchestration harness in our server

36:35

where when we talk to our main agent

36:36

at some point, that main agent is gonna decide

36:39

hey, I wanna spin up

36:40

ten different sessions to do work

36:44

we'll get that, oh, I skipped a step

36:46

sorry

36:49

For, we're, we're gonna update the skill

36:51

first

36:52

so I, I wanted to show you that these skills are

36:55

are live artifacts that you can update

36:57

so you can improve the skill

37:00

in this case

37:01

we're adding the ability to look at a

37:02

new form

37:04

and then when you do so

37:05

any new sessions that

37:07

reference that skill will get that new information

37:11

so this is something that you can continuously

37:12

update

37:13

You can look at previous sessions to see

37:14

how they performed. We're gonna see that

37:16

at the end of this

37:18

demo that, that happens with, with something we call dreaming

37:22

but, yeah, this, this is what is happening

37:26

okay, I'm

37:26

I'm gonna move on to the orchestration bit

37:31

so now that, yeah

37:32

that basically that skill is published

37:34

I think if I go here

37:36

we should see

37:38

Any questions so far?

37:39

I know there's a lot to take in

37:41

Travis, there's a question over here

37:43

Yeah

37:52

I, I

37:52

I know someone came up to me asking

37:54

about like how long it takes to get

37:55

the credits. It should be only five minutes

37:58

I know the form says one to two days

37:59

but you should see it within five minutes

38:03

I was curious about the

38:04

the data access, from the SEC that the

38:07

the research agent is grabbing

38:09

Does that require some sort of credentials

38:11

and are you gonna walk through the part of how the

38:14

how that credential injection or

38:16

or brokering works? Yeah

38:17

so for, for the SEC

38:19

it just requires your email and your name

38:22

for this demo

38:22

we just inject that in the prompt

38:24

But in a real production application

38:25

we would use something called Vaults

38:27

the way Vaults work is you define a secret

38:31

that you wanna u- inject in an environment variable

38:34

and instead of putting the real secret in

38:36

what we put is in

38:36

is a placeholder

38:38

and then when

38:39

Claude is doing work and it wants to

38:41

use that environment variable

38:43

it just injects the placeholder value

38:46

We at egress swap it out

38:48

for the real value

38:50

Only on the host domains that you specified

38:52

are allowed for that secret

38:54

So if you say

38:56

my super secret MVAR can only be used for the SEC

38:59

and then cloud tries to use it for anthropic.com

39:02

we won't inject, we wouldn't do the swap

39:04

But when it tries to call the SEC.com

39:06

it would do the swap for that

39:09

It's essentially like an API

39:11

or you can use it on console

39:12

where you just go create a vault

39:13

give it a name

39:14

you'll get an ID

39:14

and then you essentially include that ID in

39:16

your session

39:17

as a way for us to authenticate with your MCPs

39:20

your CLI

39:21

For example

39:22

we could do like a Google Workspace CLI

39:24

that you install on your

39:26

sandbox

39:26

and then you can give us like your

39:28

API key as your environment credential

39:29

and again, we'll, we'll kind of follow the same method that

39:32

Gabe talked about, so that

39:33

that allows us to

39:35

But bots also work with OAuth credentials for peace

39:38

and they also work. Gabe

39:39

I don't know if you want to just show it on

39:41

on console, like, the, quickly, like the vault creation

39:45

Yeah, so you can, you can create a vault here

39:48

and then in a vault

39:49

you would bundle credentials

39:51

so you can have MCP auth

39:53

maybe you do Notion

39:55

you know

39:55

we have a few predefined things for

39:57

OAuth things

39:59

so whenever you reference a session

40:02

you inject that vault ID

40:04

and it will bundle all the credentials into your session

40:07

and whenever it tries to use any of

40:10

the domains that you specified in your credentials

40:12

it will swap for the real

40:14

credential instead of the placeholder

40:17

Yeah

40:18

quick question

40:18

what part of the capabilities in the cloud agents SDK

40:23

and what's about?

40:26

Can you, what do you

40:27

what do you mean? You have the agent SDK

40:29

right? I presume

40:30

this is built on top of the agent

40:31

SDK

40:32

I want to know what part of the

40:33

capabilities are in the agent SDK layer

40:35

and what's outside of it

40:37

So the, the agent SDK is packaged for

40:40

like more, more

40:41

it's like a headless cloud code version

40:43

So it's actually

40:44

we built a generic version of that

40:46

We've taken a lot of inspiration

40:48

about the benefits and lessons from the agent SDK

40:51

but we've rebuilt, the harness to be a more generic

40:54

not just coding specific thing

40:56

So it's sort of a fork?

40:58

yeah

40:59

Yeah, it's basically like its own harness

41:02

Agent SDK is basically like a

41:04

cloud code in the form of a

41:05

binary that you can

41:06

take and run wherever you want

41:08

This is a fully hosted solution running in our cloud

41:11

and we have our own harness that's

41:13

heavily inspired by cloud codes harness

41:15

and we frequently stay

41:16

stay in touch for like any enhancements across the board

41:20

I, I saw another hand in the

41:21

in the back, over there

41:23

We have a question here

41:24

Thank you

41:25

I guess the question as you're orchestrating between multiple agents

41:29

is there some guardrails that those agents would

41:31

not go into loops and

41:33

like, consume a lot of tokens

41:35

as they're generating or working together?

41:39

some of that we leave up to

41:40

you to orchestrate and figure out

41:42

I mean, we're, if you're building these sort of custom harnesses

41:45

there's nothing stopping you from using

41:49

like other

41:49

providers if you wanted to do a sort of multi

41:53

multi-provider system

41:55

we are exploring the idea of what it means to do

41:58

like, budget, tighter budget controls that you can say

42:01

use no more than this amount of money per session

42:03

use no more than

42:04

this amount of tokens per session

42:05

but that's not something we offer today

42:10

yeah, go ahead. One more hands

42:12

Question

42:13

earlier you were showing us in console kind

42:15

of a transcript of the manager and then

42:17

the sub-agents

42:19

And you can kind of go through and

42:20

see what they were doing

42:22

If you were to come across maybe

42:24

a mistake or like it was going down

42:26

the wrong path or you wanted to change history

42:29

I mean, obviously that's what's happened

42:30

Is there some way to

42:32

correct it or to kind of go back

42:33

to that point in time and kind of start over

42:35

try it with some additional guidance?

42:37

Is there any, anything like that in here?

42:39

You can steer it. So I could

42:40

I could, I think here if I say

42:44

send event, I could send a new user

42:46

dot message event and steer the agent

42:49

it'll queue it up. It's

42:50

you know

42:50

you can send the ten messages

42:52

at a time and it's fine

42:53

We don't currently have a way for like

42:55

rewind and roll forward

42:57

but that's definitely something that

43:00

if, if people are interested

43:01

we, we would look into

43:02

Yeah, session forking is essentially what you're asking for

43:04

We've, we've had a couple

43:06

a few customers who have asked for that

43:08

It's, it's on our roadmap

43:09

so hopefully we'll have it at some point

43:11

Thank you

43:13

yeah, I see a couple of hands here

43:15

If you want to

43:16

yeah, thank you, Rohan

43:19

so

43:20

Where are you? Okay

43:21

okay, go ahead

43:24

Assuming all of this is running in the cloud

43:25

like, you know, I guess

43:28

I guess I need roadmap to

43:29

get collective knowledge across these agents

43:32

Like, let's say

43:33

this one particular application you have ten users

43:35

Would, would you mind speaking up?

43:36

It's a little hard to hear

43:38

Oh, so, so if you

43:41

assuming all of this is running in the cloud

43:43

and this application is being distributed across

43:45

tens and hundreds of users

43:47

and all of them are using it for the purposes of

43:49

like, you know, doing this financial research

43:51

do you have, I

43:52

I know you guys have session logs

43:54

Is there a collective

43:56

memory that you guys are

43:58

looking into or even have a roadmap of?

44:01

We have it today

44:02

We actually use it in this demo

44:03

So we use a

44:04

a memory store

44:06

that allows you to share knowledge across sessions

44:10

and you can see here that what's happening is

44:13

The agents are each populating

44:15

individual information about what they've done so far

44:19

and then

44:20

I'll show you at the end how you

44:21

can

44:21

consolidate that knowledge over time using dreaming so

44:24

that you can continuously improve the memory stores

44:27

as different agents run into different edge cases

44:29

and collectively learn from each other

44:32

Just to follow up on this

44:33

so you're storing this in Anthropix infrastructure

44:37

say if you want to delineate that and

44:39

and have things running in AWS Bedrock

44:41

and I have my own

44:43

like, Daytona clashers or something for sandboxing

44:46

then you mentioned about self-hosting

44:48

How much of a lock-in is this piece versus

44:52

like, how flexible is it?

44:54

So all of this API based

44:55

so you

44:56

you can extract the

44:58

memories and put them in

45:00

different systems if you want to

45:01

We don't currently support out-of-the-box other memory providers in the session

45:06

but

45:07

you would just build an MCP

45:08

or a custom tool to do so

45:10

is the way I would imagine doing it

45:11

And

45:11

and also to add to your initial question

45:13

about like having user level org level memory

45:16

we do allow for attaching multiple memory stores

45:19

So you can imagine having

45:20

a user level memory store and then having

45:22

some sort of org wide memory store and

45:23

then you bring

45:25

both to each session so you can start

45:26

collecting user level

45:27

at as well as org level memories

45:29

And then like Gabe said

45:31

none of this is locked in

45:32

You can like take your data and go

45:34

anywhere else if you wanted to

45:35

Everything is an API call

45:37

Yeah. I had a quick question

45:40

As you walk through the agent workflow

45:42

I noticed that the

45:44

grader is at the main agent level

45:46

right? Like

45:46

so how do you

45:48

grade these sub-agentic tasks that are happening and

45:51

then how do you manage memory across those

45:53

as well?

45:54

That's a good question. So

45:56

there's two dimensions to this

45:57

Sub-agents in the way that we've built them are meant to

46:02

Augment your context and leverage the capabilities of your

46:05

of your session by collaborating between them

46:08

the outcome definition happens at like an overall arching

46:13

session level

46:14

So you define the outcome and you say

46:16

this is what I want you to accomplish

46:17

The main agent then uses

46:19

you know, uses sub-agents to accomplish that work

46:22

It collaborates between them and it eventually says

46:25

I'm done

46:26

there is no

46:27

currently support for like a sub outcome definition

46:32

Follow up, like, so as you have this dreaming feature

46:35

right? Like, so how do you learn?

46:37

Like if the sub-agents have some sort of

46:39

you know, you know, off course sort of actions

46:42

then how do you sort of learn from that?

46:43

Yeah, today the dreaming's constrained to

46:46

memory stores. So I'll

46:48

I'll show you that

46:50

in, in a second. But basically

46:52

the way Dreaming works is they will analyze

46:54

the transcripts and the memories created across multiple sessions

46:58

so that it can update its memories so

46:59

that future agents know what to pull

47:02

but you can imagine that we

47:04

can continue to expand that for skills and

47:06

for prompts and for agent definitions so that you can

47:09

you have this continuous learning

47:11

functionality. But right now

47:12

we just have it for

47:13

for memory stores

47:17

See a hand here

47:18

and then I'll, I'll go back to

47:20

Hello

47:21

Here

47:22

Where, where is here?

47:23

Here. Oh, hi

47:24

Thank you

47:26

if you have your

47:26

let's say, company data in either

47:30

I don't know, traditional store or rack or whatever

47:32

and you want to expose it to this

47:34

type of agents through

47:36

custom CLI or MCP

47:38

what would be

47:39

the safe path

47:40

to put it in your environment?

47:42

That's a great question. There's two

47:43

two ways to do this

47:44

If you are using

47:45

if you can access that

47:47

via CLI with an API or via

47:50

an MCP, you would just create an MCP

47:53

give it, give it the authentication vault

47:57

and it would just be able to call

47:58

your MCP directly

48:00

If you don't want to expose your MCP to the internet

48:02

we have a feature called MCP Tunnels

48:05

which lets you, create a tunnel from your

48:09

network

48:10

into ours so that you don't expose it to the internet

48:13

and we Do it that way

48:15

If, if that's, too big a barrier to climb

48:17

we do support self-hosted sandboxes so that you

48:20

can bring your own sandbox

48:21

which has

48:22

which is peered with your network and has

48:24

access to your applications and services

48:26

and the agent would just

48:28

call the sandbox that you manage and host

48:31

and be, be able to access your

48:33

your resources that way

48:36

Yeah

48:38

I had a question on dependencies for the

48:40

environments

48:41

So I saw something you could

48:42

load in dependencies

48:45

is that at runtime or right before

48:48

it's created?

48:49

It's at creation time

48:51

and then we'll cache it and reuse

48:53

that environment across, like, re-re-instantiations of the sandbox

48:59

So can you specify

49:00

like, internal dependencies

49:01

or do you have to pick npm package?

49:03

Right, right now, they are all

49:06

they all have to be accessible via npm cargo

49:10

Pip, whatever, but you can upload files

49:13

directly into, the session if

49:16

or GitHub repos if you want to install

49:18

them that way

49:19

Okay

49:20

And one more thing

49:21

deployments, I see a tab for that

49:22

Yeah, yeah, I'm, I'm gonna get to it

49:24

I'm gonna get to that

49:25

Continue to the, with the demo, and then we'll

49:26

we'll do more questions in a second

49:44

Okay, so, yeah, let's, let's see what this looks like

49:49

analyze, Google and Microsoft

49:53

so ideally what would happen

49:54

and I hope this works live for y'all

49:57

is that, this main agent is going to

50:00

evaluate if it has information

50:02

and then it's gonna

50:03

give us a custom tool

50:05

where we're going to then spin up

50:07

individual sessions for each of those financial artifacts

50:10

So you'll see this dispatch analysis thing

50:13

it's gonna call it for two tickers

50:15

and then it's gonna kick runs for each

50:17

of those

50:18

so this is how we can orchestrate our multi-session

50:22

harness, and each of those sessions has multi-agent

50:25

that are doing collaborative work

50:27

and each of those

50:28

also have outcome definitions

50:30

so that there's self-supervision happening

50:32

using memory stores to learn from one another

50:35

et cetera, et cetera

50:38

so let's move on to the finale

50:39

which is

50:41

What if you want to do recurring work?

50:43

What if you want to do

50:44

you want to have a list of companies

50:46

you care about and you want every

50:48

day at 9 a.m. or first of the

50:49

month to do the analysis over all of these companies

50:53

on a schedule?

50:54

So that's what deployments are

50:56

Deployments is, everything is API based

50:57

so we

50:58

you can obviously interact with our platform via the console

51:02

but their main way that we intend to you

51:04

for you to interact with it is via

51:06

the API and the SDK

51:07

So you can build

51:08

custom UIs and custom workflows

51:11

all of these are just building blocks

51:13

that you can compose together to build your

51:15

application into something bigger

51:18

so

51:18

you can see here what Cloud did is it created a

51:22

deployment, and that is right here

51:25

A deployment is, takes in a agent definition

51:30

in this case it's this

51:31

research, desk weekly memo using this specific agent

51:36

it takes in a schedule

51:38

of when it's gonna run

51:39

and it gives it an initial set of

51:40

messages that it's gonna run with

51:42

You can also pass in

51:44

resources like

51:46

memory stores or GitHub repos or files

51:48

that it's gonna use on each of those

51:51

and so what this means is that every

51:54

I think this is

51:55

first of the month at 1 PM or something like that

51:57

it's gonna run, this, specific session

52:02

I could run one now

52:03

so I could trigger it via the console

52:05

it's gonna do that

52:08

but I could also build a UI for it

52:10

and that's what we've done here

52:12

so I can, you know

52:13

I have all the CRUD I can list and

52:15

and search, for my deployments

52:18

and then I can also kick one off

52:20

at any given point. And I don't know why they're failing

52:22

but I'll ask Cloud about it later

52:26

and it, this has come up a few times

52:27

which is, what about dreaming, what about self-improvement?

52:31

so before I get to your questions

52:32

which I'm sure there are many

52:34

I'm gonna show you a preview that we've

52:37

have out for, for dreaming

52:39

So what dreaming is

52:40

it lets you select

52:42

so, okay, before I show you a previous one

52:45

it lets you select the memory store

52:47

In this case, it will

52:49

will use the memory store that we've been

52:50

using

52:51

You can select the model

52:53

you can give it custom instructions if you want to

52:55

but we'll use the default prompt

52:57

and then we're gonna let it

52:59

run, the

53:01

the last 30 days of sessions through this

53:03

dreaming analysis

53:04

So when I click start dreaming

53:05

what it's gonna do is it's gonna spin up a session

53:09

using managed agents actually

53:11

spin up

53:12

many sub-agents to analyze the transcripts of each

53:14

of the other sessions that we told it to evaluate it

53:17

all of them which use the memory store

53:20

and it's gonna analyze

53:21

hey

53:21

did the memory store encode all the information

53:23

that was required? Did a session

53:25

run into any issue that

53:27

that caused problems? Can we

53:29

learn anything from it?

53:30

Did this happen multiple times?

53:31

Can we update the instructions in any way?

53:34

And you can see here what it

53:35

actually did is that it found that

53:37

the skill that we created actually had a few gaps

53:40

on the way that it was written

53:42

and it updated the memory store to tell it

53:44

even though there is this skill

53:46

you should extend this knowledge with the fact

53:48

that we've learned from it

53:49

and you can imagine that

53:50

we will continue to extend this to

53:52

not only improve memory stores

53:54

but improve skills directly

53:56

agent prompts, sub-agent prompts, et cetera, et cetera

54:01

that's, that's it, that's the demo

54:03

I, sorry about the Wi-Fi

54:04

I hope you follow

54:06

later on when, when you get home

54:08

and let's, keep going with some questions

54:10

Okay, we can all take questions

54:12

Okay, we have a question over here

54:13

Yep

54:15

thank you for the demo

54:16

we didn't see any

54:17

telemetry, instrumentation in the agent

54:21

Is that a default native feature?

54:23

What, what do you mean by telemetry?

54:24

Because you have the full observability of all of the events

54:27

that happen, right? If you go into the console

54:30

you can see it live

54:31

but you can also

54:32

obviously get it from the API where

54:34

this transcript view is playing like a nice view

54:36

but if you go to debug

54:37

it's every event that occurred in that session

54:40

so you can see exactly

54:41

for example

54:42

when did we start calling the model?

54:44

when did we finish calling the model?

54:46

So you can get latency metrics in that

54:47

Is there something else that you mean by telemetry?

54:50

Yeah, the thing is that

54:52

usually in software development cycle

54:53

you kind of define

54:54

what you want to observe

54:56

how did we

54:57

kind of pick what did we

54:59

want to observe?

55:01

right now you get all the events

55:02

and you just filter them on your end

55:05

If there are things that you think are missing

55:07

hit us up, and we'll talk about adding them

55:09

Okay, and can I bring my own observability stack?

55:12

In here, you mean like export to hotel?

55:15

Yeah

55:16

you, you could, because it's an API

55:18

you could build that connector fairly easily

55:21

we just, everybody has different requirements and we

55:24

we don't want to be opinionated

55:25

but everything is just API calls

55:28

Thank you

55:31

I, I had a question

55:32

So how long can these agents run?

55:35

I, I, I know you mentioned this dreaming thing

55:37

but

55:38

can you run them indefinitely or do you

55:40

see that persistence store keep increasing in size

55:43

and at some point you need to recompact

55:45

it offline or something?

55:47

We are running compaction

55:49

for you so that you can run these

55:50

for days. We've seen ten hour sessions in production

55:54

from some of our users

55:55

So you could run these agents

55:57

like for a year or indefinitely in theory

55:59

Yeah

55:59

yeah. Are you talking about running the agent

56:01

for a year or are you talking about

56:02

like one specific session that keeps going for?

56:05

No, basically the agent, if I had to

56:08

if I had a task that I need

56:09

to do every day

56:09

can I just keep it running for six months?

56:12

You could, yeah. Will it not blow its

56:13

persistence towards? That's right. Yeah

56:15

And is there any orchestration

56:17

piece like where I could have three agents

56:20

agent A does something and delegates to B

56:23

and C? Is that some capability that you have or?

56:26

For, for now you would spend

56:27

you would do sub-agents and you would tell

56:29

Claude what you want

56:31

We find that

56:32

There's a desire as humans to constrain

56:35

what Claude does and the way it does

56:37

it. As Claude gets smarter

56:38

we have continuously removed those guardrails and let Claude

56:42

just let it cook

56:43

let it do what it thinks is best

56:45

So

56:46

right now you think maybe

56:47

three sub-agents is the best topology for it

56:50

but if we just let Claude figure that out

56:52

it actually ends up outperforming

56:55

But any topological or like graph-based workflow thing

56:58

in the UI here

57:00

like what I could have?

57:02

Not yet, not yet, not yet, but, but maybe soon

57:08

you, you can probably just like speak up

57:10

yeah, sorry, sorry, we'll come back to you

57:14

Hey, over here on your right

57:17

Yes

57:18

So question, looks great

57:20

a lot of integration points

57:22

What I'm curious about for

57:24

you know, a complex organization that potentially manages multiple agents

57:30

you know, I think they want some sort of separation

57:35

RBAC, whatever else

57:37

How is that, what is, I guess

57:38

the recommended approach for

57:41

a complex organization that has

57:43

those different agents in here to effectively manage?

57:47

We have workspaces that's our primitive today

57:50

where you can

57:51

put different customers in different workspaces and then

57:55

segment them that way and give access to

57:57

specific users into those workspaces

58:01

But

58:01

we know that there is still a gap

58:03

in what we offer and That's

58:06

that's something

58:07

Yeah, our, our back is on our roadmap

58:09

so we, we do want to provide more like granular controls

58:11

but right now

58:12

workspace is like a clean way of like

58:14

separating

58:15

your agents where you have like a group

58:16

of people access to workspace A and they

58:18

can access all agents within that

58:20

You can have workspace B

58:21

with like a

58:22

different set of users that can have

58:23

different levels of access

58:25

So, hi, excellent presentation, really excited

58:29

two questions. One is deployment

58:31

My understanding was that

58:32

deployment is like at a snapshot

58:34

a moment in time of

58:37

All the agents

58:38

no? The plugin is essentially a way for

58:40

you to run

58:41

sessions for a specific agent on a schedule

58:44

So you can say like

58:45

oh, research desk agent, go research NVIDIA once every quarter

58:50

and I'll just create a session every quarter

58:51

to go research and produce you that report

58:54

And then memory, which is a state

58:56

is persisted across all sessions

58:59

You attach your memory store to the session

59:01

So you decide which

59:03

which session

59:04

you want to attach which memory store to

59:06

So the memory store is like a

59:07

independent entity that you just attach whenever you

59:09

need to

59:10

Right

59:11

And the final question is harnesses

59:14

Right. One, what is harnesses for you?

59:17

Because everyone have their own definition

59:19

And second is the confidence

59:21

Right. So when I change something here

59:23

how do I know that it's not

59:26

it's not, right. My own confidence as a developer

59:31

for

59:32

you're right. Everybody has a different definition of a harness

59:35

we, When we talk about a harness in CMA

59:38

we mainly mean the tool loop execution

59:40

so

59:41

managing the context and managing the session

59:44

state and making sure that

59:46

the agent can do the work that it

59:47

wants to do

59:48

When most builders talk about harnesses

59:50

they, they really mean

59:52

oh, they really mean

59:54

the ability to coordinate

59:55

multiple agents and sessions together

59:58

as a unit of work to achieve some

59:59

sort of

60:00

bigger task in a topology of sorts

60:03

we're giving you the abstraction of a single

60:05

session

60:05

as a unit of work for you to

60:07

go out and build your harness

60:09

with it as

60:10

the session being one of the primitives

60:13

and then your second question about how do

60:15

how do I know how my agent is

60:16

performing?

60:17

our best recommendation would be to run

60:19

evals on your agent with like different types

60:21

of tasks that you want the

60:22

agent to perform and see

60:24

how the agent does on those tasks

60:26

outcomes is a really good way where we

60:28

steer the agent for you to get you

60:30

the specific outcome where you can define

60:32

an exact rubric for like

60:33

hey

60:34

this is the exact outcome

60:35

I want and you can

60:36

define max iterations and the agent will keep

60:38

going into it until it gets you that outcome

60:42

Yeah, the, the, the verifier agent that

60:44

that Gabe talked about

60:46

Hi

60:46

So, after

60:49

Cool

60:50

Hey, thanks for the demo

60:51

I enjoyed it

60:53

my

60:53

my question to the core is a bit

60:54

about how to choose between skills and agents

60:58

And I wanted to ask

60:59

how would you all

61:00

say what's like the fundamental difference?

61:02

Is there an advantage to the harnessing that

61:04

surrounds an agent

61:05

that makes it produce better results than a

61:07

skill?

61:08

and one of the reasons I'm asking is

61:10

cause I'm afraid of always investing in new kind of like

61:15

I guess, harnessing and tooling around models

61:17

and it's

61:18

it can be kind of tiring always trying

61:20

to do a new thing

61:21

so yeah

61:22

I just want to hear what you think

61:23

It's like, is there a big difference

61:25

and is it going to change any time?

61:26

I, I, I, yes, no

61:30

the question is, is thick harness, thin harness, thick skills, is

61:35

is a question that's constantly evolving

61:37

I think what we find is that as cloud gets smarter

61:39

the boundaries and, and

61:42

and the help that we want to give

61:43

cloud start to melt away and disappear

61:46

And so

61:47

what you need today might not be what

61:48

you need tomorrow

61:50

The boring answer is just run evals and

61:52

and, evaluate these things

61:54

That sounds exhausting, obviously

61:56

like every time a new model comes out

61:58

I need to evaluate all these things

61:59

what

62:00

what we find is

62:02

skills is a good abstraction today

62:04

it gives the model enough

62:05

Information to do what it needs to do

62:07

but I presume that

62:08

it will start to

62:10

shift and melt and become thinner and thinner

62:11

where

62:12

the model will just be able to do

62:13

things on its own without any special requests and instructions

62:18

down the line. So

62:20

multi-agent seems to be more useful

62:23

in my opinion, but your

62:24

your mileage may vary on your task

62:29

Hi. Hey. Hi

62:30

So my question ties up to last two question

62:33

and especially on top of evals

62:35

One thing I'm excited about is dreaming

62:37

but

62:38

how will dreaming actually affect the final outcome?

62:41

Something that's where evals come into the picture

62:44

but you can have multiple iterations of dreaming

62:46

so that can go in different directions

62:49

So is there a plan

62:50

or do you already support some automated

62:53

kind of dreaming with run with evals and see that

62:56

hey, how did the different direction go into

62:59

which one should we pick based on what's going on?

63:02

Yeah, so, so dreaming what it does is it looks

63:04

oh, you

63:05

you can choose the

63:06

duration for which you want the agent to

63:08

dream over. So you can say

63:09

I want to dream over

63:11

this agent's session over the last thirty days

63:13

and then

63:14

look at all the transcripts and

63:15

analyze what tasks you're trying to achieve

63:17

and then

63:18

come up with like a new memory store

63:19

that's more geared towards like what you were

63:21

trying to do

63:22

And we typically will run evals

63:24

on our end to make sure that dreaming

63:26

is providing you with better outcomes

63:28

and we've seen customers were able to get

63:31

get a significant improvement on their evals by

63:34

running dreaming and then using the new memory store

63:36

But again, again, everything kind of ties back into evals

63:39

so like

63:40

that's how you would know whether something is performing

63:42

Or, or do you expect it?

63:52

Yeah I think it would be good to

63:53

have your own set of evals

63:55

yeah

63:56

dreaming is in research preview so

63:58

these are

63:59

early early it's like a big experimental thing

64:02

we see an uplift

64:03

on it generally but this is

64:05

we we hear you right like

64:07

how do I gain the confidence of this

64:08

yeah it's like

64:09

evals connected to dreaming and then

64:11

we can find out multiple pathways that

64:14

that has been taken and decide oh

64:16

this makes more sense because even though

64:19

nine sessions may have been

64:21

saying that this is wrong

64:22

one says that hey this is good

64:24

so

64:25

if you just rely on that

64:26

session dreaming thing

64:27

I feel like we may not be grabbing

64:29

all the insights without evals but if I

64:31

tell you just to trust me is that

64:35

thank you

64:38

yeah

64:39

hi thank you very much

64:41

this is very impressive

64:42

those of us who have been working with

64:43

cloud code for a year or more

64:47

Have been doing things that are similar to this

64:49

but this

64:49

it's nice to see this in a really mature form

64:53

I, one question I have is

64:55

how much of this was just built on

64:57

top of cloud code?

64:59

That is, if somebody was, you know, ambitious, could, they could

65:03

they build something similar to this?

65:05

And, what I wanted to ask you was

65:08

encourage you to keep things open

65:10

some of us are already harvesting what's in the

65:13

JSON session files, and starting to experiment with things like

65:17

HEPA rag and that kind of thing for

65:20

for managing the, the memories and, and creating a, a memory pool

65:25

So, congratulations, thank you

65:28

Please keep it open

65:29

and, can we, is there

65:32

are you doing anything here that's not just built on site

65:35

on top of cloud code?

65:36

Thank you

65:38

We

65:38

we took the cloud code harness as an

65:39

inspiration. We found that it was

65:41

very well attuned for coding tasks

65:44

and Different

65:46

it's different levels of flexibility for other tasks

65:48

We're building an API where we want

65:51

you to bring your context

65:53

your information, your capabilities, and build on top of it

65:55

where Cloud Code is trying to

65:57

give you a product out of the box

65:59

So we have slightly different

66:02

you know, value, props

66:05

and so it's not just

66:07

Cloud Code in a box

66:09

though, though, you know, ideally it performs as well

66:13

if not better, in all sorts of tasks

66:16

so not, not exactly, but it's definitely inspired and motivated by

66:20

by some of the

66:21

uplifts that we have seen from Cloud Code

66:24

And, and Cloud Code is a terminal tool

66:27

and we obviously want to be an API product

66:29

and, and infrastructure, that is gonna scale

66:32

that's gonna be reliable

66:33

so there, there, there were, like

66:35

challenges that I talked to you about

66:37

in the beginning of the presentation that we

66:39

really worked it in

66:40

to make sure we're providing a really solid

66:43

For

66:46

you all to use. Yeah

66:48

so for AWS Bedrock

66:49

is there

66:50

a console like Cloud Console like you have here?

66:52

it's quite, great, great for the question was

66:54

yeah

66:54

I just want to manage and see everything

66:56

This is not available on Bedrock

66:58

but it is available via Cloud Platform for AWS

67:01

so there is a AWS version of

67:03

console that you could go use this with

67:08

Yeah

67:10

first, is there any, how to say

67:14

is there any restriction on the model to be used

67:16

like, non-anthropic models?

67:19

Just like, Cloud Code, I can run on non-anthropic models

67:22

For, for now, we, we, yeah, it's

67:24

it's Cloud Models only for now

67:26

Oh, okay, so, so the second question is

67:29

what about the end user use?

67:30

do they have to have a console account

67:32

in order to use it

67:33

or just a free

67:35

I can create a customized UI for this

67:37

and they can use it?

67:39

So

67:39

Cloud Console helps you with creating your API

67:41

key

67:43

So that's how you get access to our APIs

67:45

So you, Cloud

67:46

you're not paying separately for Cloud Console

67:48

It's free to sign up

67:49

you create an API key

67:50

and then you're paying based on

67:51

like, token usage. So for

67:53

for, for the case like

67:55

SaaS, like, maybe we have multiple tenants

67:58

right?

67:59

So in order to provide this

68:01

agent, what kind of, how you say, model you have already

68:06

depict on, so that we can follow

68:08

for example, what, what kind of, each company are supposed

68:10

tenant are supposed to have a

68:12

a different space, or, or even there is a

68:16

a sub module for this

68:19

each tenant. So are

68:20

are we going, how do we

68:21

are we actually going to organize that in

68:23

terms of this? We

68:24

we

68:25

we

68:25

have seen that the best model for multi-tenancy

68:27

is not bring your own key

68:29

because you lose observability

68:30

because it runs on your end user's API key rather than

68:34

than yours. So we encourage you to model

68:36

things with workspaces as a tenancy primitive

68:41

And we're improving that so that it's easier

68:43

and easier to build

68:44

on top of this for multi-tenant applications like

68:47

Yeah, so I

68:48

I think currently you could either have

68:50

put all your customers data into like a

68:51

single workspace and then you maintain a mapping

68:54

on your end on like

68:55

which agent maps to which of your customers

68:58

Alternatively, like Gabe said

68:59

you could create a workspace per customer of yours

69:02

but that can get more complicated cause then

69:04

you have to like recreate your agent

69:06

So, so is any, you see the examples that

69:09

people did the additional security layer or RBAC layer over the

69:14

this workspace? Because to me

69:15

it may be a little bit

69:17

you know. Yeah, that, that would probably be like a good

69:20

good way to

69:21

do it right now cause we don't have any RBAC

69:24

controls, on our level other than like workspace

69:27

So everything

69:28

within your workspace is accessible to everyone in the workspace

69:31

but we are working on

69:32

making our back better and then making the

69:34

experience better if you're working

69:36

with cloud managed agents but building on behalf

69:38

of your customers and how do we make that easy

69:41

So

69:41

so is there any

69:42

way you can show how to scale this

69:44

if, if there's a multiple

69:46

how to say, tenant working on that?

69:48

Is there any scaling parameters

69:50

like, what is the, how to say, the, the minimal, pod or whatever, these parameters, that was, yeah

69:58

I mean, you, you're, you're defining on a

70:00

you can create as many sessions as you want

70:02

Yeah. Like, there's no limitation

70:04

I mean, we have some rate limits

70:05

obviously, but, but there's no scaling parameter for you to say

70:10

now I need to run a 10,000 pods

70:11

So, so currently no, no such thing as

70:13

like, like HPA in the Kubernetes

70:16

like you can do these kind of settings

70:19

scaling, parameters, right? Like timeout or whatever

70:22

You, you don't need, that, that

70:23

You don't need to

70:24

cause we're managing it all for you

70:25

You're just making API calls

70:26

that's all

70:27

Okay, okay, sure

70:29

thanks for your, answer. The last, by the way, I

70:33

the offering link does not work for me

70:34

I'm not sure if it's the only one

70:37

Thank you for the sharing

70:38

Thank you

70:40

Yeah, go ahead

70:41

Hi

70:42

First, thank you very much

70:43

I've been having a ball with everything that

70:45

you've been releasing

70:46

And

70:47

I'm really interested in how you're using UI

70:51

you're spinning up a server

70:52

you're distributing this, but how might we deploy that either internally

70:56

but I'm interested in to

70:59

sort of a customer base

71:01

If I want to make

71:03

for example, an astrologer, That person has to enter sensitive data

71:08

so it's actually really nice for me to

71:09

be able to like have that in the

71:11

same house

71:12

So there are some procedural things

71:14

You get like a star chart example

71:16

I'm not building this

71:17

but it's a good example

71:19

Like

71:20

I can have a subagent do

71:21

an interpretation that could last for like

71:23

90 seconds. It could create some assets

71:26

et cetera

71:27

When the outcome is met

71:28

it could shut down

71:29

So this gives me a way to actually

71:31

just make a really clean

71:32

implementation where all I need is auth

71:35

And then everything can spin up from there

71:38

So there are a couple of things like

71:39

MCP channels that kind of point to this

71:42

with like fake chat

71:43

for example

71:45

a way for

71:45

you to kind of just use Claude

71:48

and have a channel to communicate with it

71:50

through UI

71:51

But

71:52

I'm seeing a lot where you're just spinning

71:53

up a server inside a project that's easy to distribute internally

71:57

but how can you deploy that at scale?

72:01

I mean, we've, this is, being deployed today

72:04

It's game production, for example, Notion

72:06

external agents are built on top of this

72:08

So this is something that exists today

72:10

that you can go and use and

72:12

basically

72:13

leverage cloud managed agents through Notion and Atlassian for Jira integrations

72:18

for example

72:19

there, there's definitely ways to do it

72:20

I think if you have specific questions

72:23

we could talk, Yeah, we, we, we can talk after this

72:25

but typically you would probably build your product

72:27

your UI that you're distributing to your customers

72:29

and then calling into us for like your

72:31

actual agent

72:33

The phrase, everything is through the API

72:36

I think that's the

72:37

thank you. So you will manage all the

72:38

API calls and build your product and UI

72:41

around that

72:42

Yeah

72:44

Hi

72:45

Hi

72:46

Hi, first, thanks for a great workshop and demo

72:48

And, my question is not really to

72:51

like, the agent itself. It's just

72:53

like

72:53

I saw the

72:54

software just released the Sound N5

72:56

Yes. So I just was wondering if any

73:00

anything you guys could share with us

73:02

Yeah

73:03

It's a great model

73:06

Try it

73:09

It is supported in cloud managed agents

73:13

Go, go, go try

73:15

Hey there, if we've got, business partners that are using

73:18

like, cloud co-work today, or some of them more tech savvy

73:21

cloud co to build some

73:22

like, spot solutions for specific business cases

73:25

what do you see as sort of the

73:27

level of sophistication up level to start using

73:31

like, managed agents instead? Because this seems

73:32

like, more powerful, but the, you know

73:35

cloud platform can be intimidating to folks that aren't engineers

73:38

let's say

73:39

Yeah, I think, I think the console quick start is probably

73:41

like, the part of my demo where you're literally just

73:43

like

73:44

telling cloud the agent that you're trying to

73:46

build should make that pretty easy for

73:49

even non-technical users to come in and build their first agent

73:52

but obviously this is

73:53

an API product and is naturally going to

73:55

be a little bit more intimidating

73:57

but they're, they're comfortable with cloud code

73:58

We do have a

73:59

managed agent skill with cloud code that basically just

74:02

like

74:02

in natural language will walk you through building your first agent

74:05

and that would probably be

74:07

My advice, to try it out

74:09

I think the biggest difference is coworker

74:11

you're building personal agent. With this one

74:13

you're trying to build probably

74:14

an agent that's like available to like your team

74:16

your internal organization

74:18

or you're building a product that you're

74:19

shipping to your end customers

74:21

But if you're building personal agents

74:22

coworker is still a really great

74:24

tool to continue doing that

74:33

There's a skill to interact with our API

74:36

so that quad is probably good enough

74:38

So if you say

74:39

here's what I did in coworker

74:40

and here's a skill to call the API

74:43

make the translation

74:44

I feel pretty confident. Yeah

74:46

On five, we'll, we'll do it for you

74:50

my question is about enterprise

74:52

there are cause a lot of enterprises would

74:54

like to run these things in-house

74:56

because they have sensitive data sets that they

74:58

won't want going out. Also

75:01

they may have already set up their own

75:03

secure proxies to Anthropix models

75:06

do you guys

75:06

and also the other question was about scaling

75:08

So within the enterprise

75:10

now you can actually allocate compute to scale

75:12

it our own way

75:13

right?

75:14

so an on-prem kind of solution

75:17

have you all thought about it?

75:19

definitely

75:19

been a customer ask from a bunch of our enterprise customers

75:22

and, and we're seriously thinking and considering it at the moment

75:26

but we, we don't already have something that we can

75:28

we can share with you

75:29

but we're, we're, we're seriously considering it

75:33

Yeah

75:34

could you show the last slide

75:36

that show the QR code of the workshop?

75:40

are you looking for the repo or the credits?

75:43

Very good, okay

76:02

We have a question here

76:10

Hey

76:10

how do you CI CD the agent context?

76:16

that's a great question

76:17

so we have something called Ant CLI

76:19

which is a CLI that

76:21

basically wraps our API to make calls

76:24

and allows you to do sort of Terraform

76:26

like things where you pass in YAML

76:28

So you can put your

76:31

long term

76:33

resources in YAML in your code base and then build

76:37

CICD to basically update the ANSI

76:40

with ANSI LI using those YAMLs

76:43

every time you make a change to your

76:45

to your resource. Does that make sense?

76:48

so I assume you guys use the same internally too?

76:51

Yeah, that's how, if

76:52

if we need to build durable long-term resources

76:54

we just, we can check them in

76:59

so I think you mentioned you can

77:01

send messages whenever you want

77:04

can you still send messages if the model's

77:05

waiting for a custom tool result?

77:08

not when it's waiting for a custom tool

77:09

result. We basically pause execution

77:12

at that time

77:13

but if the model is doing other work

77:16

then we do

77:17

allow you to queue up messages

77:19

but when it's doing a custom tool result

77:21

we're expecting you to send that

77:24

so because of ordering purposes

77:25

we, we block the calls

77:27

But what happens if you do send the

77:28

message? You'll get a 400

77:30

Okay, that makes sense

77:33

there was an ask to see the QR

77:34

code slide one more time

77:38

The, the credits one, yeah

77:43

Any, any other questions?

77:45

If you're, if you're shy, we

77:46

Yeah, we'll hang out after for any

77:48

any questions you don't feel

77:50

Yeah. Yeah, I'm looking through a repo

77:57

There's a lot of interesting workshop talks

77:59

Is there anything that's missing from the

78:01

like, slides that would help us go through other sessions

78:06

like, that you can post?

78:07

Or is it pretty much self-explanatory through the

78:09

the README files? You just

78:10

like, have, you know

78:11

because you had, like, a skill that did workshop

78:14

Yeah, yeah. the other workshops came from our

78:16

Code with Cloud conference

78:18

most of which you can find online

78:20

for the version within London about a month ago

78:23

Yeah, all, all

78:24

all those talks are on

78:25

YouTube if you're interested in the other

78:27

repos that are in there

78:28

but this one should give you

78:29

like, a pretty good overview

78:30

and it's pretty similar to what we've done

78:32

at other conferences

78:34

Yeah

78:35

Yeah, so just going back to evals

78:37

can you talk a little bit about

78:38

you know, how you guys are maintaining quality

78:40

best practices around the

78:41

the eval harness?

78:44

yeah, we run a lot of evals

78:46

especially as new models come up

78:47

as new capabilities come up

78:49

We have access to the models before you all do

78:51

obviously

78:53

so one of the things that we

78:54

as a managed offering

78:56

want to make sure we do is that we

78:58

update our assumptions about the harness

79:01

as early as possible so that we can

79:02

remove things that are no longer needed

79:05

and we do all of that in

79:06

eval base. So we have

79:08

internal tooling that basically looks like

79:10

different data sets that we can then run

79:12

our different sessions. It's

79:14

because it's all in the cloud

79:16

it's easy to say

79:16

like, spin up 10,000 sessions to do this

79:19

all in parallel

79:21

and then we evaluate those results and

79:23

tune prompts and tune assumptions

79:25

remove things, add things

79:27

to make sure that for every model release

79:30

we're doing the best we can

79:31

and we're evaluating

79:33

All sorts of things for generic purposes

79:35

not just coding

79:37

So

79:37

so how does that extend into the agents

79:39

that we would build? Is there a recommended path for

79:42

for evals

79:43

or is it kind of just bring your

79:44

own solution?

79:45

I think the

79:46

the best advice we can give you is

79:47

get high quality data

79:50

to evaluate over. I think cl- cloud

79:52

cloud code is pretty good at building an eval for you

79:55

but garbage in, garbage out

79:56

We've seen that some of our initial

79:59

ideas for how to evaluate things were sort of ill-defined

80:03

and you get one hundred percent

80:05

but what does that mean in performance?

80:07

So

80:07

iterating on your data and making sure that

80:09

you're setting up the problem correctly

80:11

I know it's like not

80:13

out of the box solution

80:14

but it generally the advice that we've seen

80:16

works

80:16

And you can probably look at like your

80:18

historical sessions and like what

80:20

results you got and then

80:21

sort of try to score it and see

80:23

how the model's doing

80:25

that would be a good way of like

80:26

building your data set

80:27

looking at your historical sessions

80:32

Hi. Over here. Hey

80:33

thanks for sharing

80:34

I have a question about how to

80:36

share those agent and all the resources across all the

80:39

the entire organization

80:41

The reason I'm asking is because we use

80:43

Cloud Desktop quite a lot

80:44

We building shared skills

80:46

we building shared connectors

80:47

but for the, everything you shared today

80:49

for example, agent

80:51

memory store, or MCP channel, can we actually share them

80:55

or can we partially share them

80:57

across the organization to different departments?

81:00

So what is boundary

81:01

what is the privilege about those resources?

81:04

Are those resources bonded to my account

81:06

or how can I control this?

81:08

So what's the

81:09

what's the way to do that?

81:09

They're bound to your workspace

81:11

so

81:12

any API key

81:13

within your workspace can have access to the

81:15

resources that

81:16

So can my

81:17

can my teammates edit or modify what I'm building

81:21

or is completely controlled

81:23

right? So I, I think your, cloud platform admin

81:26

so you'll have an org for your organization

81:29

and then your admin will

81:31

be the one setting up the workspaces and

81:33

deciding who gets access to which workspace

81:36

So I think you would basically design your

81:37

workspaces depending on your access control

81:40

Correct. You would probably want to do like

81:42

Whatever group of people you want access to resources X

81:46

put them in one. So you can share whatever you want

81:48

even partially of the

81:49

you know, you build an agent that can do something

81:51

but I only want to share part of that

81:53

For example

81:53

I share the memory

81:54

to someone that shared the MCP channel to

81:56

the others. Is that possible?

81:57

Or I need to share the

81:59

entire tech stack of what I'm building?

82:01

We don't have primary controls of like this to share

82:04

this is not, but you as a

82:05

as a builder and integration would be able to control that

82:08

pretty granularly so that your end users

82:11

can have access to partial sharing

82:14

Cool. Sweet. Thanks very much

82:15

Thank you

82:16

Yep

82:18

Yeah

82:18

so

82:19

basically we can generate a lot of code

82:21

with a lot of agents

82:22

and then we're getting to the bottleneck of

82:25

reviewing that code

82:26

making sure you can actually merge it

82:29

how do you guys think about that in terms of

82:31

are you deploying, like I see you deploy every 30 days

82:35

which, might be a different definition of deployment

82:37

but

82:38

is it deploying something you can actually test

82:40

to make sure it works well

82:42

or how are you reviewing

82:44

this many lines of code from all the agents?

82:46

So the

82:47

the original workshop that

82:49

we wanted to do was actually answering that question

82:52

how do you

82:52

how do you build agents to review code?

82:55

what we found is

82:56

you can build an integration with CMA

82:58

so that on PR it kicks off One

83:02

two, three

83:03

many sessions with many sub-agents to

83:05

evaluate what has changed and find bugs

83:08

Internally, we, we have

83:09

these sorts of agents do a lot of

83:11

the uplift for us

83:12

and we've seen that that enables us to

83:15

reduce the amount of time we need to

83:17

spend

83:18

on a per PR basis and reduce the

83:19

number of bugs and keep the quality

83:22

high

83:23

but this is

83:25

partially an open question of now we've removed the boundary of

83:29

typing

83:30

and exploded the number of PRs we can make per day

83:33

hour, second

83:35

how do we keep up?

83:36

And we're still exploring that

83:37

but, yeah, CMA's part of the answer

83:40

I think

83:41

Yeah, and you can iterate in a loop

83:43

you know, to, like, review, add comments, review until it's done

83:46

but then

83:47

at some point you still need to run

83:48

the thing

83:49

to make sure it works well locally

83:51

So I think that's where the deployments can come in

83:53

where you can, you know

83:56

auto-deploy something that you can test with a custom deploy agent

83:58

like you said

84:00

it just depends, like

84:01

if you have different environments for each branch

84:03

but

84:04

You

84:04

you can run your

84:06

application in the sandbox as well and have

84:08

Claude self-validate the changes that it made

84:12

and make sure that

84:13

like, it acts as a QA agent as well

84:17

Are these, like

84:17

long-running sessions that you can go in and

84:19

actually play around on the UI?

84:22

We don't have, kind of tunneling today

84:25

but. So you would have Claude go in there

84:27

play around with a browser

84:28

That's right. And then

84:29

and validate it. And then maybe even give

84:31

you back a video

84:32

maybe

84:32

Yep, exactly

84:34

Cool. Thanks. Hello

84:40

for an enterprise company

84:41

if at all we want to maintain true

84:43

tenant isolation that we want to support

84:46

I heard that you are not supporting yet the enterprise

84:49

company, use cases probably. We're not supporting on-prem

84:54

On-prem. Yeah. But

84:55

what if I want to have

84:57

let's say I have hundred customers that we

84:59

support as an enterprise company

85:01

Can I have separate

85:03

cloud managed agent instances per

85:06

company of ours that we want to support?

85:09

Is that a viable option that you support?

85:11

Is it only one instance that we get

85:13

or can we get

85:14

one instance per company that we support?

85:17

It's however you choose to define it

85:19

You can define an agent per customer

85:22

and then that way the data will

85:24

be segregated. You can use a workspace

85:26

per customer if you want

85:27

like true isolation

85:29

you could do an org per customer

85:32

so it depends on how you want to set up

85:34

your agents

85:35

and

85:35

you would probably want to maintain that mapping

85:37

of like which customer maps to which agent

85:40

Okay. Thank you

85:41

I think, I think I have a follow-up question

85:45

So, just given the scenario that

85:49

we have two customers

85:50

and we want to build

85:52

the agent once

85:53

but

85:54

the agent gonna be shared by the two

85:56

customers

85:57

How to achieve that?

86:00

There's many ways to do it

86:02

You could do it through automation

86:03

so you could have your agent definition live

86:05

in YAML in your code and then build

86:07

tooling to

86:08

update it to both of your workspaces and basically synchronize them

86:11

or you can model everything in a single

86:13

organization and then do the modeling that way

86:17

So you could create like a single agent

86:19

across all your customers and then you manage

86:22

the mapping of like which customer is using which session

86:26

Yeah, then in which workspace I create this agent

86:31

Yeah, if you do it per workspace

86:33

then you need

86:34

we give you tooling to synchronize

86:36

a single

86:37

definition in your code to multiple places so

86:39

that whenever you update it in your code through CI

86:42

you can update all of the definitions at

86:44

once

86:45

So you can either make a copy of your agent

86:47

which is sent to your template of system prompt

86:50

model name, tools

86:52

and you can like make a copy across

86:54

every workspace. Let's say you go down like

86:56

the one workspace per customer

86:58

or you can just put all your customers

86:59

in a single workspace and then you maintain

87:01

the mapping of which

87:04

either you do again have like copies of

87:05

your agent or you use a single agent

87:07

and then have

87:09

the mapping of like which session maps to

87:11

which customer. We have like a metadata field

87:13

on the session where you can put in

87:14

like

87:15

let's say a customer ID so you know

87:17

which session mapped to which customer

87:20

Okay

87:20

that

87:21

you know like putting all the customer data in one workspace

87:24

how do we satisfy the data segregation requirement

87:31

you know?

87:31

Yeah, if you care about data segregation

87:33

then you need to put them in individual

87:34

workspaces and the resources are allocated per workspace

87:37

But

87:38

there's basically tooling

87:40

to enable you to do

87:41

each of To those things

87:43

And when you say data segregation

87:44

like is

87:45

is that like a legal requirement or is

87:46

that something just you want to?

87:48

Yes, something legal

87:49

like customer doesn't want to share data with other customers

87:53

you know, they want to dedicate

87:55

Yeah, you do a per workspace

87:57

Okay, okay

87:59

Thank you. I saw a question here

88:01

You have to speak up or

88:05

is there a way to modify the Q environment

88:09

meaning

88:11

in some cases I may need

88:13

to have higher memory

88:15

for my agent to do the work or

88:16

I need a GPU for my agent to

88:18

actually do the work

88:20

Is it supported in this or?

88:22

To, to the

88:23

today if you want to do

88:25

custom resource allocations for your sandbox

88:27

you would use self-hosted sandboxes

88:29

you would use something like Modal that has GPU offerings

88:32

and we have cookbooks and kind of

88:34

First party integration so that

88:36

you can hook up your modal

88:38

instances to run your agent loop

88:41

inside of modal instead of our provided sandboxes

88:44

which today are all the same

88:46

so if you care about different scale

88:47

you would use a different provider for that

88:49

And do you have like a list of

88:53

specifications of what the environment these agents have access to?

88:56

Yeah, you, you can find that in our docs

88:57

and you can find cookbooks to integrate with

89:00

third party providers as well in our docs

89:02

Yeah

89:02

we have a bunch of cookbooks on like

89:04

how to work with modal or EB2

89:06

Daytona

89:07

whatever provider that you want to use for

89:09

sandboxes. Yeah. Yeah

89:11

Hi. Hi

89:12

Thanks again

89:13

So out of curiosity

89:14

how to build a similar

89:16

similar version of cloud tech using CMA

89:19

in a different chat channel

89:21

especially for the multiplayer mode?

89:25

this definitely is something you could do

89:28

today where

89:29

you would on a per thread level map

89:32

it to a session

89:33

and then any time a user messages that session

89:37

you inject that as a user dot message

89:40

letting the agent know

89:41

hey

89:41

this is coming from Gabe or it's come

89:43

from Priyanka

89:44

And so everybody can message

89:46

that agent independently

89:48

You can have it share memory across sessions

89:50

so that it learns

89:51

from each other

89:52

You can give it tools to reply back

89:55

using like the Slack post tool or something like that

89:58

so that

89:59

the agent can decide when it wants to

90:01

post. You don't necessarily wanna

90:03

just mirror whatever the agent is responding

90:05

You want it to be explicit

90:07

but you could definitely model this

90:09

using CMA

90:10

And if you wanted to be like CloudTag

90:11

you'd probably do like one agent per Slack channel

90:14

and then every agent has its separate

90:16

memory and only has access to

90:19

messages within that Slack channel

90:23

Yes

90:23

Hey, thank you for the presentation

90:25

very useful

90:27

Quick question, in terms of, looping and doing long runs

90:33

in this cloud console

90:35

how do you see that?

90:37

is this something you have

90:38

like, some specific module that I

90:41

I'm not finding, or, are you, you planning to develop that

90:46

more in the future?

90:47

thank you

90:49

Do you mean how do

90:50

we provide the observability for really long sessions

90:53

or?

90:55

No, no, no

90:56

so more focus on

90:58

using this platform to build those

91:01

long running agents, like hours and hours

91:04

like plus six hours

91:05

for example

91:07

is this something that is already embedded in

91:09

what you have built in the agents

91:11

or?

91:12

I'm more curious to know

91:13

if you are developing something specifically for that

91:16

or how you see that trend

91:18

The

91:18

the outcome system that I

91:20

demoed is the best way to do really long sessions

91:24

because you define an exit criteria

91:27

and Cloud will just keep going until it

91:28

meets that criteria. And it could be a really long task

91:31

like, you know, rebuild blank

91:35

and it will just mercilessly iterate until it matches the

91:39

the rubric. There, there

91:41

there's nothing special you would need to do

91:42

to get the session running longer than six hours

91:45

It should, it should just keep

91:47

keep going

91:49

One quick follow-up. So

91:50

if I run the agent for hours and hours

91:53

and I want to understand

91:54

like, checkpoints status

91:55

the best way to that would be through

91:57

the session tab? yeah. It would be the session event stream

92:01

yeah, there's a events API on

92:03

under the session. Okay. You could have a custom tool called

92:06

like, update Status for human or something like that

92:10

and Claude would

92:11

on intervals whenever it feels like it's made progress

92:14

call that tool and give you a status

92:16

update

92:17

Thank you. Thank you very much

92:19

Hi, any other final questions?

92:21

Yes

92:23

following up on that previous question about

92:25

using this for Claude tag or similar functionality

92:28

I've been, analyzing a whole bunch of

92:32

agent frameworks

92:34

So if you get a message while the

92:37

agent is waiting for a custom tool result

92:39

for example, human in the loop

92:41

you said it would 400

92:42

Do you have

92:44

any suggestions on how you would deal with

92:46

that?

92:48

Can you do custom tools on a sub-agent?

92:52

You could do custom tools on

92:53

Cause then

92:54

cause you could basically have like

92:55

a sub-agent that's handling your custom tool

92:57

and then that way your main kind of

92:59

coordinator agent's always free for receiving

93:02

In the API

93:03

can you directly return a

93:06

Custom tools result for the sub-agents tool?

93:09

Well, the sub-agent would emit the custom tools use

93:11

and then you would

93:13

populate it back to them

93:14

So essentially you define like your main coordinator agent

93:16

and then you can say

93:18

agent A can talk to agent B and C

93:20

and then agent B is the one

93:21

let's say, processing your custom tools

93:23

and your agent B definition is where you'll

93:25

define your custom tool

93:27

and then that's the session thread that you

93:29

would listen to to get the

93:31

Is that a different session thread or the same?

93:33

No

93:33

it's part of like the same

93:35

core session thread. It's a different thread technically

93:37

but you only interact with the main thread

93:39

and we correctly delegate it

93:40

because we know where it needs to go

93:42

based on IDs

93:43

Okay. And then the main agent will

93:46

pass it on to the sub-agent

93:48

So the session is still idle even though the If

93:52

if, if a sub-agent is doing work

93:54

it's not idle

93:55

But like if the main coordinator agent is idle

93:58

the sub-agents could still be still doing work

94:01

Yeah, you, you can see that here

94:02

Okay, that's really impressive. This

94:04

gap here, the main session is

94:06

is idle

94:07

The main

94:07

the main thread is idle while the sub-agents

94:09

were doing work

94:10

And they could be waiting for tool results

94:11

Yeah, yeah

94:12

So if you're trying to do something where

94:13

you want like the main agent to be always chatty

94:15

then

94:16

I think the coordinator pattern

94:17

works pretty well for that

94:19

That's awesome

94:20

I think we're, we're at time, but we'll

94:22

we'll be just outside if you want to

94:24

come talk to us

94:25

thank you for having us

94:26

All right. Thank, thank you so much