This is f*cking insane. SpaceXAI engineer Lauren Tan: "99% of people use only a fraction of GrokBot

Transcript

0:01

personally, I think that, the best use of you

0:06

of AI is really is like a

0:08

like a pair, like someone you're pair programming with

0:11

and not someone as

0:12

not, not a

0:12

not a tool to just replace the act of writing code

0:15

or worse, outsourcing your thinking

0:19

And I think there's a tendency

0:20

like, you know, because AI is so exciting

0:22

you wanna put

0:23

AI everywhere and be seen as someone who's very

0:26

you know, what do you call it

0:28

like, on the ball, I guess, with, with AI, that you feel this pressure of

0:34

like, you know, I'm just gonna, I gotta increase my productivity

0:37

I gotta ship, like, a hundred PRs this week

0:39

and barely understand what I'm doing

0:41

and, you know, vibe code my way to a

0:44

a million ARR or whatever

0:48

but I think You know

0:51

one of the things that I've personally seen

0:53

is that although when I use AI

0:55

I save some time

0:56

you know, writing the code

0:58

I actually find myself spending

0:59

more time reviewing what the AI did and

1:02

like, correcting it

1:03

So, you know, like, I've, I've tried the

1:05

the thing that a lot of people say

1:06

like, you know, you

1:08

instead of prompting an AI to just build the feature outright

1:11

you sort of get it to

1:13

build a spec for you first

1:15

then you review that spec

1:17

and then you do that whole thing

1:18

Then it's like... Okay

1:19

so I assume you can see my screen?

1:21

Yes, yeah, we're good

1:23

So yeah, today, yeah, I think

1:25

I think the big theme for me

1:26

as I've been using agents to write code

1:29

and I'm sure a lot of you have

1:30

had the same experience as well

1:31

is how do you trust it?

1:33

You know

1:34

especially if you are an engineer that's been

1:37

writing code for a very long time

1:38

you have

1:39

a lot of

1:40

opinions and lessons that you've learned about doing

1:43

good engineering

1:45

And when you see agents just

1:46

you know, winging it and, you know, guessing, hallucinating, you know

1:51

confidently stating that they found the smoking gun

1:54

for the hundredth time

1:56

but it's actually not the real problem

1:58

you lose a lot of trust

1:59

And when you lose

2:00

when you don't have much trust in your agents

2:03

I feel like you

2:04

you really can't get the most out of

2:05

them

2:06

And for me, the parallels like with management

2:10

so if I'm an

2:11

a manager, an engineering manager of a team

2:13

and I have a bunch of

2:15

you know, I have a team of engineers

2:17

on my team, and I don't trust them

2:20

then

2:20

the mode of operation I'm going to be

2:22

in is going to be like micromanagement

2:24

right? I'll have to

2:25

spend a lot of time

2:26

looking over my report's shoulders and

2:29

checking that they're doing their work well

2:31

you know

2:31

that they're not shipping bugs to production

2:35

And so I drew this chart because

2:38

it's not, it's not a very scientific chart

2:40

but like this is how I imagine

2:43

myself and my journey through using agents

2:46

So, you know, like fast forward or back forward

2:51

or fast back, fast backwards, like a year or so when

2:56

you know, nobody was

2:57

or not many people were using agents to code

3:00

I think you

3:02

you get into this mode where you are

3:06

In very heavily in the loop with one or several

3:09

like a handful of agents

3:11

and you find yourself just constantly

3:14

you know, trying to understand what your agents are doing

3:17

and you're very, very in the loop

3:18

You're watching every single output

3:20

You are sitting there prompting

3:23

and you really can't parallelize beyond that because you don't

3:27

again, you don't have that trust

3:28

right? You can't go to a hundred agents

3:31

like swan a hundred agents when you don't

3:33

even trust the output of one agent

3:36

So over the past five months

3:38

I feel like I've really been able to

3:41

like ascend this

3:43

Trust curve, and now I'm at the point where

3:47

I actually have, this sounds kind of scary to say this

3:49

and it makes me sound like a slop artist

3:51

but I, I promise I'm not

3:53

But I actually have my agents now

3:55

auto-merging PRs for me

3:58

which is like a wild thing to say

3:59

but

4:01

like I woke up today

4:02

and there were like 20 PRs landed

4:04

and I just reviewed them on main

4:06

like they were already landed

4:08

and they were good

4:09

so how did I get to that point?

4:11

It's basically what I wanted to talk about today

4:15

and again, like, yeah

4:16

feel free to jump in if you have questions

4:18

Colin

4:20

but, oh yeah, of course, I got to show this

4:24

this chart, where

4:28

Don't, don't do that

4:30

Trust, someone requested to control my computer

4:33

probably won't do that

4:35

but yeah, so this chart, I think, I, I

4:39

I'm sharing this chart not to kind of like flex

4:41

but to kind of show like the journey

4:43

Like, so you can see like the curve

4:45

like it sort of like inversely matches

4:47

the

4:48

contributions I've been able to land at Cursor

4:51

So I joined five months ago

4:52

and five months ago

4:53

like I, you know, my first month

4:55

I was like not very productive because I was

4:57

you know, I was learning the code base

4:59

didn't know what the heck was going on

5:01

And as I got more confident in

5:04

in my agents

5:05

I've really been able to kind of ramp

5:07

up my productivity

5:09

and again, like, yeah

5:10

like last month I shipped a thousand PRs

5:13

which is ridiculous

5:15

and then this month

5:16

we're only on the 12th

5:17

I'm already at like almost 800 PRs landed

5:22

so the velocity is definitely high

5:25

and you, you

5:26

I'm

5:26

sure a lot of you will definitely be questioning like how

5:28

how much of this code is actually good

5:31

and I think, yeah, like that's definitely fair to question

5:35

but, yeah, I think

5:38

I think if you set up your agents well

5:42

you can definitely get to

5:43

a very similar level

5:46

and so I'm going to talk about how

5:47

we do that

5:51

so for me, I think, I'm curious, like, I guess, Colin, your experience as well

5:55

but, for me

5:57

I think the most important skill

6:00

that you should have in your toolbox

6:03

when you work with agents is verification

6:06

and by verification

6:06

I mean

6:07

the ability for an agent to actually run the code

6:12

or take CPU traces or heap snapshots or

6:17

you know, open an iOS simulator

6:19

whatever, you know, however your application is exposed to your users

6:24

it can do the same thing and

6:27

run it for real

6:28

and actually test and verify it will work

6:31

Because that's the thing that really

6:33

closes the loop

6:35

it doesn't guarantee your agent writes good code

6:38

but it allows them to at least write correct code

6:42

which is a big

6:42

a really big step forward for

6:44

being able to trust your agents

6:48

I will, I can share one example that we have

6:52

within Cursor

6:56

oops, where, let me open this

7:00

Let me, make my full screen

7:07

There you go

7:12

so for the, for Cursor's agent window

7:16

so this is actually an interesting story

7:18

when I joined Cursor five months ago

7:22

they're actually, well

7:24

I was supposed to join a different team

7:25

I was supposed to join

7:26

like, the cloud agents team

7:28

but then since I have a lot of

7:29

experience working on React

7:32

and agents window is a React application

7:35

I was

7:37

I was asked to basically help out with

7:40

the agent window work

7:43

but

7:45

there wasn't really a lot of

7:46

like, skills to help me

7:48

So I just found myself like

7:49

okay, agent window's gonna launch in

7:51

like, a week, right? We have a really tight deadline

7:54

and, there was, you know, I was just sitting there like

7:58

okay, I'm gonna open up the performant

8:00

the, the Chrome DevTools and just

8:02

like, take a trace, look at it myself

8:04

and try to make sense of this flame

8:05

graph

8:06

And keep in mind

8:07

I was just, like, in my first week

8:09

so I had no idea what I was looking at

8:10

no idea what, you know, I mean, I had some idea

8:12

but, you know, the

8:13

the code base was completely fresh to me

8:16

and I realized, like, my agent had no idea either

8:20

you know, like, I would take a screenshot of the trail

8:21

download trace, or send it to it

8:22

and it'd be like

8:23

yeah, it kinda looks like this

8:25

you know

8:26

and it would, like, confidently state, like, it's this thing

8:29

and then I'd try to fix that

8:30

and turns out that's not the actual thing

8:33

So this was a very

8:34

very slow process

8:35

And if you've ever done any like performance work yourself or

8:38

you know

8:39

just even

8:39

development with an agent where you don't have a verification skill

8:43

you are the verifier

8:45

right? You've, you're the bottleneck

8:46

You, you

8:47

you tell your agent to do something

8:49

and then it goes off and writes some

8:50

code

8:50

Then you open up your

8:51

you know, local dev build and then you start to say

8:54

oh, you know, it doesn't work

8:55

Then you got to copy paste screen

8:56

you know, screenshots or console errors or whatever

9:00

and then your agent like slowly kind of like

9:03

you know

9:03

works with that and then tries to understand it and

9:08

fix the thing

9:09

But then you're constantly just in the loop and

9:11

and being a bottleneck. So there's really no

9:13

way to parallelize

9:14

So the control glass skills are like one

9:16

of the first skills I built

9:18

for cursor

9:20

and glass, by the way

9:20

is the code name for agent's window that we use internally

9:23

but it's just Pressure

9:26

I guess

9:28

and so this skill

9:29

is, I guess that

9:30

that the code itself is not super interesting

9:33

Your agent can very easily make one for you

9:35

where if

9:36

if you're building an electron app or a

9:38

web app or even iOS

9:41

applications, you can teach your agent how to use

9:44

like, the Chrome DevTools protocol or through

9:48

Apple has some

9:50

utilities as well for running the simulator and

9:52

taking traces and controlling programmatic control as well

9:57

so that's really useful

10:00

but one thing I actually want to talk about is

10:03

the, this thing

10:07

or you have to read me

10:09

so this skill comes with this very unique feature called

10:13

or not feature

10:14

unique file called a feature map

10:17

And so the story then is like

10:19

I built this skill

10:20

and so now the agent was able to

10:23

actually run

10:24

the agent window and take traces and whatnot

10:28

but it had no idea what

10:30

what the agent's window was

10:31

So, you know, like someone would say like

10:34

oh, the

10:35

the left sidebar is like laggy or something like that

10:39

Or, you know, the right side, the

10:41

the PR tab is not working

10:42

And the agent would just be like kind

10:44

of flailing around. It would spend a lot

10:45

of time trying to like look up the code and

10:47

you know, where is this feature?

10:48

How do I actually get to it

10:50

on the UI?

10:51

Which made it basically completely useless

10:54

you know, like we would, I would run To scale locally

10:58

and, you know, it would spawn a dev build

11:01

but then it'd just be churning

11:03

like, I'd just try to click here

11:04

It, it wouldn't know how to get to things

11:07

and it was just an awful experience

11:09

so, who is putting arrows on my screen?

11:13

so, Yeah, this, this feature map has been really useful

11:18

because it teaches the agent how to get

11:20

to all of the features that you have

11:23

and in PSAC, the plugin that I

11:26

I've made, if you search for PSAC cursor on Google

11:30

you'll, you'll find it

11:32

but there is a create verification skill in that plugin

11:35

where it actually helps you

11:37

set up something like this for yourself

11:39

including the feature map. So it will actually

11:41

explore the code

11:42

and build up this initial feature map

11:44

that tells your agent

11:45

how to get to

11:46

all of the different features that you have

11:49

and this is extremely powerful because

11:51

now that you have these user reports that come in

11:54

you

11:55

you can actually map

11:56

even like a vague report

11:58

or even a screenshot

11:59

So we have this

12:01

internally at Cursor, where, we have a Slack channel with

12:05

you know

12:05

lots of

12:06

people giving us feedback on the agents window

12:08

and Rockbot and whatnot

12:11

and oftentimes the report is very bad

12:14

like very low quality. Like someone will just put

12:16

very often we get like a screenshot

12:18

like, and then someone just says

12:19

question mark, question mark, question mark, like, what is this?

12:22

And, you know, like, without this, your agents are like

12:24

I have no clue

12:26

right? But with a feature map like this

12:28

it has a lot more

12:30

context and understanding of how to actually navigate

12:34

how to get to all the different features

12:36

so like, you know, example, like, I guess, like the sidebar, like

12:39

what is the sidebar?

12:41

you know

12:41

like all the different sub features

12:42

that are present in it

12:45

like from the user point of view

12:47

here's where to, how to get to it

12:48

all the different keyboard shortcuts

12:51

even like the, the

12:53

what do you call it?

12:54

The DOM elements or yeah

12:56

like the attributes that you use for selecting

12:58

things through the CDP

13:01

are all there

13:02

So, again, yeah, this is like really

13:05

really powerful, for, for agents

13:09

and a P stack ships

13:10

that create verification skill

13:12

but also a maintain verification skill

13:15

so you can keep this up to date

13:17

Cool. Yeah. I was just going to ask

13:18

how you created that. So do you mind

13:19

sharing a little bit more about

13:22

that

13:22

that process in the context of P stack

13:23

and maybe just what P stack is for

13:24

the folks who aren't familiar?

13:27

Yeah, so PStack is pretty interesting because

13:30

well, first of all, the name is kind of goofy

13:32

Like, the P, the P in PStack is like potato

13:36

potato stack, because I, so, there is a pretty

13:43

famous person, Gary Tan, who is the CEO of Y Combinator

13:47

and he's come up with this plugin called GStack

13:51

GaryStack, and, funnily enough, we share the last name

13:55

we have no relation

13:56

but I thought it'd be funny to kind of

13:58

you know

13:59

poke fun at Gary and make PStack my version of

14:02

of, of, of his plugin

14:05

but kind of just tailor it to

14:06

my own set of prac

14:08

engineering practices

14:11

but I honestly actually never set out to

14:12

build PStack

14:14

it just started with a bunch of skills

14:16

right? Like, I started with that control glass skill

14:18

and then I started with another skill

14:20

like, called HAL, which I also noticed through

14:23

like, observing agents

14:26

so Like, you know, in the early days of me

14:28

you know, trying to climb this ladder

14:29

I was like super in the loop

14:30

and I was

14:32

basically nitpicking my agents to an extreme degree

14:35

I was like, I would tell it

14:38

you know, this feature has stopped working

14:39

Here's a bug report

14:41

Like, why isn't it working?

14:44

And very often, the agent would just like confidently state like

14:47

"Oh, it has to be this

14:48

right? It has to be this thing."

14:50

And I noticed like when I looked at

14:52

the actual tool calls

14:54

I noticed it wasn't actually reading the code

14:56

that I thought should be affected

14:59

And that made me just extremely suspicious

15:01

And at that point

15:01

I was just like

15:02

I'm not gonna

15:02

I can't trust any this agent anymore because it's just

15:05

it's just completely hallucinating

15:07

And I think I think it's very easy to just

15:11

you know, like build up that distrust and not

15:14

and kind of feel helpless

15:16

like, you know

15:16

you don't know how to help your agents succeed

15:19

But like, again, I think the, the

15:21

the management analogy is super helpful because like

15:23

imagine if you were a manager of an

15:25

engineering team

15:26

and you had

15:27

an engineer on your team who was a really good coder

15:30

no business contacts whatsoever

15:31

you know, they, they just, you just hired them and they

15:34

they onboarded, you know, like five seconds ago

15:38

and so how do you actually teach that

15:40

person to be effective?

15:42

So how you do that is through a

15:43

skill

15:44

a skill being just

15:45

you know, it's just marked down

15:46

right? But, you know, and, and codes a lot of information

15:49

instructions, a lot of

15:52

you can really draw out

15:53

a lot of Intelligence from an agent by

15:57

well, some people on Twitter call it like

15:59

you know, pull the agent to a different latent space

16:02

which is kind of like a fancy way of just saying

16:04

like since, you know, LLMs are sort of like

16:06

they predict the next token

16:09

when you give it some high quality tokens

16:11

to begin with, then, you know, it

16:13

it can kind of pattern match on

16:15

like a higher space that's

16:17

you know, smarter

16:19

so that's like a very interesting model there

16:23

But yeah, I built Peaceback very

16:24

very incrementally

16:26

so, started with just really observing how agents

16:30

you know, all the fail, different failure modes of

16:32

of that agents were having

16:34

And every time I saw that

16:35

I just, okay

16:35

I'm just going to make that a skill

16:37

Right, like stop hallucinating, actually go and search up

16:40

look up the code

16:42

use a lot of sub-agents

16:44

and yeah, stop guessing

16:49

Yeah, that makes sense. One

16:50

one kind of follow-up question here

16:51

both from myself and from a bunch of

16:52

people in the chat. So

16:54

I guess it's two

16:55

two parts. So, one is like

16:56

how do you maintain these skills?

16:57

So, like the product changes over time

16:59

Obviously

16:59

there's a lot of people who should be

17:00

against the code base

17:01

So, how do these skills get maintained?

17:03

and then second to that is like

17:05

how do you know when your verification is

17:07

is good enough?

17:08

like in, you know

17:09

you can trust that the

17:10

verification loops that you've built are gonna

17:13

I guess you trust that the outputs

17:14

when, when they're done

17:18

yeah, maybe I'll talk about

17:21

I think I have somewhat really

17:22

maybe I'll start with this one first

17:24

So, like, how do I maintain these skills?

17:29

So, if you're not familiar with this concept

17:32

an eval is essentially like a way to

17:35

well, I think the mental model I have is like

17:37

it's like a unit test for an agent

17:40

and, you can actually make your own evals

17:43

You don't need like a special framework for

17:45

them

17:45

You can build, you can

17:46

you can build one depending on like

17:48

you know

17:49

how

17:49

scientific and how rigorous you want to be

17:53

my screen is red

17:55

Yeah, there's a little button

17:56

sorry

17:57

Oh, am I like disabling the drawing or something?

18:00

I, I can't see my screen

18:01

Yeah, sorry. If you guys could not draw on the screen

18:03

that'd be great, but, there's a little button in the

18:06

yeah, the, the little drop down

18:10

yeah

18:12

Okay, yeah. You got it

18:14

perfect

18:15

Good to know

18:17

Yeah, so evals are a way to unit test your skills

18:21

basically

18:22

And actually, in P-Stack, we ship

18:24

under potato mode, there's a playbook, if you search for it

18:27

called eval playbook

18:29

and it's

18:31

it's like not, it's actually pretty

18:34

pretty rigorous, the way it's done

18:36

but

18:37

essentially what I do

18:38

is I spawn a lot of different sub-agents

18:41

I have, like, my main coordinator agent

18:44

come up with a rubric for

18:47

what I want the skill to do

18:50

And then it spawns all these sub-agents

18:51

and it, it creates individual directories for them

18:56

which are cleverly named to not

18:58

let the sub-agent know that it's being evaluated

19:01

because, agents can actually tell

19:04

and when they do

19:05

they change their behavior

19:07

but it does a bunch of stuff like that to

19:11

essentially, yeah

19:12

like test whether or not the skill

19:14

I'm making or changing is actually doing

19:16

what I think it does

19:18

and one of the really nice things about

19:20

Cursor is that we are

19:21

we, we support so many different models

19:24

so

19:24

you can actually eval your skill across all

19:26

sorts of different models

19:29

and, you know

19:29

get a sense of how well it performs

19:31

across that different matrix

19:34

especially for the models that you use

19:38

so I do this a lot

19:39

Every time I, I modify a skill

19:40

I will run one of these

19:42

like the email playbook

19:44

and make sure that

19:45

you know

19:46

it's actually leading to a result I want

19:49

but I will say

19:50

like, maintaining skills is actually pretty hard

19:54

it requires, I think

19:55

a lot of taste and observation

19:58

So you kind of need to

20:00

be very good at being a backseat driver

20:02

You know what I mean?

20:03

Like, if you do pair

20:04

if you've ever done pair programming

20:05

for example, and you watch a coworker code

20:08

and you just like

20:09

you, you could probably do this better

20:10

You know, you could do, you know

20:11

like, why did you not do this

20:13

right? You, you ask a lot of questions to your coworker

20:15

and it's kind of a similar thing here

20:17

Like

20:17

you don't want to just be a passive

20:19

observer of the agent. You want to be

20:20

very

20:21

in the driver's seat in the initial stages

20:23

when you're building up your own set of skills

20:26

you know, obviously you can use something like GSAT

20:28

but if you're building your own set of skills

20:30

it's very, I think, you know, opening up the

20:33

all the tool calls and like reading the

20:34

code and Reading all the

20:37

the agent behavior and their thinking blocks

20:40

is a really great way to see where they

20:43

they fail, right? Like what, what

20:46

you know, where are they being done?

20:47

And then you can go and build a

20:48

skill for that

20:50

And then with verification

20:51

how you trust it is

20:53

it's, I think it's also a very similar iteration loop

20:56

where, you know

20:56

like I actually did the same process for

20:59

verifying the verification skill

21:01

where I actually get

21:04

So one thing that's interesting about evals is

21:06

that you can sort of hill climb them

21:08

meaning that, your eval can produce a score

21:12

right?

21:12

a score that you can get your coordinator to produce

21:16

but also

21:17

you can have a judge agent of a different model to

21:21

kind of

21:22

cross-reference and make sure that the

21:24

first model is not being biased

21:26

right? The model that's judging all of the

21:28

sub-agents that are running the thing

21:30

but you can also

21:30

like, hill climb, so meaning that you can

21:33

you can use, like, slash loop in cursor

21:36

and you can say

21:36

okay, keep looping on this eval

21:39

right, until everything is ten out of ten

21:41

as an example

21:43

and I did the same

21:44

the basically the same approach with the control skill

21:46

and so I kind of

21:47

it was very, it was very hands-off

21:48

actually

21:49

so, you know, I, I kind of built

21:52

I built that skill that way

21:53

like the CLI in that skill

21:56

and over time, it's gotten really good

21:59

but yeah

22:00

it was definitely not super smooth

22:02

at the beginning. It required a lot of

22:04

iteration

22:05

And I think there's an analogy here for me

22:08

which is, well

22:10

I make this analogy later in a different

22:12

slide on my drawing here

22:14

but I think of it like

22:17

you know, as a, as a engineer now

22:19

you're sort of more like

22:22

like maybe a manager

22:23

or the analogy I like is like you're like a

22:25

a chef in a restaurant

22:28

you know, you're the head chef

22:29

you're not cooking all the food yourself anymore

22:32

You have a team of cooks

22:33

right? You have line cooks

22:34

you have a sous chef

22:35

you know, all these different stations

22:39

and it's your job to really design the environment

22:41

You know, you

22:42

you're in charge of setting up the kitchen

22:43

You're in charge of

22:45

you know, like giving tasks to different people

22:48

So

22:51

yeah, it's a very interesting way of working

22:55

but yeah, that's, that's how I basically built

22:57

these verification skills

22:59

Yeah, just, just one follow-up there on like

23:01

I'll have to go

23:02

try to go one layer deeper

23:03

So are you, let's say we wanted to build

23:07

an eval or a skill for

23:08

for something

23:09

and we wanted to kind of get better on its own

23:11

which is, is what I think you're suggesting

23:14

are you doing that in like a work tree

23:16

kind of isolated with like the sub-agents and

23:18

and then the reviewer agents and

23:20

and all that? Is it happening like in

23:22

some type of cloud-hosted environment?

23:24

Like what's the, the more, the practical steps?

23:25

If I wanted to go do this

23:27

and like set up a verification system for something

23:29

what would I

23:29

what would I do or where would I

23:30

start?

23:34

I think that, the best place to start is local

23:37

because you can observe

23:39

You can definitely observe what your agents are doing

23:41

So, if you're building a verification skill for yourself

23:45

I would definitely start local and

23:47

just have your agent bring up the application

23:49

whether it's like a CLI or

23:51

a desktop app or whatever

23:53

And so you can actually observe

23:54

right? You can see how the agent is interacting with the

23:58

the application

23:59

You can see it

24:00

you know, how it calls like the different APIs that

24:04

that allow it to interact with the

24:06

the application

24:09

but, for me personally

24:12

I have basically been kind of all in

24:14

mostly all in on cloud agents because they're

24:16

extremely powerful

24:19

and the really powerful thing about Cursor is the

24:21

the cloud agents actually

24:23

where if you spend a little bit of

24:24

time setting up your environment

24:26

These control skills

24:28

these verification skills pay a

24:29

huge amount of dividends

24:31

because it's not just something that

24:33

makes you as a single engineer better

24:36

it actually levels up your whole team

24:38

and even your whole company because

24:41

you can actually start thinking about cloud agents

24:43

you can start thinking about automations that automatically do things like

24:49

I'll, I get, I

24:49

I kind of talk about this a bit later

24:51

but I'll just kind of Get into it

24:54

where, where, you know, for example, like

24:56

I talk a lot about this agent we have called Benny

24:58

right, who, who, you know

25:01

takes all of the bug reports that we get

25:04

and it automatically goes off in the cloud

25:06

opens up a cloud

25:07

its, you know, its desktop

25:09

It runs Cursor in its own computer

25:12

and it uses the same control skills

25:15

to interact with the application and try to reproduce the bug

25:18

or the user report

25:19

right? And this is so

25:21

so powerful because at once I can immediately

25:23

I

25:23

I get so much information from this automatically

25:26

Like here in this example

25:27

you can see that

25:28

the Benny actually reproduced the bug

25:31

but it's already fixed on me

25:34

So it actually confirms that we fixed this problem already

25:37

and all I need to do is just

25:38

release another build of

25:40

of Cursor

25:42

so that's like huge information

25:43

there that I didn't have to go off

25:45

and sit with an agent

25:46

you know, and spend an hour trying to figure

25:47

like, is this fixed? Is this not fixed?

25:50

So you, you, you gain back so much time

25:53

You know

25:53

everybody on my team benefits from this

25:55

Everybody in the company benefits from this

25:58

so definitely think that

26:00

you know, keeping these

26:02

using cloud agents is super powerful

26:05

but yeah, it's like a journey

26:06

You have to trust it first

26:08

right, before you, you get to this point

26:11

And that's, it goes back to what I was saying here

26:12

where, you know, it's very hard, it's, it's almost impossible

26:16

and I would

26:16

definitely encourage you not to try to jump from

26:19

you know, like, if you're still in this zone

26:22

you don't want to jump to

26:23

like

26:23

I'm going to spawn a hundred

26:25

or thousands of cloud agents right now

26:27

because you're just going to waste a lot of tokens

26:31

and it's going to be extremely expensive

26:33

Yeah, so just to kind of recap so far

26:35

basically the

26:36

if we wanted to go on the journey

26:37

that you've kind of gone on

26:38

it would be just start with verification

26:41

building some, some skills and some

26:43

some ways of determining that the agents are producing at least

26:46

like, correct code, whether, like you said

26:47

whether it's good code or not is maybe a separate question

26:49

but, like, it's, it's technically solving the problem by looking at

26:52

you know, Stack traces, looking at, you know

26:54

the actual behavior in the app and so

26:56

on

26:57

And then once we trust it locally

26:58

then we can start to think about scaling

27:00

into the cloud and running more

27:02

agents that are picking up signals

27:04

I guess, on their own, right?

27:04

So whether that's like a bug report that

27:06

comes in or something

27:07

and they can go and pick it up

27:08

and

27:08

solve the problem

27:09

and give us back a PR

27:11

And then maybe the last step is like auto-merging the PRs

27:13

which is where you're at

27:15

and maybe not where I want to go

27:17

And then reviewing them on main

27:18

But is that about right?

27:21

Yeah, exactly. I think, yeah, that's why I do this

27:23

this, this, this curve, right?

27:24

Because that, this, this basically describes my journey of

27:28

you know, when I started, barely could use a couple agents

27:31

and I was just observing every single thing

27:34

I think there's really no shortcut for going

27:35

from here to there

27:37

because this is really about your personal level

27:39

of trust in agents

27:41

right?

27:42

obviously, you know, as a, as an engineer

27:43

you don't want to just slot code into

27:44

production

27:46

So how do you actually build up that trust?

27:48

Takes, a lot of, I guess, taste and judgment

27:53

but, you know, like

27:54

I think plugins like PSAC definitely can help you

27:58

get up to speed much quicker

28:01

and so I guess it's like

28:03

if you trust me and you trust PSAC

28:05

then in, by extension

28:07

you can maybe trust your agents

28:08

But if you don't trust me

28:09

and I

28:10

I definitely would not encourage people to blindly trust me

28:15

you know

28:16

if you build up your own set of

28:17

skills that you can obviously

28:18

you know

28:19

take a look at PSET and kind of fork it

28:21

make it your own

28:22

improve the skills, definitely encourage that

28:25

but for me, it's really all about

28:27

it, it, it just keeps coming back to trust

28:29

You know

28:29

every one of us

28:30

here in this chat have a different standard for engineering

28:34

and there are different things that

28:35

are important for us in our codebase

28:37

and

28:38

when you

28:40

are able to encode all of that into

28:41

skills and you can verify that your agent

28:43

is actually doing them

28:45

that allows you to really kind of ascend this curve

28:47

and, you know, start automating things

28:52

there's another piece I wanted to talk about

28:55

if there's

28:56

Yeah, go for it. I'll

28:57

I'll think of more questions as I go

28:58

Yeah, I think there's a

28:59

a third part to this

29:00

which I haven't talked about yet

29:01

which is kind of an interesting one

29:03

which is like refactoring and rewriting

29:06

Like one of the

29:07

I guess most controversial

29:09

one of the most controversial topics in the industry

29:12

I think, is like

29:13

should you rewrite your app or not?

29:16

because I think engineers are very prone to this

29:19

where, especially when you join a company

29:21

you come in and you see like the

29:22

code base and you're like

29:23

oh man, this is shit

29:25

Like who wrote this code?

29:26

You know, it's, it's terrible

29:27

I want to rewrite the whole thing

29:29

There is a very common inclination and

29:31

I think a lot of

29:33

you know, before agents, and I guess arguably even now

29:36

people will definitely discourage you from re-rewriting stuff

29:40

But I'm actually here to make a case

29:41

for why you might want to consider it

29:45

because

29:47

I think it really depends

29:49

you know, Brownfield applications, I think

29:51

are actually in a pretty good spot

29:53

especially if they're set up well already

29:57

and like recently I've been talking to some people

29:58

but, you know, I, I was just observing

30:01

I, I just noticed this parallel

30:05

which is that

30:06

a lot of big tech company problems are

30:08

now everybody's problems

30:10

and the big tech company problem

30:12

you know, like when I was working at Meta

30:13

like we had this giant monorepo

30:15

we had like, I don't know

30:17

tens of thousands of engineers just

30:19

you know, like banging on their keyboards and

30:21

and shipping code

30:22

And A lot of really great engineers at Meta

30:27

but, I'll say like, you know

30:29

you'll be surprised that the code quality is

30:31

actually not that good

30:33

and so I often joke that like

30:35

you know, before AI slop, we had human slop

30:38

and so, you know

30:39

I think a lot of big tech infra

30:41

like, like what Meta has or Google

30:44

you know, you know

30:45

really big tech companies are actually designed for that

30:47

where you're, you're sort of like

30:49

you're catering to the

30:50

the, you know, like, sounds, this sounds so bad to say

30:53

but like the, the least capable engineer on your team

30:56

right? You build, you build frameworks, you build conventions, you build guardrails

31:01

you know, you restrict credentials so that

31:03

you know, your intern doesn't wipe your production database

31:08

There's, you know, if you have that level of infra already

31:12

I think your agents can actually already do

31:14

a very solid job

31:15

right? Because they have

31:17

the

31:17

the guardrails are already in place for

31:20

agents to not cause havoc or not cause too much havoc

31:23

in your codebase

31:25

and you can always add more

31:27

you know, guardrails

31:29

but I think like Greenfield applications

31:30

especially are, you know

31:31

like the brand new applications are like the biggest risk

31:35

in my opinion, and also the greatest opportunity

31:38

Because, you know, if you vibe code a project

31:41

a prototype, like we did for Grokbot

31:44

you know, Grokbot was spun up very

31:45

very, very quickly

31:47

and if you, if you haven't heard of

31:49

of Grokbot, it's like a

31:50

a new application we just launched yesterday

31:53

it's, it's really cool

31:54

lets you orchestrate your

31:57

create like individual agents that have their own identity

31:59

and you can kind of orchestrate them

32:01

It's super cool, definitely check it out

32:04

but yeah, that was, it's like a very

32:05

it was a very greenfield application

32:06

like most prototypes are

32:09

So it was like vibe coded very quickly

32:10

Humans were not reading the code at all

32:13

And, I had this tweet recently

32:17

where I said something about organic architecture

32:22

maybe I'll find it

32:24

but the idea is that

32:28

when you have a completely vibe coded application

32:30

you essentially have no guardrails whatsoever

32:32

So, your agents When you give them a task

32:37

they will just solve it in whatever

32:39

method is the most convenient

32:41

And over time, you get into this

32:44

situation where you have a code base that

32:46

is spiraling out of control

32:48

because you don't understand it

32:50

your agents understand it

32:52

I guess, in a way, but, like

32:53

they've built something that is

32:55

you know, optimized for short, for shortcuts

32:58

and, you know, it will, you will suffer, you have a lot of

33:02

of issues with that application

33:05

so I think starting your code base with

33:09

like, very strong constraints is Very much needed

33:14

because like when you have a code base

33:16

that you can trust

33:17

right, when you have guardrails that actually help you

33:21

help your agents, right, good code, you can get into the

33:24

you know, like into this part of the curve where I

33:27

I, where I, like I, I said, you know

33:29

I woke up today and I had like 20 PRs merged

33:32

by my agents, and that's because I invested a lot

33:35

a lot of time

33:37

over 600 PRs, I

33:39

I, I calculated yesterday

33:41

when I refactored all of GrokBot to this

33:44

new architecture that I've been building

33:48

and yeah, I've, I've gotten to a point where I

33:51

I don't really look

33:52

I really don't look at the code anymore

33:54

and, I say that not just

33:56

you know, to sell you tokens

33:57

but Because I, you know

34:00

it

34:00

took a lot of work to get to

34:01

that point. I spent a lot of tokens

34:03

to get

34:03

the codebase to this point where I no

34:05

longer have to look at it

34:07

But I'm very excited because

34:09

you know, of the potential where

34:11

you know, it's not just

34:12

it just doesn't just benefit me

34:13

It benefits everyone contributing to GraphBot

34:17

and it also empowers

34:18

you know, designers and product managers and

34:21

you know

34:22

even GTM people to add features to GraphBot

34:25

And I don't have to worry

34:26

you know

34:26

I don't have to wake up

34:28

at night in the middle of the night and worry like

34:30

oh shit, someone's just merged a perf regression

34:32

right? I have a ton of constraints and CI

34:35

It's like, it's actually very annoying to write code in GraphBot

34:39

but like agents absorb all of that annoyance

34:43

But yeah

34:43

I'm happy to talk about what exactly that is

34:47

Yeah, I think one question

34:50

Before we get into the

34:51

this part here is

34:52

just around that element of like what your

34:55

your

34:55

your CI looks like or maybe some of

34:56

the constraints and then also like the average

34:58

PR size. I saw a question about that

34:59

earlier

35:00

Just to give people a

35:01

you know, kind of a glance

35:02

It doesn't have to be like mathematically average

35:04

but just a, you know

35:05

like what

35:06

generally the size of the PR is

35:09

If it's only a couple lines of code or

35:10

you know, yeah. I think it depends

35:16

Let me I'm

35:18

trying to do this in a way where

35:20

I'm not going to like

35:21

Yeah, you don't have to share the actual number of it

35:23

like the actual average. This is fine

35:25

But like we have

35:27

so okay, this is not that interesting

35:30

but well

35:30

fun fact

35:31

is that virtualization in Grokbot and in

35:36

Cursor is actually powered by pretext

35:39

which is a

35:40

sort of new library that someone's built

35:44

That's really interesting. You should

35:45

you should check it out

35:47

But that's not really that important

35:48

I think the average PR size

35:50

I actually don't

35:52

know if I want to click on these

35:54

I probably can

35:56

but I would say like

35:57

they can range anywhere from

35:58

a few hundred lines or 50 lines to like a thousand

36:02

depending on what the thing is doing

36:05

so like here, I'm actually like deleting a bunch of files

36:07

so I expect that it's just like mostly deletion

36:11

but yeah, it kind of varies

36:13

There's no like hard cap or are

36:16

they all like 50 line PRs?

36:18

There's no hard cap

36:18

but

36:19

I do encourage my agents to split up

36:20

their work into multiple PRs

36:23

I do that mostly because

36:27

I like, I like the idea of

36:29

I guess maybe this is much harder to

36:30

do now as

36:31

in the world of agents and you have

36:33

like so many commits

36:34

But I like the idea that

36:35

you know

36:35

the git history is a very rich source

36:37

of context

36:39

and I like the

36:41

I like each PR to sort of

36:42

atomically describe what that Small piece of things doing

36:47

which also makes it easier for me to

36:48

revert changes and like figure out

36:50

you know, oh, I shipped a bug and it's just

36:52

it's here

36:52

right? It's not in this

36:54

40,000 line PR where who knows what

36:57

landed in there

36:59

but I

37:00

I don't have a hard cap

37:01

on PR size

37:03

Cool. And then, yeah

37:05

also quick question on like CI

37:07

So again, you don't have to go into like

37:08

the screen share like your

37:09

your CI does, but just generally

37:11

would you describe what the CI kind of looks like

37:14

or how strict it is?

37:17

yeah. So, well, specifically for Grokbot, so Dune is the

37:22

is the sort of cheeky code

37:24

code name for the architecture that we built

37:26

for Grokbot

37:28

the CI looks pretty annoying because there's checks for everything

37:34

So, like literally I have

37:37

well, if you've written any React

37:38

for example, you know

37:39

you know that one of the biggest foot

37:40

guns in React is useEffect

37:43

so in Dune

37:46

and in Grokbot we've banned use effect

37:48

So Dune is just an

37:49

you can, the, the mental model of what Dune is

37:52

you can kind of think of it as like Next.js for

37:55

electron apps, and it's designed for agents to write

37:59

and it's like custom for

38:00

you know, our agent powered applications

38:04

so the CI checks are very like specific to that

38:06

like, you know, don't use use effect

38:08

It's, it's, it's banned, like CI will fail

38:11

and yell at you

38:12

We have, like

38:14

some of the more interesting ones that people

38:15

might raise eyebrows is like

38:17

I actually ban code comments as well

38:20

which is very interesting

38:22

but I've noticed that

38:24

99% of the time

38:25

agents just write code comments that

38:28

kind of describe some historical thing that is

38:30

actually totally irrelevant to the code

38:34

like it will often say like

38:34

you know, oh, Lauren said you should never do this

38:36

and it's now in

38:37

in a code comment. I'm like

38:38

what?

38:39

Like, why? That, that was

38:40

I didn't say that as like a durable

38:42

you know, global rule. I just meant like

38:44

Your

38:45

this PR sucks and you should change that

38:47

part

38:48

agents don't really understand us that well

38:51

surprisingly

38:53

and or they kind of assume too much

38:55

and they kind of do things in like

38:56

very stupid ways

38:58

So like, yeah, we just ban everything

39:00

Everything you can imagine like the agents are bad at

39:03

we ban

39:04

so

39:05

one example that we actually suffer a lot

39:07

in the agents window is we have

39:11

you know, if you've used agents window

39:12

you've definitely seen performance issues and

39:14

you know, we're constantly trying to fix them

39:16

but it's like a

39:18

It's a never-ending struggle

39:20

because there's so many pull requests that get

39:22

merged

39:23

Any one of them could just regress

39:25

performance or stability or reliability

39:27

you know

39:28

the agents window doesn't have this architecture yet

39:30

I plan to do

39:31

bring this learning back there

39:33

and kind of refactor everything there

39:36

but, it just regresses super often

39:39

because, there's, just one example is like

39:42

we have very poor

39:44

isolation between processes

39:46

So

39:47

like on, you know, on, on Electron

39:48

you have a renderer thread that renders your UI

39:51

but you also have like a main thread

39:52

that you can run other code that

39:53

you know, doesn't need to block the renderer

39:57

but we do a poor job of separating those things

40:00

and so oftentimes you just accidentally

40:02

have code that gets pulled into running on the renderer thread

40:06

right? And then all of a sudden you're competing with the

40:08

the renderer that, you know, that has a very

40:11

if you want like 60 FPS

40:12

you have to

40:13

every frame that gets drawn

40:15

has to be done in 16 milliseconds

40:17

So very, very small. You know

40:18

deadline per frame

40:20

if you want, you know

40:21

a very smooth product

40:23

and when you start building

40:24

bringing in, accidentally bringing in, you know

40:26

things that are

40:27

like very computationally heavy or they have a lot of IO

40:31

then you just get into like a lot

40:33

of jank

40:34

And your, your FPS really drops

40:36

you start, you know, losing frames

40:38

you get long tasks that take more than 16 milliseconds

40:41

and then you just get this really choppy

40:43

experience

40:44

So all of those patterns that we've learned

40:46

basically building electron apps

40:48

we've encoded into this framework

40:50

and it becomes like a hard failure

40:52

So I literally, in, in Grokbot

40:54

we literally have a directory called electron main

40:57

electron renderer, and we have, import CI, I guess

41:03

where we actually

41:04

check the dependency graph to make sure they're

41:06

not accidentally importing

41:08

code from one directory to another

41:10

so that's enforced by CI

41:13

as well as BugBot

41:15

which is our, which cursors

41:18

like code review tool that runs on CI

41:21

you know, in our agents MD

41:22

it's everywhere, like, so I, I, I

41:26

I have this thing here where I talk about like

41:29

you know, like there are multiple layers

41:31

I think, for building a good code base

41:34

obviously the code base is one where

41:37

if you have an architecture like this

41:39

where it's extremely strict

41:41

you know, the, the

41:42

the way to build features is very conventional

41:44

that's like the strongest

41:46

strongest level of enforcement

41:48

because agents just love to copy existing patterns

41:51

So

41:52

one example of this in Robot is like we have this

41:55

these concepts called like a feature

41:57

and we have entry points and transcript cards

41:59

like, oh, you know

42:00

the cards that you see in the chat

42:02

These are all like

42:03

like

42:04

nouns, I guess, in, in the framework

42:06

and so there's a very conventional way of creating them

42:09

and So like a feature is all in

42:12

in a single directory as an example

42:14

And so all of the code that contributes

42:16

to that feature lives in one directory

42:18

So it's all co-located in one place

42:20

Makes it super easy. You know

42:21

agents don't have to like

42:23

grab around and try to figure out like

42:24

where all the things are

42:26

It just looks at the feature and like

42:27

oh, okay

42:28

I'm working on the onboarding feature in Grokbot

42:32

I'm just going to work in this directory

42:33

And for 80% of the work

42:35

it's mostly just very encapsulated there

42:38

But, like, it's like very, it's like designed again for

42:43

you know, like the dumbest agent

42:44

like you don't have to think

42:45

right? The, the, the

42:48

one of the key principles I have for

42:49

this framework is like the shortest

42:52

the shortest path is the best path

42:55

So

42:55

because that

42:56

plays exactly to how agents love to write codes

42:59

Like, they like to take shortcuts

43:00

really

43:01

You know

43:01

they'll find the quickest way to

43:03

solve the problem

43:05

So

43:05

why not make that the best

43:07

way to solve the problem?

43:09

so, I, I

43:10

I probably won't get into all the specific details

43:12

and

43:15

this framework's really more of a collection of

43:16

ideas and principles rather than something that will

43:19

open source

43:20

you can, you can, you know, screenshot this, I guess, if you want, and

43:24

tell your agent to

43:26

do some, build, build something like this for you too

43:30

yeah, but it's really all about the layers

43:32

you know, like the

43:33

the code base is one part with features

43:36

and directories and, you know, import, blocking import dependencies, that shouldn't be imported

43:43

but, and

43:44

and it all enforces that and static analysis

43:46

So like, there's CI checks

43:49

we have a lot of lint

43:50

for bad patterns that we observe

43:53

compiler diagnostics

43:55

there's also rules and bug bot

43:57

which are, I think like three

44:00

four, five are more soft

44:02

right? These two actually make

44:04

make CI red, right?

44:06

So that, you know

44:07

there's a hard constraint where the agent can't

44:09

just write crappy code

44:12

For rules and skills and bug bot

44:15

Your agents can still forget

44:17

right? You can still

44:18

or it may not always consistently apply them

44:21

So I like to layer them

44:24

but I don't

44:25

I don't like to

44:26

rely on them as the only source

44:28

of enforcement because it's very

44:30

very soft, right? And if you

44:31

if you only have rules and bugba and

44:33

skills and a style guide for your code

44:35

you will

44:37

it's only a matter of time before your

44:38

code base looks like complete trash

44:41

sorry to say that

44:42

but, I definitely recommend, yeah, like, you know, investing

44:45

in, you know, things that can be hard enforced

44:48

right? And this is why

44:50

you know, maybe

44:51

the choice of tech stack that you use

44:52

is also very important

44:55

like I think, for example, Rust is sort of making

44:58

you know, it's like getting super popular again

45:01

because the compiler is so strict

45:03

right? The compiler enforces so many different things

45:06

you know, there's a borrow checker that you have to appease

45:09

and if

45:09

as long as you make sure your agents

45:10

don't write unsafe code blocks

45:13

you can more or less feel

45:14

somewhat confident That if the code compiles

45:17

it probably works and it's good

45:19

But it gives you that level of

45:21

trust and confidence

45:23

that

45:23

you

45:24

as a human engineer no longer

45:26

need to go and check it yourself

45:28

You know

45:29

you rely on

45:30

code and static analysis to actually

45:34

make that a lot smoother

45:38

And I guess the worst part

45:39

the worst

45:40

place to be in is if

45:42

you are stuck in code review land

45:44

where you actually enforce all of the constraints

45:46

the invariants in your code base by

45:49

literally the human person saying

45:51

you know, reading the code and like

45:52

okay, you should not do this

45:54

right?

45:55

Every time you have to do that

45:56

you should consider that as a code smell

45:58

like a anti-pattern, and you should say

46:00

okay, instead of me commenting on the PR

46:03

how do I turn this into a hard rule

46:06

right? How do I turn this into a

46:07

lint rule?

46:08

How do I turn this into a CI

46:09

failure?

46:10

Or how do I even categorically eliminate this problem entirely?

46:17

I, I can talk about another migration I've done

46:18

but I'll probably pause here

46:20

Sure. Yeah, I feel like that's

46:22

that's where I am

46:23

to be honest, is, is what you're describing right now

46:25

which is that, like

46:25

I don't have all of these rules

46:27

so I have some things to go do after this session

46:30

in terms of being able to scale my agents

46:31

I'm, I'm definitely on, like, the

46:34

you know, maybe a couple of parallel ones locally stage

46:36

like two to three locally

46:38

and I'm sure most people here are on the same

46:39

So, yeah

46:41

I know we only have a couple minutes left

46:42

Lauren, was there anything else that you wanted to

46:44

to highlight?

46:45

I, obviously, there's lots of questions

46:46

so I can grab more

46:47

but I want to give you a few minutes

46:49

if there's anything else you want to talk about

46:50

Well, I think I've been yapping for quite a lot

46:51

so I'm, maybe let's just do questions

46:54

Okay, cool

46:55

one question that had

46:56

a couple of

46:57

came up a couple of times was just around

46:58

like, token usage

47:00

So the, the question is, like, is

47:02

is what you're describing a realistic thing for

47:04

people who are on

47:06

you know, a

47:07

a normal set of token usage they don't have

47:10

you know, basically unlimited tokens, to work with?

47:14

I think that's a really good point

47:15

I mean, like, obviously, you know

47:16

I work at a AI lab where we have unlimited tokens

47:19

so, I definitely cannot say that

47:24

you know

47:24

this is something everyone

47:26

should do in the exact same way that

47:28

I did it

47:28

I think it's possible to get to this point without

47:30

you know, breaking the bank

47:33

But, you know, if you're like an engineering leader or

47:35

you know, you have a startup that you lead

47:39

I think the thing is a question of

47:40

ROI

47:41

and it's like, yes

47:44

you spend

47:45

a lot of money

47:46

on tokens in the upfront stage

47:48

You know

47:49

like refactoring your code base is going to

47:50

take a lot of tokens

47:52

adding all these things

47:53

is going to take a bunch of tokens

47:55

But

47:56

if we're heading to a world where agents

47:58

are writing all the code

47:59

and, you know, you want to be very lean

48:03

right? You don't want to have to hire

48:04

you don't want to be

48:05

you don't want to become like meta

48:06

right? Like, I mean, like in terms of

48:08

you don't want to become

48:09

a 10,000 person engineering org because I mean

48:12

that's a cool problem to have

48:14

but also, you know

48:15

you have so much overhead

48:16

There's like planning, you know, like

48:19

it's, it's, personally, I, I wouldn't, it's not super fun

48:24

But, I think you want to stay very nimble

48:26

right? And you want to

48:27

you want to be

48:28

like

48:28

agents are all about

48:29

allowing you to do

48:30

things that you couldn't do before

48:32

That's really, to me, like, the value of agents

48:35

You know

48:35

it's not just throwing tokens on every single little thing

48:39

But, to me, like, the thing I couldn't do before is

48:41

like, enforce this level of constraints in a codebase by myself

48:47

right? Like, I'm just a single person

48:49

you know

48:50

it would have taken me years to build this framework

48:53

and do all the refactoring

48:55

and Test everything myself and verify

48:59

you know, like run, imagine if there, it was just me

49:01

right? No, in, in pre-agent era

49:03

just like running, you know, by my

49:05

it would take me so long

49:07

right? And my salary is pretty high

49:09

right? Like, so, you know, the

49:11

the question I think an engineering leader might

49:13

have is just then

49:14

you know, like, what is, there's a trade-off of

49:17

do you hire someone to do this

49:19

or do you spend the tokens to set

49:21

up a code base so that even the

49:23

the most naive, right

49:25

the dumbest agents can do a good job?

49:28

And when you actually get to this point

49:30

like even agents that are not

49:32

you know, fable size do an excellent job of writing code

49:36

and this pays a lot of dividends as

49:37

well for me personally

49:39

where I've empowered not just myself

49:42

but again, like PMs, designers

49:45

engineers who are not familiar with Grokbot to

49:47

just contribute in a way that is sustainable

49:52

So I think, yeah

49:53

it's definitely like a trade-off for sure

49:55

You know, like nothing's like free for sure

49:57

and tokens are pretty expensive

49:59

but oh, actually, I, I

50:02

I don't know how many of you have seen this

50:04

but we actually announced Grok 4.6 today

50:06

So very exciting, finally out

50:09

so yeah, Grok 4.6 would be like a great

50:11

it's very, very smart

50:13

it's really good on the

50:14

on the benchmarks

50:16

and it's the same

50:17

the tokens, well, hopefully I'm not saying this incorrectly

50:20

but

50:21

I believe the

50:22

cost per token is the same as 4.5

50:25

So you're actually getting

50:26

more intelligence for the same cost

50:30

I think this is an area that Cursor tries to

50:32

Cursor and SpaceX AI try to really optimize for

50:36

like that Pareto frontier of

50:38

you know, cost versus intelligence

50:41

you know

50:41

we don't necessarily want to build the biggest

50:43

model ever because that is extremely expensive to

50:46

run

50:47

It's really about like

50:48

how do you find that sweet spot

50:49

right? You don't, you don't need a giant model

50:51

but it's just super Right

50:52

and it's not very expensive for inference

50:56

but, yeah, I think to kind of round it up

51:00

I think it's like a

51:01

it's, it's, there's a, if you do your own analysis

51:04

I feel like it's pretty positive

51:06

It'll be pretty positive that

51:08

the ROI you get from investing in stuff like this

51:12

just empowers not just yourself

51:14

but your whole team to be so much more productive

51:17

right? Like

51:17

imagine if you have an army of

51:19

engineers like me who are shipping so much improvements and

51:23

and bug fixes, you know, every day, right?

51:26

Like, that is pretty exciting

51:29

Cool

51:30

one last question before we wrap up

51:32

this one is for the people in product on the

51:34

on the call

51:34

So

51:35

let's say we do have a number of

51:36

engineers who are shipping like Lauren

51:38

I'm just curious

51:39

like

51:40

how is the product team or other functions

51:41

of your company keeping up given that

51:44

like if you're shipping so quickly

51:46

have, have

51:46

are they using AI more to do their

51:47

jobs? Like as much as you can speak to that

51:49

and obviously you don't have

51:51

like you're not in that role

51:51

but just curious about how that works

51:54

I think this is where Grokbot has been actually exceedingly powerful

51:59

where, so before Grokbot, like, you know

52:02

obviously cursor only had cursor

52:04

like we only had agents window

52:05

we had a CLI

52:06

we had an IDE

52:08

and these are really like power user tools

52:11

right? Like they're designed for developers

52:13

so it's very, very developer centric

52:15

You can do knowledge work in them

52:17

but it like the UI is not really

52:18

optimized for that

52:20

So we actually didn't really have

52:23

well, I think like a lot of people like

52:25

you know, GTM product, like they might have used cursor

52:29

to do their work

52:30

but it definitely wasn't like a delightful experience

52:32

for them

52:34

I think now with Grokbot

52:36

it's become

52:38

Grokbot is basically like their cursor moment for

52:41

people who are not in tech

52:42

in my opinion. Like it's like

52:44

it's like a very

52:45

very accessible way to use agents in a very comfortable

52:48

very familiar interface. It looks like iMessage

52:52

and it's very fun to

52:54

you know

52:54

you can give your agent a fun name

52:56

you can have

52:57

you can kind of

52:57

do orchestration

52:59

within a very like natural way where you can sort of

53:02

you know

53:02

each agent's like a person and now you

53:03

got a team of agents like working on

53:05

you have one

53:05

one agent per account that you manage as

53:07

an example

53:08

Or if you're a PM

53:09

you have, you know

53:10

you can have an agent that summarizes all

53:12

the work that Lauren did last night

53:13

And then now you know what I did

53:14

right?

53:16

So I think our PMs are leveraging that a lot

53:18

and their shipping code too

53:20

so, you know, like oftentimes they will just say

53:22

"Oh, here's a bug, I fixed it

53:24

Can you look at it?" And then I'll go review it

53:26

and actually, it's just perfect on my okay stamp

53:29

so, that, I think that shows that

53:31

you know, the, the dune architecture is holding up

53:33

right?

53:34

The, all the

53:35

the really strict constraints allow

53:38

people who are not experts in engineering to

53:40

contribute at a high level

53:42

so I'm

53:42

I feel like I'm already seeing

53:44

that pay off a lot where

53:46

you know, designers and PMs are just able to

53:49

to, to ship features directly

53:52

and that just makes the Grokbot team super fast

53:56

right? Where we can ship so quickly

54:00

and we have a lot planned

54:01

so I'm very excited

54:03

to, you know, to, to ship more

54:06

ship more stuff