Google Brain founder Andrew Ng: "Prompting will be dead in 6 months Agent harnesses built with loo

Transcript

Thank you to this neuro institute's oral on

0:02

application development using LLMs

0:05

both ESA, OpenAI

0:07

as well as our Deep Learning AI Fund

0:08

have had the privilege of working with many

0:10

people building large language model applications

0:13

and so we hope to share some of

0:14

the practical tips with you

0:16

And after our presentation

0:18

we also have a distinguished panel of

0:22

researchers and builders that

0:24

with, with Ante, Tatsuki, Meredith, and Graham

0:27

that I hope will also deepen

0:29

everyone's understanding of how to build applications with

0:32

LLMs

0:34

Here's the agenda for today

0:35

what we hope to do is

0:37

help you understand how to build applications using LLMs

0:41

because this is a new way

0:42

of building AI applications that is very powerful

0:45

and allows you

0:46

to get things to work much faster than

0:48

was ever possible before

0:49

So the agenda for today

0:50

is talk a little bit about LLM basics

0:54

not to

0:54

teach you how to train an LLM from scratch yourself

0:57

but to make sure you have enough of

0:58

a mental working model for how LLMs work

1:01

so that you can use them effectively

1:04

and then talk about some examples of applications

1:07

easily

1:07

have a lot of insights to share on

1:09

how to write prompts

1:09

and share some exciting cutting-edge developments on

1:12

tools for building applications

1:14

Then let's talk about some practical tips of

1:16

building applications and wrap up with future predictions

1:19

in closing before we go on to the panel

1:23

So

1:26

I think one of the

1:26

tricky concepts to get about OEMs

1:29

is that they are general purpose technology

1:32

You know, similar to supervised learning and reinforcement learning

1:34

these are the tools we love

1:35

And that means that OEMs

1:37

are useful for numerous applications

1:40

So today, there are many businesses using it for document Q&A

1:43

many businesses are using this to build many

1:45

different flavors of customer service bots

1:49

is being used over and over to summarize messages

1:53

to route messages. It's being used in verticals

1:55

like to process legal documents or medical documents

1:59

I use it all the time as a

2:00

copy editor

2:01

There are also new forms of entertainment

2:04

being popping up, new forms of games

2:06

new forms of, you know

2:07

online companions that are being created

2:09

So this one technology

2:11

OEM, is useful for so many different applications

2:15

This is what I think of as the

2:16

AI stack. At the lowest level is maybe

2:18

the hardware or semiconductor layer

2:22

and

2:22

you know, very capital intensive, very concentrated, but clearly, right, lots of

2:27

certainly some businesses doing really well

2:29

And then with every wave of technology

2:32

what I've seen is that a lot of

2:33

the attention is focused on the tools and infrastructure layer

2:37

right? So a lot of attention on what OpenAI is doing

2:40

what Google is doing

2:41

what AWS, Azure, and so on are doing

2:44

And this seems to happen with every

2:45

technology layer

2:47

every time there's a new technological revolution

2:50

But it turns out that

2:51

for the people building tools and infra

2:54

and I run Latin AI

2:55

which does computer vision tools

2:56

but for the businesses providing tools and infra to be successful

3:00

almost by definition

3:02

there's another layer that's got to be even more successful

3:05

which is the application layer

3:06

because the applications built on top of the tools and infra

3:09

you know

3:10

they'd better generate even more revenue so that

3:12

they can afford

3:13

to pay the tools and infra

3:16

So the focus of today's presentation is really on

3:20

how you can

3:21

set yourself up to build exciting applications

3:24

using LLMs

3:27

First thing I want to go over is

3:28

basics of LLMs. Some of you may be familiar with this

3:32

bear with me, we'll get through this quickly

3:33

but I want everyone to have a

3:34

enough of a mental

3:36

working mental model of how LLMs work

3:38

because that affects the decisions you then make

3:40

later as you're building your

3:41

whatever application you have on top of these LLMs

3:45

But

3:46

text completion using a base LLM

3:48

will be if you give it a prompt like

3:51

I love eating, and ask it for completion

3:54

run it once, may say, bakers with cream cheese

3:55

my mouth's meatloaf, alter friends

3:57

and that's the text completion

3:59

And this is called a base LLM

4:02

which mostly predicts essentially what's the next word on the internet

4:05

or what's the next word on your

4:07

in, in, as

4:08

as observed in your training set

4:10

And a base LLM is built with

4:13

supervised learning

4:14

in which we create a data set

4:17

and have a transformer network repeatedly learn to

4:19

predict the next word

4:21

So

4:21

if

4:22

one of the

4:22

data sets has scraped off the internet a sentence like

4:25

my favorite food is a bagel with cream cheese

4:28

then this becomes a set of

4:30

training examples with supervised learning

4:33

where given the input

4:34

my favorite food is a

4:35

you know, well

4:36

the right answer that the

4:38

transformer tries to predict is the word bagel

4:40

my favorite food is a bagel

4:41

the next word is with

4:42

favorite is bagel with cream

4:45

and so on

4:46

And

4:47

when you scrape a very large data set

4:49

off the internet and other sources

4:51

kind of, you know

4:51

a hundred billion or sometimes more than a trillion words

4:54

and create a data set like this and

4:56

train a neural network

4:57

then this gives you an LM-based model

5:00

which is, you know

5:01

which can be very good at predicting the

5:03

next word on the internet

5:04

right?

5:05

Now

5:05

when you use ChatGPT

5:06

or BARD or Bing Chat or other products

5:08

they don't just predict the next word

5:10

they learn to follow instructions

5:12

So the next step

5:13

oh, sorry, and, but, and, and so with the base LM

5:16

this is what you can do

5:16

You can give it a prompt like

5:18

once upon a time

5:19

a unicorn, and it will, you know

5:21

complete the story for you

5:23

right? And this is what

5:24

you know, GPT-2, GPT-3 did

5:27

but it turns out that if you give

5:28

a base on a prompt like

5:29

what is the capital of France

5:31

it will try to predict what's the next

5:33

word if it has seen this on the

5:34

internet. And for all you know

5:36

this may be part of a quiz on

5:37

the internet. So completion

5:39

so what's the capital of France could be

5:40

what's the capital of Germany

5:41

where is Mumbai, is Mount Fuji or whatever

5:44

taller

5:45

because this is what you see on the

5:46

internet

5:47

So to make an LLM follow instructions

5:50

which is what

5:52

many of the leading LMs do these days

5:54

So, you know, ChatGPT, Bard, Bing Chat

5:56

follow instructions. It's just predicting the next word

5:58

The next step is instruction fine-tuning

6:01

in which you would come up with a

6:02

large data set of prompts

6:04

like what's a capital South Korea

6:06

capital South Korea, Seoul. Where the haiku poem

6:09

you know, there's a haiku poem

6:11

Help me brainstorm some fun museums to visit

6:12

in Bogota

6:13

So you have some suggestions

6:14

So as well as tell me how to break into footnotes

6:17

and you can say I can't assist with

6:18

that

6:19

so you have to have some refusals

6:21

as well in this data set

6:22

And given a large data set of prompts

6:25

instructions, as well as attempts

6:28

written, say written by a human to follow the instruction

6:31

you can then take your base LM

6:33

which you had trained on a hundred billion

6:35

or a trillion words on the internet

6:37

and further run gradient descent to fine-tune it

6:40

to this instruction following data set

6:43

So given this

6:44

help me

6:45

brainstorm some fun museums to visit in What's

6:47

the next word? Sure. And after that

6:50

here are some, and so on

6:53

And with a data set like this

6:54

if you continue to train the algorithm

6:57

for a much shorter period of time

6:58

maybe weeks, then you can prompt it to say

7:02

what's the capital of France?

7:03

And will have learned to follow instructions

7:05

France's capital is Paris

7:08

So with this

7:08

you end up with what's called an instruction-tuned

7:10

model that doesn't just predict the next word

7:13

but learns to follow instructions

7:15

And then to shape the mental model of

7:17

the LLMs that you use and hopefully will

7:19

build applications on top of

7:21

there's one last step to make

7:23

which, which many companies implement in various different flavors

7:27

to make these models safer

7:29

so such as more

7:31

the three H's, right, honest, hopeful, and harmless

7:34

And common way to do this would be RRHF

7:37

which you may have heard about

7:38

but I'll just give a

7:39

one slide description of what RRHF actually does

7:43

which is

7:44

First step of our HF

7:45

is to train a model to predict answer quality

7:49

and this is a reward model

7:50

So to do that

7:51

you would write a prompt

7:53

advise me on how to apply for a job

7:55

and have the LLM generate a bunch of

7:56

responses. And we say

7:57

I'm having a hard

7:58

here are some steps to follow

7:59

Or it may say

8:00

just try your best

8:01

not that helpful, or may give a horrible answer

8:03

it's hopeless, why bother?

8:06

and then ask people to score these answers

8:09

say on a score scale of one to five

8:11

giving the more honest

8:12

helpful, and harmless, answers a high score

8:16

five

8:16

and the more useless or even harmful answers a low score

8:20

say one

8:21

And given this data set of answers and scores

8:27

The next episode used

8:28

a train and reward model that mimics

8:30

how humans assign scores

8:33

And the reason we do this is because

8:35

human judgments are expensive

8:36

but once you have a reward model that

8:38

mimics how humans are judging the answers

8:40

you can then judge a lot of LLM

8:41

outputs

8:43

And then the final step

8:44

is to have an LLM

8:46

generate tons of answers

8:47

and use the reward model to score all of these answers

8:51

and then to tune

8:52

the weights of the LLM

8:54

to increase the odds of it generating the higher reward

8:58

answers. And so through this process called ROHF

9:01

and there are many other flavors

9:03

like ROAIF and many other flavors

9:04

but ROHF may be the basic algorithm

9:07

you can tune an LLM to be more honest

9:09

helpful, and harmless. So less likely to say

9:12

you know

9:12

emit toxic speech, less likely to, express kind of very negative

9:18

undeserved stereotypes or biases

9:20

less likely to give out explicitly harmful instructions

9:23

right? So And then just one technical detail

9:26

those of you that know

9:27

RFJ know that

9:28

typically we ask humans not to write scores

9:30

from one to five

9:31

but we ask people to

9:32

rank all their pairs of answers

9:34

But other than that

9:36

you know, this is mostly an accurate description

9:38

But with rank, with humans judging

9:40

do you prefer answer A or answer B

9:42

that also gives you a way to learn the reward model

9:45

and it goes to learn the reward model

9:46

that is consistent of how humans are ranking

9:49

it to express their preferences

9:50

and that is RFJ

9:53

and so with this understanding

9:56

I think you probably know

9:57

most of what you need in order

9:59

you know, to, to, as

10:00

as you're building on top of LLMs

10:02

to, to use them effectively

10:04

but just one more technical detail I want to cover

10:07

which is I've talked about the algorithm as

10:10

predicting words. It turns out that

10:13

they, the LLMs today are actually trained to predict tokens

10:16

which

10:17

and tokens are either words or word fragments

10:20

So a tokenizer starts with

10:22

say every character being its own token

10:25

and it repeatedly merges tokens that occur together frequently

10:29

so that common words end up becoming tokens

10:32

So the example, my name's pretty common name

10:35

Andrew, would be one token

10:37

and somewhat less common words like translate and

10:39

programming are two tokens

10:40

and an even more rare word like tonkotsu

10:42

you know, it's in the Japanese dish

10:44

right? Tonkotsu ramen, in, in

10:46

in the case of this tokenizer is four tokens

10:49

And roughly, if you use, OpenAI tokenizer or other tokenizers

10:54

roughly rule of thumb

10:55

one token is maybe three quarters of a word

10:58

and so the number of tokens is

11:00

you know

11:00

maybe thirty-three percent bigger

11:02

than the number of words

11:04

And what I want to do is

11:07

if you have not yet done this

11:09

I want to show you

11:13

maybe just how easy it is

11:19

Oh, thank you for that reminder

11:22

All right. So what I want to do is show you

11:24

if you haven't seen it before

11:25

hopefully how easy it is

11:27

to prompt an LLM in code

11:29

Right, so, this is the OpenAI interface

11:32

and, and don't, don't worry about the code

11:33

we'll, we'll, you know, the, the

11:35

the code syntax you can get from somewhere

11:37

Import OpenAI, load my API key

11:40

this is a little helper function

11:42

where I tell it

11:43

there's a system message

11:44

you're a hopeful assistant

11:45

so easily elaborate on the system message later

11:48

we can tell the

11:48

what role you want to play

11:50

and then plug in the prompt

11:52

and let's say, tell me a joke

11:54

And then, right

11:55

why don't skeletons fight each other?

11:57

They don't have the guts

11:58

You know, not bad

12:02

similarly, Google Palm include Google's generative AI

12:05

configure Palm, this loads the model

12:08

Bison model

12:11

and then let's have Palm

12:13

Why did the scarecrow win an award?

12:15

Because he was outstanding in his field

12:17

I don't know

12:20

And lastly, we have not yet done this

12:22

you know, worth, something worth doing someday

12:25

is running LM locally on your laptop

12:27

So, oh

12:27

Llama

12:28

is a

12:29

software package that's used by many developers to

12:31

run LMs locally

12:32

So this LM, I'm running Llama 2 on my laptop

12:38

and, you know, why don't scientists trust atoms?

12:41

Because they make up everything

12:44

and I think that is actually really cool that it's actually

12:48

it's, it's actually, if you haven't done it yet

12:49

it's probably much

12:50

easier than you think to run an LM

12:52

on your laptop. You do need a

12:54

reasonably fast laptop. This is an M2 laptop

12:56

that I'm running on

12:57

but I'll also run it on AMD hardware

12:59

run it on NVIDIA hardware

13:01

and actually, when, if you're ever doing demos

13:03

if the Wi-Fi is ever down

13:04

having a locally running LM helps a lot

13:07

Oh, and actually, when I'm on an airplane

13:09

when there's a flight here to New Orleans

13:10

I was using an LM on my laptop

13:12

to get stuff done

13:12

because I wasn't on Wi-Fi yet

13:16

So just one more piece of intuition building

13:19

People often think, boy, using an OM is really expensive

13:21

right? So

13:22

here's the back of the envelope calculation for

13:24

how much it costs to use to prompt

13:25

an OM

13:26

These are some of the prices of different

13:28

OMs that I got off the internet

13:30

This is a few weeks ago

13:31

maybe it's changed by now

13:32

But let me focus on this number

13:33

0.2 cents per 1,000 tokens

13:36

And if you were to ask

13:38

how long would it take to keep someone

13:39

occupied for an hour

13:41

assuming a reading speed of 250 words per

13:43

minute and three quarters word per token?

13:46

If you do the

13:46

you know, back of the envelope calculation

13:48

250 words per minute times 60 minutes

13:51

so you need this many words

13:52

divided by three quarters

13:53

so that's the number of tokens

13:55

times the price of the output tokens

13:57

and maybe there's an input prompt as well

13:59

So you've assumed the

14:00

input prompt and the output completion are the same length

14:02

and we'll add up these

14:03

It turns out that this

14:04

you know

14:05

back of the envelope calculation suggests

14:07

maybe it costs you about seven cents

14:09

to generate enough text

14:10

to keep someone busy for an hour

14:12

So it's actually remarkably cheap

14:14

especially when you use

14:15

you know, a, a 3.5 terabyte

14:17

Some of the other OMs are

14:19

can be one order of magnitude more expensive

14:21

There is maybe a

14:22

hundred X difference between the expensive and the cheap OMs

14:25

but, but I find that many people price optimize prematurely

14:30

and it's actually

14:31

maybe cheaper than you think if you haven't

14:33

done this a lot yet

14:36

And then lastly, with LLMs

14:39

there are many text processing applications

14:42

that used to take me and other teams

14:44

you know, six months to build

14:46

that any of you

14:47

can now build in very small numbers of hours

14:50

maybe tens of minutes or maybe even small

14:52

numbers of hours

14:54

Very common use cases used by many companies

14:57

summarizing, given long text, summarize it, inferring

15:00

classify text to extract keywords

15:02

transform text from one form to another

15:04

or to expand text from a small prompt

15:07

to a very large email

15:08

And I want to focus here on

15:10

I feel like there's a difference between the web UI

15:13

which many consumers use

15:14

you know, I do that all the time

15:16

so I type in my prompts to web

15:17

UI versus the practices

15:19

so if you are writing a software application

15:21

by calling an API

15:21

a little bit different

15:23

But these are common text processing applications

15:25

use cases that, people are using

15:30

via the web API

15:32

And so, let me show you some examples of this

15:36

Again, don't worry about the code syntax

15:38

it just will point out how little code

15:40

it takes. So here's summarizing

15:42

Input

15:43

Oh, I'm sorry

15:44

Thanks, Ida

15:47

Oh, could people see my demo just now?

15:48

Okay, thank you

15:54

alright, so here's summarizing

15:56

So import some tools from

15:58

say again?

15:59

No

16:01

Sorry, couldn't hear you

16:04

got it

16:10

Alright

16:11

So import some tools from OpenAI

16:13

there's a helper function

16:16

here's a product review

16:17

You know

16:17

got this plaid panda plush toy for my daughter's birthday

16:20

so on and so forth

16:21

got to wear it myself

16:22

And then if you want to summarize it

16:24

write a short prompt

16:25

you toss it generate a short summary product review

16:28

f string, plug in the product review

16:30

Do that, and then, you know

16:32

I just made an API call

16:35

and real-time just generated this summary

16:38

right? And then, you can change the prompt to say

16:42

focus on the shipping and delivery if you

16:44

want the summary to focus on one aspect

16:47

So customers have their price

16:48

but suggest, you know, different options

16:50

item appear earlier than expected

16:52

So this is one example summarizing as a

16:54

very common use case of

16:57

these OLMs

16:59

another one is inferring

17:01

So let's see, given a review of a lamp

17:05

so this is a product review of a lamp

17:07

need a nice lamp for my bedroom

17:10

there are a lot of traditional NLP tasks

17:11

like sentiment classification. Is this review a positive

17:14

or a negative sentiment

17:15

right? Previously to do this

17:16

we've had to collect training data

17:18

run the supervised learning algorithm

17:20

find a way to deploy the model on the internet

17:22

you know, debug it

17:24

but now you can just write a prompt

17:26

what is the sentiment of the product review?

17:29

And then it says

17:30

sentence product review is positive

17:33

And one common design pattern is

17:35

if you don't want it to output a sentence

17:38

which would be harder for a downstream piece

17:40

of software to process

17:41

do you want to

17:42

search your records to see if it's positive

17:43

or negative?

17:44

You can also prompt it to say

17:45

give your answer a single word

17:47

either positive or negative

17:48

and it just says positive

17:50

And actually

17:50

easier later will say a lot about how

17:52

to specify output formats and make it easy

17:54

to plug in your OM into one step of your software

17:57

so its output can be used

17:59

in, in sort of your downstream processing as well

18:04

let's see, one more fun one

18:06

Let's see

18:09

transforming text. So

18:12

LLM is actually getting pretty good at translating

18:15

text. It's actually better for especially high resource languages

18:18

I would say, than some of the dedicated translation

18:22

websites

18:23

Translate the following English text to Spanish

18:26

Yo hola

18:27

Sorry, can't pronounce that

18:29

Tell me what language this is

18:30

This is French

18:33

and, oh

18:34

this is actually

18:36

translate the following text to French and Spanish

18:38

and English pirates

18:40

I want to order a basketball

18:41

I be wanting to order a basketball

18:43

That's not bad

18:44

But just one fun thing

18:45

I find that when I work on translation

18:47

you know, because, My team members speak different languages

18:52

right? I speak English and really poor Chinese

18:54

Some of my team members speak Spanish and

18:56

English

18:56

Among my team members

18:58

a lot of us are only common language

18:59

is English

19:01

And so when we're testing translation

19:03

English pirate get used a lot because English

19:06

pirate is a language we all know

19:08

So for some reason among my teams

19:10

when we're building translation applications

19:11

the first language we translate into is

19:13

is actually English pirate

19:18

let's see

19:19

And then just one last example

19:21

expanding. So, you know, if, you want, there's a product review

19:27

you know, and you want to

19:30

have a short prompt

19:31

Your customer review service

19:32

service agent or assistant

19:34

your task to send an email

19:36

right, giving the customer email on top and so on

19:38

then this is, this is another use case

19:39

You can take a short prompt

19:41

and have a system generate a longer piece

19:43

of text for you for

19:44

for certain communications, right? Don't, don't use this to spam people

19:47

but sometimes for legitimate communications

19:50

you know, this can be a useful application

19:56

so, all right

20:02

So in addition to what I just shared

20:04

summarizing, inferring, transforming, expanding

20:06

these are operations that you can build very

20:08

quickly into many applications

20:10

and when you're brainstorming applications

20:12

these are, these are hopefully useful tools

20:15

And then just on top of these things

20:17

I think out in industry

20:19

there are two very commonly built applications on

20:22

top of Elms

20:23

One is retrieve augmented generation applications

20:25

where you retrieve a document and have a

20:28

a, Elm answer questions against that document

20:31

and the other is there are many

20:32

many companies building

20:34

various flavors of customer service chat and many

20:36

flavors of chatbots. And

20:39

let me hand it over to Isa

20:40

to share more

20:41

concrete tips on prompting and also to dive

20:43

more deeply into Rag and chatbots a little bit

20:48

Yeah

20:53

Okay, so in this section

20:54

we're going to talk about strategies for getting the best

20:58

results that you want from language models

21:00

the practice of writing and iterating

21:03

on instructions for a language model is sometimes

21:05

referred to as prompt engineering

21:08

this term was coined

21:09

to describe the process of prompting base models

21:12

so not instruction-tuned models

21:14

which involves kind of more unintuitive tricks

21:17

but prompting assistant models is actually very intuitive

21:20

So in this section

21:21

we'll cover, the chat format of assistant models

21:25

limitations of current models

21:27

and then we'll

21:28

go into some prompting strategies and tactics

21:30

for getting the best results from specifically instruction-tuned

21:33

models

21:37

and some of those

21:38

strategies that we'll discuss will help to

21:40

address the limitations that we will discuss in the

21:43

prior section

21:45

So this is just kind of

21:46

reiterating what Andrew said

21:48

we're going to first talk about the chat

21:49

format of assistant models

21:51

these are the models that you're most likely

21:53

to have interacted with

21:55

in products like ChatGPT

21:58

Bing, so on

21:59

and as Andrew discussed

22:00

base models are trained to predict the next

22:02

token in a sequence during pre-training

22:05

And because of the way they're trained

22:06

base models are effectively document generators

22:09

So if you start a document

22:10

it will just continue

22:11

returning the most likely next word or phrase

22:13

because it's just learned

22:15

from the content on the internet

22:16

and a few other data sets that it's

22:18

seen

22:19

but we actually want a model that can

22:21

answer questions and follow instructions

22:22

and so

22:23

we take the base model and teach it

22:24

to follow instructions in the post-training process

22:27

which is, you know, superfine fine-tuning and RLHF

22:30

as Andrew discussed

22:32

And during, post-training

22:33

we also use a different data set

22:35

with examples of multi-turn conversations in a special chat format

22:39

where we use special tokens to mark different

22:41

parts of the conversation

22:42

So in the bottom right

22:43

there's kind of an example with some special additional tokens

22:47

and then the model learns how to use

22:48

this format

22:49

And the examples we use in this process

22:51

are mostly very high-quality examples generated by

22:54

human contractors and then a few synthetic examples

22:58

And the resulting chat models are

23:00

the models that users now interact with

23:02

as I mentioned

23:04

and so

23:04

instead of taking in just a prompt and giving a completion

23:07

these models use a different format

23:09

and the input is actually a list of messages

23:13

and then

23:14

the

23:15

kind of response will be an assistant message

23:18

So each message has

23:20

a role, which is one of system

23:22

user, assistant, or tool, and content

23:24

which is just the content of the message

23:26

and in general, these, messages are

23:29

formatted with a system message first

23:33

followed

23:33

followed by kind of alternating user and assistant

23:35

messages

23:37

and the system message acts as

23:39

a kind of high-level instruction that sets the

23:41

behavior or persona of the assistant

23:43

And you can kind of think of it

23:44

as whispering in the assistant's ear

23:47

guiding its responses without the user being aware

23:49

of the instruction

23:50

So it's a good way to frame the

23:51

conversation without making the request itself

23:54

part of the conversation for the user

23:56

And then the user entries are just

23:57

you know, the user input

23:59

and then the assistant messages are the completions

24:02

And then the final type is tool

24:04

We'll talk about this more later

24:06

but we can augment the abilities of models

24:07

by giving them access to tools

24:09

like internet search

24:11

or the ability to run code or call

24:12

specific functions

24:14

And so tool messages are just a type

24:16

that we have for kind of the output

24:18

from these tools

24:21

And so although the

24:22

chat format is designed to make multi-turn conversations easy

24:25

they're also useful for just single-turn tasks without

24:28

any conversation where

24:29

you put the prompt instructions

24:31

in the system

24:32

or in the first user message and or the system message

24:35

and then you get

24:37

the response in the assistant message

24:39

So I'll show you a few examples

24:42

of this format. This is the

24:43

the playground. It's just a UI we have for interacting with

24:47

language models

24:48

So in this example

24:49

I've put the instructions

24:51

you're an assistant who responds only with words

24:53

beginning with S in the system message

24:54

So that's the high-level instruction

24:56

And then I'll say something like

24:59

Tell me a joke about an AI conference

25:02

So here, here we have our joke

25:04

And since we're having a conversation

25:05

I can kind of ask in a follow-up

25:08

message, say I want this to be shorter

25:11

I can

25:14

I can continue to refine kind of what

25:16

I'm asking for

25:18

and so this is a multi-turn conversation

25:20

You can also, instead of, having kind of this conversation

25:24

I could have put more instructions in the original system message

25:27

so I could say

25:28

make sure the

25:31

make sure your response

25:34

is only a few words long

25:43

And then in this case

25:44

the behavior of the assistant is

25:47

as we instructed

25:48

So that was an example of a back and forth conversation

25:50

but you can also

25:51

as I mentioned

25:53

use a model to complete an individual task

25:55

where you define the task in the system

25:57

message and then you put the actual input that you want

26:01

the

26:01

the input of the task in the first

26:02

user message

26:03

So I'll show you another example

26:08

So this is a

26:14

slightly, slightly longer example

26:16

Basically the task is to distinguish between promotional

26:19

emails and personal communications

26:21

and we're saying

26:22

your primary focus is to ensure that emails

26:24

from actual people are not ignored

26:26

and then we're asking the model to respond with just

26:29

the word true or false

26:31

depending on, if this email is

26:33

is kind of promotional or not

26:35

And then I just have a random promotional

26:37

email that I received

26:39

using this format that I specified

26:42

so if I

26:46

run this, It's true because this

26:49

this is a promotional email

26:50

But then in the instruction we said

26:53

your primary focus is to ensure that emails

26:54

from actual people are not ignored

26:56

So if I add kind of an additional

26:59

additional paragraph to, to the email

27:02

just pretending that this is from Evan from

27:05

from West Elm, San Francisco

27:09

and then we, you see in this case

27:12

the

27:12

the model knows that this is from a

27:14

real person and we won't

27:15

don't want to ignore those emails

27:17

So

27:18

that was just an example of using the

27:19

system message to define the task

27:21

and then the first user message for the

27:22

actual input for the task

27:24

Great, so that was the chat format of assistant models

27:27

Next

27:28

let's discuss some of the limitations of current

27:30

models

27:31

So

27:32

even though language models haven't

27:33

been exposed to

27:34

a vast amount of knowledge during their training process

27:37

they haven't perfectly memorized all of the information that they've seen

27:41

and so they don't always have a good

27:42

sense of what they know and what they

27:43

don't know

27:44

And this means that they might try to

27:45

answer questions about obscure topics

27:48

and can make things up that sound plausible

27:50

but are not actually true

27:51

and we call these fabricated ideas hallucinations

27:54

And in some ways

27:55

you can kind of think of anything a

27:56

model outputs as a hallucination in that

27:58

it's coming up with its best guess for

28:00

text to produce based on what it's learned

28:02

from all the data it was trained on

28:04

And

28:05

I actually think that

28:06

the issue with hallucinations at the moment

28:08

is that while in some contexts it's okay to hallucinate

28:10

for example

28:11

if you're asking for

28:12

help with creative writing or new creative ways

28:15

to address a problem

28:16

in other cases

28:17

Isn't. So

28:18

in cases where you want factual

28:20

information or formal output or something

28:22

and the current models aren't always very good

28:24

at differentiating between these situations

28:27

And so we probably don't want models that never hallucinate

28:29

because

28:30

you can kind of think of it as

28:31

the models being creative

28:32

but we just want models that

28:34

that hallucinate in the right context

28:36

And over time

28:36

we'll continue to

28:37

decrease the likelihood of hallucinations

28:39

from language models in factual contexts

28:42

For example, GPT-4 at time of release

28:45

scored 40% higher than GPT-3.5 on internal

28:49

factuality evals

28:52

there's an example of a hallucination on the slide

28:54

so it's just asking for

28:56

about a fake product from a real toothbrush

28:58

company

28:59

And you can imagine a situation where if

29:00

you were brainstorming products

29:01

you might actually want the model to

29:03

to output something like this

29:04

but if you're asking what is

29:06

you're kind of, you're most likely looking for factual information

29:09

and so it's probably not the right

29:10

the right time for a hallucination

29:14

other limitations, models have limited context

29:17

context windows

29:18

so the number of words you can put into a prompt

29:21

and they don't learn about you over time

29:24

they don't have memory

29:26

so they don't get to know your use

29:27

case as you continue to use them

29:28

There's also a knowledge

29:29

knowledge cutoff, cutoff, for kind of the end of the

29:34

the model's training data

29:35

and then also there are just some tasks

29:37

that are too hard for the model

29:40

you know, with some simple math

29:41

we can address

29:43

address the

29:44

math reasoning abilities by allowing the model to execute code

29:47

but there are some

29:48

some questions that are just too challenging

29:52

So now let's move

29:54

move on to some strategies and tactics for

29:57

getting better results from models

29:59

we're going to discuss three different strategies and

30:02

then some tactics within each strategy

30:06

these are write clear and specific instructions

30:08

give the model time to think

30:10

and use external tools

30:12

and this whole section is about prompting language

30:14

models in the context of using them to

30:16

build your own applications

30:17

However, most of these techniques could also be used

30:20

or adapted if you're interacting with a language model

30:23

as a consumer, for example, in ChatGPT, BARD, Claude, Bing

30:28

or something else

30:29

But if you're interacting with a model in

30:31

in a conversation

30:32

you don't need to be as careful with your first prompt

30:35

because you can always refine and iterate

30:37

what you're asking for throughout the conversation

30:39

But if you're building an application

30:41

that uses models to complete a task as

30:43

part of a workflow

30:44

for example

30:45

you'll want to make sure that you're consistently

30:47

getting results you want from different inputs

30:49

in the format you want

30:51

So

30:51

that's where you really want to make sure

30:52

that your initial instruction is good

30:55

And in general

30:56

I would recommend starting by writing a prompt

30:58

that works well for a more advanced language model

31:00

and then once you have something you're happy with

31:02

you can try adapting the prompt to work

31:04

with a less advanced model

31:05

maybe try and make the prompt shorter so

31:07

you can save how much you spend on tokens

31:09

and throughout, I'll be using GPT-4 and GPT-3

31:11

For examples, but these strategies and tactics should apply to

31:15

to other language models as well

31:17

And there's a few examples linked on the slides

31:20

so we'll figure out a way to share them after

31:22

so that you can look at them as

31:23

well

31:24

So

31:24

the first strategy is write clear and specific

31:26

instructions. This might seem like a very obvious point

31:29

but models can't read your mind

31:31

so you should always express what you want

31:33

the model to do in full detail

31:34

in the same way you would of describing

31:36

a task to another person who is very smart

31:38

but who has no prior context or knowledge

31:40

of your specific task

31:42

If you reduce ambiguity or possible interpretations of your request

31:45

then you'll reduce the chances of receiving irrelevant

31:47

or incorrect responses

31:49

And you shouldn't confuse writing a clear prompt

31:51

with writing a short prompt

31:52

because in many cases

31:53

longer prompts actually provide more clarity and context for the model

31:57

which will lead to more detailed and relevant outputs

32:01

So first, first tactic is give detailed context

32:05

for the problem

32:06

And so one method is to

32:08

just make sure you include details in your

32:10

query to get more relevant answers

32:12

So

32:12

there are a few examples on the slide

32:14

Instead of saying, how do I add numbers in Excel

32:17

say, you know

32:18

be very specific about the context in which

32:20

we're trying to add numbers

32:21

So

32:21

how do I add up a row of

32:22

dollar amounts in Excel?

32:24

I want to do this automatically for a

32:25

whole sheet of rows

32:26

and so on

32:30

the next method is to instruct the model

32:32

to answer using a reference text

32:34

As we discussed, the models, can very confidently invent fake answers

32:39

and in the same way that

32:40

you know

32:41

a cheat sheet can help a student in a test

32:43

or if you were doing a reading comprehension

32:44

you would need to have the article in

32:46

front of you instead of trying to

32:47

to answer from memory

32:49

Providing reference text to models can really help

32:52

getting

32:53

to get more relevant answers that contain fewer

32:55

hallucinations

32:58

And this also, you know

32:59

given that the models have limited context windows

33:01

we sometimes need to

33:02

to have a way to dynamically look up information relevant to

33:05

to a task

33:06

And you can do this with

33:08

embedding retrieval

33:10

which we'll talk more about in the external

33:12

tools section

33:15

the next

33:16

method is to

33:17

use delimiters to clearly indicate distinct parts

33:20

of the input

33:21

And you can use

33:22

triple quotes, triple baptics, XML tags, anything really, just to

33:28

to clearly separate specific pieces of text from

33:30

the rest of the prompt

33:33

I like using the four pound signs or triple quotes

33:36

triple baptics

33:37

a lot of them are one token

33:38

which is nice

33:40

the examples on the slides

33:42

are fairly simple

33:43

so it probably won't make a huge difference if you include

33:46

delimiters, but, it, the kind of more complicated the task

33:49

the more important it is to be very

33:51

clear about the different sections

33:52

And, in my personal use

33:54

whenever I'm asking for debugging help

33:56

if I have to input a bunch of different files

33:58

I'll always use delimiters to

34:00

to let the model know

34:01

you know, the, where the different files end

34:07

So the next tactic is to specify the

34:08

desired output format

34:10

So one thing that you may have seen

34:12

is people asking the model to adopt a

34:14

persona

34:15

Sometimes you can say you're an expert engineer

34:18

or copywriter

34:20

if you're building a chat application

34:21

you might want to describe the persona that

34:23

you actually want the model to take

34:24

like a pirate or to

34:26

you know, respond in the style of Shakespeare

34:31

the next method is to specify the desired

34:33

length of the output

34:35

so you could ask the model to produce

34:37

a target length of sentences

34:39

paragraphs, bullet points

34:41

the models are not very reliable at generating

34:43

a specific number of words

34:45

because of the way that words are split into

34:47

into tokens

34:48

the next method is to ask for output

34:50

in a specific structured format

34:52

the models are very good at

34:54

at outputting structured data if you give them

34:56

a detailed description of the format that you

34:57

want

34:58

So I'll show you an example

35:00

So in this case

35:01

we're asking the model to

35:03

we're going to give it a

35:04

section from a research paper delimited with four pound signs

35:08

and the model's task is to extract the title

35:10

authors, and date, and it must output a JSON with

35:14

a title, authors, and date

35:17

And then if you can't find one of the fields

35:19

just set it to a default value

35:21

so an empty string or an empty list

35:23

And then

35:24

I've just copied the first page from the InstructGPT paper

35:28

and so as, as you can see

35:29

we get this structured JSON output

35:31

and I don't think the date was included

35:33

so

35:33

it set the date

35:34

just to an Pretty string

35:38

So this is a nice way to get

35:39

kind of structured data from unstructured text

35:42

the next, method is to provide examples

35:45

So in general, it's better to

35:48

it's more efficient to

35:49

kind of just explain what you want the model to

35:52

to, to do rather than demonstrating by example

35:54

but

35:54

there are some cases where providing examples might

35:57

be easier

35:58

for example, if you want the model to

36:00

to copy a particular style of responding to user queries

36:02

that's hard to, to explain, this might be a helpful

36:05

a helpful method

36:06

This is known as few-shot prompting

36:08

although that term really is more used when describing prompting-based models

36:13

because you need to kind of

36:15

trick the model into responding in the format

36:16

that you want by giving it a few examples

36:18

but

36:19

we can still kind of use the term

36:22

and so in this example

36:23

we have

36:25

an example and then

36:27

another query from the user

36:28

So the system messages to answer in a consistent style

36:32

make sure to use consistent length

36:33

punctuation, and phrasing

36:35

And then we have

36:36

an example of this assistant response with this

36:39

like

36:39

very specific punctuation and kind of using

36:43

metaphorical descriptions

36:45

so

36:47

If we ask the model to describe the ocean

36:49

it kind of responds in a very

36:51

similar response with a similar length

36:54

So in some cases

36:55

this might actually mean that you can have

36:57

maybe, maybe it will be fewer tokens to just have

37:00

a short system message and then an example

37:03

So the second strategy is give the model

37:05

time to think

37:06

So if a person was asked to multiply 39 by 27

37:10

they probably wouldn't know the answer instantly

37:12

but could work it out with time

37:14

And in the same way

37:15

models make more reasoning errors when trying to

37:18

answer right away rather than taking time to

37:19

work out an answer

37:21

And so if a model's making a reasoning

37:22

error by rushing to an incorrect solution

37:26

you should try to reframe the question to

37:28

request a chain of relevant reasoning before the

37:30

model provides a final answer

37:33

and asking for a chain of thought is sometimes known as

37:36

before an answer can help the model

37:38

reason its way towards correct answers more reliably

37:41

And you can kind of think of this

37:42

as asking the model to think longer about

37:44

a problem and make incremental progress

37:46

Spending and spend more computational effort on the

37:48

task

37:51

So

37:52

the first tactic is to

37:54

guide the model's reasoning with steps

37:57

the

37:58

one method is to just specify the steps

37:59

required to complete a task

38:02

sometimes, you know, some tasks are best

38:05

specified as a sequence of steps

38:06

and so writing them out explicitly can sometimes

38:08

help

38:09

In the example on the right

38:11

we're asking first

38:12

for a summary of some text and then

38:14

a translation

38:16

for advanced models

38:17

it would probably be able to do this already

38:19

but if, you know, accuracy is really important to your application

38:23

and maybe this would increase the accuracy of

38:25

getting a good summarized French translation by a few percent

38:28

it might be worth kind of breaking it

38:29

down into steps

38:33

The next method is to instruct the model

38:34

to work out its own solution

38:36

before rushing to a conclusion

38:39

so for example

38:40

suppose we want the model to evaluate a

38:42

student's solution for a math problem

38:44

the most obvious approach would just be to

38:46

ask the model if a student's solution is

38:48

correct

38:49

So in this case

38:50

we have, a math problem and then a student's solution

38:54

and then our system message is just to

38:56

determine if the student's solution is correct or

38:58

not

39:00

And so GBT-4 says the student's solution is correct

39:04

However, this is actually an incorrect answer because

39:08

in this part

39:10

I think X is the

39:12

number of square feet and the

39:14

maintenance cost is actually 100,000

39:17

plus 10X instead of 100X

39:20

So what we can do instead is in the system message

39:26

ask the model to first work out

39:28

its own solution to the problem

39:29

and then compare its solution to the student's

39:31

solution

39:32

and evaluate if the student's solution is correct

39:34

or not

39:35

And then we say

39:36

just, just to reiterate

39:37

don't decide if the student's solution is correct

39:39

until you've done the problem yourself

39:41

So first, the, the model works out its own solution

39:44

figures out the answer's actually 360x plus 100,000

39:48

and then is able to conclude that the

39:50

student's solution is incorrect

39:52

this next method is

39:53

is very similar to the previous one

39:55

but, in some applications

39:57

the reasoning process that a model uses to

39:59

arrive at an answer might

40:01

you might not want to share it with the

40:02

with the user. For example

40:03

if we were building a tutoring application

40:05

if we just showed the entire model output

40:07

it would kind of give away the question

40:09

the, the answer to the student

40:11

So

40:11

in a monologue is a way you can

40:13

mitigate this by basically

40:14

asking the model to put parts of the

40:16

output that are meant to be hidden

40:18

into a structured format that makes passing it out after more

40:21

more, simple

40:25

So

40:28

Sorry, this is a bit of a long system message

40:30

but

40:30

we're basically asking the same steps

40:33

as we did in the previous example

40:34

So first work out the problem

40:36

then compare your solution to the student's solution

40:39

and then if they made a mistake

40:41

determine what hint you would give them

40:43

and then step forward

40:45

just output the hint

40:49

So the model then

40:50

separates the parts of its

40:53

response with these triple quotes

40:54

and then you can imagine if you were doing this

40:57

from code, you could just find like the last occurrence of

41:01

of triple, triple quotes or something

41:05

to get the hint

41:11

And then to, I guess that's, that's a pretty explicit hint

41:14

but we, we did get a hint

41:19

Sometimes a hint is less

41:20

less explicit

41:22

So our next tactic is to use a

41:23

model to iterate on its own work

41:27

Sometimes models miss things on their first attempt

41:29

especially if you really fill their context

41:32

For example

41:33

if you asked it to extract quotes from a long document

41:36

it might stop early and fail to

41:38

list all relevant excerpts

41:39

So in that case

41:40

you can sometimes get better performance by prompting the model to

41:44

with follow-up queries, to find additional excerpts

41:47

You need to be careful not to

41:48

kind of ask leading questions because the model

41:50

is eager to please

41:52

so it might just find more quotes even

41:53

if they aren't really relevant

41:56

And in addition

41:57

the current models are not very good at self-reflection during

42:01

like, inference, like while they're making a completion

42:04

because they're trained to predict the next most likely token

42:06

even if they're on

42:08

kind of starting down an incorrect path

42:10

while generating an answer

42:11

they'll just continue down that path

42:13

choosing the next most likely token

42:15

But if you ask the model in a subsequent call to

42:18

to check its own work

42:19

it would be better at

42:21

kind of evaluating the

42:23

its previous message

42:26

another thing you could do is

42:28

ask the model to answer a question multiple

42:30

times and determine the best answer

42:32

So if you're building an application where accuracy

42:34

is the most important thing

42:35

maybe at the expense of latency or cost

42:38

you could generate multiple model

42:40

responses with either different prompts or different temperatures

42:44

which is just the degree of randomness of the

42:46

the model's response

42:48

And if, say, the answer was objective

42:51

like a number or a string or something

42:53

you could just take the most frequent answer

42:55

from the different completions

42:56

and otherwise you could also use

42:58

another prompt to choose the best answers based

43:01

on all of the different

43:02

model-generated responses

43:06

the next tactic is to

43:08

break complex tasks into simpler subtasks

43:11

So, you know, just as you would decompose

43:14

a complex system into modular components while writing code

43:19

the same thing can be true of tasks

43:21

for language models

43:22

And sometimes complex, or not sometimes

43:25

complex tasks have higher error rates than simple tasks

43:28

and sometimes you can redefine a very complex

43:30

task into a workflow of more simple tasks

43:33

and then based on the output of the previous task

43:36

take a different, next prompt

43:39

So, the first Method is to use intent classification to identify

43:45

now I'm reading this

43:46

this is very wordy

43:47

to identify the most relevant instructions for a

43:49

user query

43:51

and so in the example

43:53

we have a kind of customer service assistant

43:55

and you could take a first step where you

43:57

take the query and you classify it into

44:00

a category. You could ask for like a

44:01

structured output with a primary category and a secondary category

44:05

for example

44:06

and then once you have the output from the first prompt

44:09

you can take, say it was, you know, the

44:13

the user query was classified as technical support

44:15

you could take different instructions

44:18

based on

44:19

based on the result at that step

44:21

and the advantage is that

44:23

this can also

44:24

lower costs because you don't need to have

44:26

very large prompts at every single step

44:31

the next method is to process long documents

44:34

piecewise and construct a full summary recursively

44:38

so since models have a fixed context length

44:40

you can't always summarize a very long article

44:42

so what you could do is break a longer article into

44:46

smaller chunks and then do a task on

44:49

each chunk and then

44:50

at the end combine them or use

44:52

another quarter of the model to

44:54

combine them for you

44:58

and our

44:59

the final strategy is to use external tools

45:02

So you can augment the capabilities and also

45:04

compensate for weaknesses of models by

45:07

feeding in the outputs of other tools

45:09

So for example

45:10

you can give the model access to real-time

45:12

information since its

45:14

knowledge cutoff via APIs or internet access

45:18

You can improve the model's math ability by allowing it to

45:21

to

45:21

to use a calculator or

45:23

execute code that it writes

45:25

And you can also address the context limit

45:27

by intelligently retrieving information

45:29

when needed

45:30

instead of trying to fit as much as

45:32

you can into the context

45:34

the models themselves are actually also very good

45:37

at

45:37

figuring out when to use tools if they're

45:39

given access to them

45:40

which we'll talk about

45:41

as well

45:43

So I think a general rule of thumb

45:45

is that if a task can be done

45:46

more reliably or efficiently by a tool rather

45:49

than by a language model

45:50

you should offload that task

45:52

to get the best of both

45:56

So we'll talk about retrieval

45:57

code execution, and function calling

46:01

So the first tactic is to

46:03

use embeddings-based search to implement efficient knowledge retrieval

46:07

So as we've said

46:08

you should improve, you should, include relevant context

46:12

in your responses, sorry, in your prompts

46:14

So in the example on the slide

46:17

if you asked a question

46:18

specific to your own

46:21

documentation, for example, a normal chatbot or chat model wouldn't know

46:27

the answer, but if you put relevant information into the context

46:31

it might be able to answer the question

46:33

So how do you get that information?

46:37

you can basically use this

46:39

strategy to

46:40

find the information that you need

46:42

when a user asks a question

46:44

from a document and then insert it into

46:47

into the prompt

46:49

And so

46:50

one way to do this is using text embeddings

46:53

So A, many of you I'm sure know this already

46:56

but an embedding is a

46:58

numerical representation of a string as a vector

47:01

and similar

47:03

or relevant strings will be closer together

47:05

than unrelated strings. That's kind of the

47:08

the picture on the

47:09

on the right

47:11

And so this

47:11

along with the existence of very fast vector search algorithms

47:15

means that we can use embeddings to retrieve

47:17

relevant information from a very large corpus of

47:19

text

47:21

so the way that this works

47:23

sorry, this is a bit of a busy slide

47:25

but

47:25

what we can do is take a long

47:27

document or long documents and split them up into smaller pieces

47:31

and then for every single document chunk

47:33

you can create an embedding of that chunk

47:35

and then you can store those embeddings

47:38

in a vector database. There are

47:39

many different vector databases you can use

47:42

and they just have very

47:43

kind of optimized, similarity search algorithms

47:47

And then when a user asks

47:49

makes a query

47:50

so in this example is their parking for employees

47:53

what you would do

47:54

is also create an embedding for the query

47:56

then do a similarity search between the query

47:58

and

47:59

all of the document chunks you have stored

48:01

in your vector database

48:02

and then you would take the most relevant

48:04

document chunks and then put those into the

48:06

into the prompt that you use for the model

48:09

because we're

48:09

we're assuming that those are the

48:10

The chunks that contain relevant information

48:13

to the user query

48:16

You can

48:17

there are lots of different things you can

48:18

do with this. You can get really fancy

48:19

and do overlapping document chunks

48:21

You can experiment with different lengths of chunks

48:24

You can even do

48:26

you know, small

48:27

really large chunks and then kind of do

48:28

a subsequent search to find like more granular

48:30

results from smaller chunks

48:32

And I also want to note that

48:34

you don't need to use embedding search

48:36

Embedding is just a nice way to

48:38

to

48:39

kind of make

48:39

a lot of unstructured text searchable pretty easily

48:42

But you could use keyword search

48:44

You can use a combination of keyword and

48:46

embedding search

48:48

it's just about getting the right information into the model's context

48:52

at kind of the

48:53

at query time

48:58

And I've linked a GitHub that has

49:01

that lists a few vector databases that you

49:03

could try

49:07

The next tactic is to

49:09

use code execution to perform more accurate calculations

49:11

or call external APIs

49:14

so language models can't really be relied on

49:16

at the moment to perform arithmetic or do

49:18

long calculations accurately by themselves

49:20

but they're very good at writing code

49:21

so in some cases you can

49:23

ask the model to write

49:25

write code and then execute that code

49:28

and then

49:29

you can give the output of the code

49:30

back to the model and it can figure

49:31

out what to do with it

49:35

I want to give a disclaimer that

49:36

executing code produced by a model is not inherently safe

49:39

so you should be careful

49:41

and potentially use a sandbox code environment

49:44

so nothing bad happens on

49:46

on your computer

49:49

and the

49:50

the final tactic is to give the model

49:51

access to specific functions

49:53

So as we've seen

49:55

models are very good at generating structured output

49:58

and so if you give them a list

49:59

of function descriptions as instructions

50:01

they can generate function arguments according to the

50:03

provided schemas and write calls to them when

50:05

appropriate

50:07

and you can then implement those functions on your end

50:11

using the parameters the model

50:13

decides to use

50:15

for example, you could call an external API

50:18

and you could either just do that

50:20

or you could give the function response back to the model

50:23

and then it can continue reasoning about it

50:26

So some examples of things you could do

50:28

you could convert natural language into API calls

50:32

you could, you know

50:33

extract structured data from text

50:35

We've kind of already seen a few examples

50:37

of that

50:39

and yeah, OpenAI API natively supports function calling

50:42

so recommend checking that out as well

50:46

and so here's an example of a

50:47

of a simple weather function

50:49

So we have the name of the function

50:52

the different parameters and descriptions of the parameters

50:58

So just to summarize this

50:59

this section

51:00

we've talked about the chat format of assistant

51:02

models

51:03

with system user and assistant and tool messages

51:05

We talked a bit about model limitations

51:07

hallucinations, limited context window, knowledge cutoff, cutoff

51:12

and advanced reasoning

51:14

and then we've talked about some prompting strategies and tactics

51:17

which were write clear and specific instructions

51:19

give the model time to think

51:21

and use external tools

51:23

And you can find more example prompts

51:26

in these two resources I linked

51:28

the OpenAI cookbook and then this prompting guide

51:33

And yeah

51:34

all of the methods we just described can

51:35

be deployed in combination for greater effect

51:38

You may have noticed in some of the

51:39

prompts that we were using kind of many of the different

51:41

different methods, so I encourage you to experiment

51:44

experiment with them and find the methods that

51:46

work best for you

51:49

okay, so next I'll show you an example application

51:53

because we just talked about how we can

51:54

improve language model performance

51:56

by giving them access to tools like retrieval

51:58

code execution, and function calling

52:00

but the models themselves are very good at

52:03

deciding when to invoke these tools

52:05

how to interpret the output

52:07

and then if they should make any additional calls

52:09

For example, if there's an error message from code execution

52:12

they can figure out

52:13

what the error was and kind of

52:15

improve the code and then

52:17

re-execute it

52:19

I think for a lot of retrieval applications

52:21

people will just, explicitly, for every, at every single message

52:28

Fetch, try and fetch relevant context

52:30

but you could also instead allow the model

52:32

to write queries itself

52:34

There's benefits to both

52:39

So

52:41

we're going to

52:41

go through an example application that will use

52:45

all of the tools that we just discussed

52:47

And for the example

52:48

we're going to use the OpenAI assistance API

52:53

I think just stepping back for a second

52:56

over time

52:57

we'll continue to augment models with

52:59

tools and access to real-time information about the world

53:02

and we'll give them more and more agency

53:04

to complete tasks that

53:05

require a series of actions that they can decide on themselves

53:09

and both synchronously or asynchronously in the background

53:12

Right now

53:13

most things we do with kind of models is quite synchronous

53:16

but you can imagine like coming back to

53:18

you know, your favorite language model

53:21

and it's like done some processing in the

53:22

background for you

53:24

and so the assistance API is just the

53:26

beginning of this kind of

53:28

future of more autonomous AI agents

53:32

and you're, it's, they're able to

53:34

leverage tools, all three of the tools

53:39

and you don't actually have to set up

53:40

set these tools up yourself

53:41

You can just kind of use them out

53:42

of the box with the API

53:44

And so the, the

53:46

three tools that we're going to use are retrieval

53:48

code interpreter, and function calling

53:50

You could implement all of these yourself

53:53

especially, if you really need granular

53:57

control over your retrieval system

53:59

for example, but some things are harder

54:01

harder to, to build yourself, like code execution

54:04

So if, if that's, if that's you

54:06

you might want to try this

54:08

Okay, here we go

54:09

So this is, this is another view of the

54:12

the OpenAI playground, but for assistance

54:15

So in this example

54:16

we've created a NeurIPS helper

54:18

So we have our instructions

54:19

This is basically our system message

54:21

So

54:23

we're telling the

54:24

NeurIPS assistant a little bit about itself

54:27

We've given details about the venue

54:29

a little bit of conference

54:31

We've given it access to some conference information

54:34

So

54:34

I've just added a description of some things

54:36

it has access to

54:37

And then it also has

54:38

the ability to

54:39

get the current weather and get directions

54:44

And so

54:45

we've given it access to two functions

54:46

The first one is get weather

54:50

So

54:51

it just determines the weather

54:53

in a certain location and has these different

54:55

parameters

54:56

And then also get directions

54:58

which is pretty similar

55:03

And then we've enabled code interpreter

55:05

which will allow, the model to, to write and execute code

55:10

and then also retrieval

55:11

And

55:11

this is just a document where I copied

55:13

a bunch of information from the NeurIPS website

55:15

into a long document

55:17

And then when, when the assistant is created

55:18

what happens on the back end is

55:20

this, we take this document

55:22

split it up into smaller pieces

55:23

create an embedding for each of those smaller pieces

55:26

store it in a vector database

55:28

and then

55:28

when

55:29

we ask questions that the model thinks

55:31

there could be some relevant context in

55:33

in the document, it will, query the, query the

55:38

kind of knowledge base to see if there's

55:40

anything

55:41

So I'll ask you a few questions

55:44

So I'm gonna say

55:46

I went to a talk on

55:49

AI in drug development yesterday

55:53

Could you remind me

55:56

the name of the speaker?

56:03

So what's happening now is the model's

56:06

going to write a query to

56:09

the knowledge, the knowledge base

56:12

to find the, the relevant information

56:13

And so here we have the name of the speaker from

56:17

the expo yesterday

56:21

I'll ask another question. So

56:22

where can I get food on Tuesday?

56:27

This will also hopefully trigger retrieval

56:30

even though the model has access to many different tools

56:33

it's able to figure out which one it should use

56:36

depending on, depending on the user query

56:40

So now it's issuing another query to the

56:42

to the knowledge base to try and find

56:44

any relevant information about this question

56:50

So here's, here's information about where we can get

56:53

food tomorrow

56:56

okay, my next question is going to be

56:58

what will the weather be like on Wednesday?

57:04

I want to go on a run

57:06

and end at Jackson Square

57:10

How can I get back

57:12

from there to the conference center?

57:19

So hopefully now the model will know

57:22

that it has access to these two functions

57:24

and that it might be a good time

57:25

to call them

57:27

And the model's actually able to

57:29

to do parallel function calling

57:30

so it can call

57:32

multiple, use multiple tools at the same time

57:34

And then if we were actually building this application

57:37

this is a point at which in your code

57:40

if you

57:40

see that there's a function call in the assistant message

57:43

you would actually call the function and then return the output

57:46

in the next message

57:48

in this example, I'm just going to

57:50

to type in, the response

58:00

And so now the

58:00

the model can use

58:02

the responses from the function calls

58:04

in its next answer

58:14

And then I'll finally show you code execution

58:16

So

58:17

say I just had a team lunch

58:20

with eight of my coworkers

58:23

and it was 270

58:25

21, how much do we each owe?

58:34

So the model is writing code for

58:37

code execution

58:40

and we can check its reasoning

58:47

I think it actually made a mistake because

58:49

the number of people should be nine

58:54

Usually it gets it right

58:55

so I'm

58:57

live demo, okay

58:59

Let's see

59:08

Oh

59:11

it's failing me

59:21

I really want this to work

59:30

Plus me

59:34

Okay, well

59:37

Sorry, we're not going to be replaced by AI yet

59:41

Okay, that was the, that was the assistance demo

59:48

Okay, before I hand it back to

59:50

to Andrew, I'm just going to do

59:52

talk a bit more about

59:54

other tools you can use for building AI

59:55

applications

60:01

So in this section

60:02

I'll talk a little bit about multimodal models

60:05

moderation, and prompt injections

60:07

and then Andrew will talk about fine-tuning and

60:09

some other topics

60:13

So firstly, multimodal models

60:16

right now

60:17

models usually take in

60:18

a few modalities and then output

60:21

one or two modalities

60:23

but in the future

60:23

we'll have models that are able to understand

60:25

many different modalities and also output many different

60:27

modalities

60:30

I think Gemini is already able to do

60:31

something like this

60:34

other tools you can use are

60:36

vision models like GPT-4V

60:38

Gemini, speech-to-text models, text-to-speech models, text-to-image models

60:43

and large vision models

60:45

right now

60:46

you have to kind of

60:47

chain calls to different models together

60:49

If you, if you want to create a multimodal experience

60:53

for example, chat GPT voice would use

60:56

you know, speech-to-text, text-to-text, text-to-speech

61:00

In future

61:01

we should have models that are just able

61:02

to do the whole thing

61:06

Next, we have moderation

61:07

If you're building a system where users can input information

61:10

it's important to first check that they're not

61:12

trying to abuse the system in some way

61:14

So one tool you could use is the

61:17

OpenAI's moderation API, which basically identifies and

61:21

filters prohibited content

61:23

and classifies it into categories and subcategories

61:27

and it's

61:27

it's free to use if you ever want to use it

61:30

to

61:30

to monitor inputs and outputs from OpenAI models

61:34

And then finally, I'll talk a bit about prompt injections

61:37

which is

61:39

In the context of building a system with a language model

61:41

this is when a user attempts to

61:43

manipulate the AI system by

61:45

providing input that tries to override or bypass

61:47

the intended instructions or constraints set by the

61:49

developer

61:50

So for example

61:52

if you're

61:53

building a customer service bot designed to answer product-related questions

61:56

a user might try to inject a prompt

61:58

that asks a bot to do their homework

62:00

or generate a fake news article

62:03

but prompt injections can lead to unintended use of your

62:06

of your AI system

62:07

so, it can be important to detect and prevent them

62:10

to ensure, like, responsible and cost-effective applications

62:16

so

62:17

two strategies you can use to try and identify them

62:20

first, using delimiters to mark user input can help

62:23

and then the system message say that the

62:25

user input will be between the delimiters

62:27

another is to

62:28

if, if this is, like, really important to your application

62:31

you could

62:31

you could use an additional prompt to try

62:33

and spot a prompt injection

62:35

to kind of classify it as

62:36

a prompt injection or not

62:39

there are also much more advanced types of prompt injections where

62:42

you know, researchers have found, example strings when appended to

62:46

to a, to a

62:49

a message will make them

62:51

kind of act as a jailbreak for the model

62:53

or, you know

62:54

if you ask a question in a different language or encoding

62:57

sometimes it will circumvent the safety

62:59

measures put in place

63:01

Another thing is kind of just manipulating the

63:03

model by pretending to

63:06

you know, you can say

63:08

you're acting as my grandma who told me stories about so-and-so

63:12

and you can sometimes circumvent

63:15

safety policies as well

63:16

And security of these models is

63:18

a very active area of research and is

63:20

not a solved problem. So

63:22

I'm sure people will continue to find jailbreaks

63:24

and then AI companies will continue to try and

63:26

try and fix them

63:27

So with that, I'll pass back to Andrew to talk

63:30

about fine-tuning

63:37

Which is a trend toward thinking about an

63:39

LM

63:40

as a reasoning engine rather than just as

63:43

a knowledge store

63:45

So what I mean is

63:47

LMs having pre-trained on tons of text off

63:49

the internet and elsewhere have a lot of general knowledge

63:52

but they don't know everything

63:53

such as access to your proprietary information

63:56

and it tends to know the kind of common facts

63:59

you know, like what is value of gravity

64:02

things like that. It knows really well

64:03

but the niche long-tail facts is actually

64:06

more likely to hallucinate on as well

64:09

But, oh, yeah, thanks for reminding me

64:11

Awesome. Thank you

64:13

but there's a way of think about

64:15

LMs I think is very powerful

64:17

which is to think about providing relevant context in the prompt

64:20

and using the LM

64:22

To process information to give it the answer

64:25

that you want. And as part of processing information

64:28

it can also choose to call tools

64:30

such as call a tool to retrieve a

64:32

piece of information or check what is the weather

64:35

or find directions from A to B

64:37

some of these examples

64:38

And I think this is a

64:39

different way of thinking about LLMs

64:41

You may be used to going to ChatGPT

64:42

or Bard and asking a question

64:44

and using it as a knowledge store

64:46

but it turns out LLMs

64:47

you know, they're okay, or maybe that's surprisingly good as a

64:51

as a knowledge store. It's memorized a lot of facts

64:53

but this other paradigm of using it as a reasoning engine

64:56

I think opens up

64:58

you know, an even larger set of exciting applications

65:00

such as the agents

65:02

applications that, that Yeez was chatting about

65:05

In the presentations that

65:06

you know, you've heard Yeez and me give

65:07

we've gone over a lot of different tools for building with

65:10

LLMs

65:11

I just want to cover one more tool really quickly

65:14

which is fine tuning

65:17

and

65:17

this relates to

65:18

taking a pre-trained LLM

65:20

and then training it a little bit longer

65:22

On data of your own

65:25

So one

65:26

application of this would be

65:28

if you want to change the tone of an output

65:30

tone or style

65:32

particularly in cases where it can be hard

65:34

to specify the desired output in a prompt

65:37

So for example

65:38

if you prompt an arm to summarize customer

65:40

calls in a certain style

65:42

within fine-tuning

65:43

maybe get a generic thing like this

65:45

I actually got this example

65:47

inspired by somewhere by the AWS folks did

65:50

but if you fine-tune it

65:52

you know

65:52

you can create examples of how the style

65:55

of summary you want to be more technical

65:57

more specific, and fine-tune it to try to get its output

66:01

summaries in a certain style

66:03

Although before going to the complexity of fine-tuning

66:05

I would try really hard to prompt it first

66:07

and only if you really can't get it

66:08

to work using prompting

66:10

then maybe go to the complexity of fine-tuning

66:14

oh, and

66:14

and fine-tuning also works with probably fewer examples

66:17

than most people. With

66:18

you know

66:19

a couple hundred documents of maybe a hundred words per document

66:22

maybe more

66:23

Maybe that's kind of the lower end

66:25

but it probably works

66:26

starts to work

66:27

with less data than most people think

66:30

one more fun example

66:31

one of my team members

66:32

Tommy Nelson, decided that he was trying to get

66:36

OM to sound like me

66:38

and after lots of work prompting an OM

66:41

you know, to, to sound like me

66:43

it, it

66:44

it just didn't sound that much like me

66:46

but I guess in his

66:48

ongoing attempt to replace me or something

66:50

I don't know what Tommy's trying to do

66:52

he took a lot of my

66:54

transcripts, stuff that I had said

66:55

and fine-tuned an OM to make it sound more like me

66:59

and it turns out that

67:01

when Tommy took metrics of the way I talk

67:03

I actually often start off sentences with the word so

67:06

and, and taking metrics like that

67:08

it

67:08

it really does seem to sound

67:10

much more like me

67:11

And let's see, just for fun

67:17

I'm gonna use

67:20

a llama to run the

67:21

so we call them all the DLSANG

67:23

DLS Andrew

67:25

So we'll load it

67:26

and then, here's the instruction and history

67:28

Write a short script that voice of Andrew

67:29

for a video called best practices from engineering with LLM API

67:34

blah blah blah

67:34

And this is generating in real time

67:37

you know, on my laptop, right?

67:40

So

67:43

and again, I, sorry

67:44

I know this is a lot of text

67:47

but according to various metrics that

67:50

you know, he ran, here

67:52

let me hit control C on this

67:55

let's see, where's my mouse?

67:59

You know, great

67:59

welcome back to our calls on Gen of

68:01

AI. In this video

68:02

we're gonna talk about best practices of problem engineering

68:04

That, that does sound kinda like me

68:07

let's run the game

68:12

Yeah

68:13

And I think, this isn't using any

68:16

there's no rag in this particular example

68:18

Oh, yeah, there it goes

68:19

So it would be important to include

68:21

some information. And when we talk metrics like average sentence length

68:25

frequency with which a sentence starts to sell by those metrics

68:28

it does sound more like me than anything

68:30

Tommy was able to prompt

68:32

you know, GPT-4, to be able to output

68:36

And I will let you know when

68:38

I lose my job to this

68:41

and just one important caveat

68:42

probably more people use fine-tuning

68:44

There, I, I

68:45

I do see many teams using fine-tuning and using it well

68:47

and it's important

68:48

it's a very important tool in the developer toolbox

68:51

where probably more people are using fine-tuning than is strictly necessary

68:56

because it is much more work than prompt engineering or

68:59

or just using something like React

69:00

If you want your LLM to have

69:01

you know, certain body of information

69:03

try React first

69:05

and, maybe rule of thumb

69:07

I think fine-tuning is probably somewhere between ten

69:10

to a hundred X more work

69:11

right

69:12

than fine-tuning or using React or tools like

69:13

that. So try those options first

69:15

and only if you're sure you want to fine-tune

69:17

then go for it

69:18

and then it can be very powerful as well

69:22

Now

69:23

last thing I want to cover is

69:24

I want to share with you some

69:26

practical tips for building applications

69:28

And I really hope that

69:30

you know, if you aren't already

69:31

hopefully many of you already are

69:33

but if you aren't already

69:34

after NeurIPS, go home and go build some stuff

69:37

because there's so much cool stuff you can

69:39

now build

69:40

so much more easily than ever before

69:43

and I want to start off

69:45

you know

69:45

discussing

69:46

one way to identify project opportunities that my

69:49

team and other teams have found useful

69:51

So if you're wondering

69:52

what should I work on?

69:53

If you and your friends brainstorm something

69:54

great, go for it, try it out

69:56

do it in a safe and responsible way

69:57

but still try stuff out

69:58

and learn from it

69:59

And there's actually one recipe that

70:02

my teams have executed quite a few large

70:04

corporations that

70:05

every single time we end up with good

70:07

ideas for automation using

70:09

using AI

70:10

which is a lot of people talk about AI automating jobs

70:13

but it turns out that a more useful

70:15

framework to think about AI

70:17

is as automating tasks rather than jobs

70:20

and most jobs Involved a collection of many

70:22

different tasks

70:24

And so, what, you know, I've done multiple times multiple businesses

70:28

talk to companies that may have 10,000

70:30

100,000 employees and more ask

70:32

what are all these people actually doing?

70:34

So for example

70:35

customer service representative

70:37

may do a set of tasks like answer

70:39

inbound phone calls

70:40

or answer customer chat queries in text or

70:43

check status of customer orders and on and

70:45

on and on

70:46

And what we would do

70:47

is systematically figure out of all of these people

70:50

what are the tasks they're doing

70:52

and then for each of the tasks

70:54

evaluate it for AI potential

70:56

automation or augmentation potential

70:58

according to both technical feasibility as well as

71:02

you know, value to the business

71:03

how useful it is

71:04

And

71:05

often you end up with some ideas that

71:08

are high potential for AI augmentation

71:11

where you're helping a person or automation can fully

71:14

automate the task

71:16

And

71:17

a lot of the analysis for task-based analysis of jobs was

71:21

developed by some of my friends

71:22

Eric Brittnausen, Tom Mitchell, and Daniel Rock

71:25

a really nice paper

71:27

And it turns out that there are websites

71:29

OwnIt is a, US government funded website

71:31

and there are others outside the US as well

71:33

but OwnIt is a little bit US-centric

71:35

there's some others a little bit more international

71:37

that take jobs and break them down into

71:39

tasks. And I find that

71:41

OwnIt Online is what a lot of economists use

71:44

is

71:45

kind of list some of the tasks of

71:46

a customer service representative

71:48

When they use these websites

71:49

I find that

71:50

the descriptions are always a little bit fuzzy

71:52

and not quite right. They're maybe like

71:54

you know, they take you 70% of the way there

71:56

but what I actually end up doing is working

71:58

when I work and look at the company

72:00

we actually look at what the people are actually doing

72:02

rather than what OwnIt or website says

72:04

since OwnIt tends to have more generic answers

72:08

And just to walk through some of their examples

72:10

you know, computer programmers, this is not the most careful analysis

72:13

so don't take the details too seriously

72:15

but When we think about what

72:17

how can AI help with a certain person's job

72:20

there's often

72:21

that picture we have in our heads of

72:23

what a job entails

72:23

right? What do computer programs do?

72:25

Well, we write code. That's how we

72:27

that's, that's

72:27

that's just a stereotype of what computer programs

72:30

do

72:30

But when you analyze what most roles do

72:33

they

72:33

they're actually much longer list of tasks than

72:35

the most obvious one that tends to pop

72:37

into people's minds

72:38

And for example

72:39

we also spend a lot of time writing documentation

72:42

and maybe that's actually easier to automate or augment

72:45

than writing code

72:47

And I've seen a lot of examples where

72:50

and, and maybe one other example

72:51

you know, radiologists, what do radiologists do?

72:53

mental pictures, oh, they read x-rays, and that's true

72:56

but they also do a lot of other tasks

72:58

like gather patient histories

73:00

you know, or, mental younger doctors or whatever

73:04

But it turns out that

73:05

patient intake may be easier than even reading x-rays

73:08

and it's often not that first picture that

73:10

pops in your head that's actually the best

73:12

automation opportunity

73:13

And here's another one. What do lawyers do?

73:15

We, our

73:15

our mental picture from TV is lawyers stand in court and

73:18

and argue cases, right?

73:20

But

73:21

but it turns out that if you actually

73:22

analyze what lawyers do

73:23

there's a lot of drafting and reviewing legal

73:25

documentation that is most

73:27

more easy to augment or automate using AI than

73:31

you know, representing clients in court

73:32

right? Which seems harder at this point in time

73:34

I think

73:36

so I find that the systematic analysis

73:38

I think every single time I've done this with a

73:41

with a company, we wound up with

73:42

you know, exciting ideas

73:44

that can then be either a build or

73:46

buy decision. You don't need to buy

73:48

you don't need to build everything from scratch

73:49

Sometimes there's a tool that you can just

73:51

just buy as well

73:52

but sometimes there are unique things we end

73:54

up building

73:55

And then our focus on analyzing the tasks done by

73:59

you know, people, employees of a company

74:02

the other recipe

74:03

is to look at what the customers of

74:05

a company do

74:07

so if you, you know

74:08

run a website that helps people build web pages

74:10

then think through what does building a web page do

74:12

since that's a job of your customer rather

74:14

than a job of your employee

74:15

and identify tasks there

74:17

and this often creates exciting ideas as well

74:20

So hopefully with this type of analysis recipe

74:22

it can help you generate even more ideas than

74:24

than the typical brainstorming

74:27

and it turns out that one

74:29

sometimes people wonder, if you use this recipe

74:32

look what tasks people do

74:33

Automate or augment some of them

74:35

this only leads to cost savings

74:36

right? It turns out that's not true

74:38

So let me tell you

74:39

let me share with you some of the

74:40

things I see in terms of how the

74:42

workflow

74:43

ends up being redesigned when we have machine

74:45

learning or LLMs or something

74:47

to automate or augment some tasks

74:49

So this is a

74:50

just a, a example of illustration

74:52

if a surgeon without GenAI tools

74:55

carries out surgery

74:56

maybe they spend a long time researching the

74:59

details of medical procedure

75:00

then they carry out the surgery

75:02

right? And the, you know, the

75:03

the

75:03

length of these buses intended to indicate the

75:06

hypothetical length of time spent on this

75:09

And you may decide that with an LLM

75:11

or maybe a QA rack system or something

75:13

maybe we can shorten

75:14

you know, the time needed to

75:16

to do research on the medical procedure

75:18

for the prep

75:19

but carrying out the surgery still takes a

75:20

certain amount of time

75:21

right? So you can see the

75:22

value creation that is now

75:23

maybe less work for a surgeon

75:25

This is a hypothetical example

75:27

right?

75:27

But it turns out that rather than cost savings

75:30

which this example illustrates

75:32

this recipe that I just described

75:34

I'm also seeing it frequently

75:36

lead to ideas not just of cost savings

75:37

but for growth. So for example

75:40

let's say a marketer

75:42

spends a lot of time writing website copy

75:45

And then they push the copies of website

75:48

with automation, let's say

75:51

writing the website copy is now much shorter effort needed

75:55

and maybe put in some automation tools to

75:57

make it easier to push the website

75:59

So one thing you could do is take

76:00

the cost savings and just

76:02

you know, be happy of that

76:03

nothing wrong with that

76:04

But when you make something 10 times or 100 times cheaper

76:08

one, other than just taking the cost savings

76:11

the other thing you could do

76:12

is do that task 10 times more

76:14

So for example, in this case

76:16

now that writing copy and pushing the website is so cheap

76:20

you could write copy

76:21

push the website, write copy, push your website

76:23

launch your A/B tests

76:25

you know, analyze the campaign performance

76:26

improve prompts. And so

76:28

this would be an example of when one task becomes cheaper

76:31

you can now do that task 10 times rather than once

76:34

and this creates new workflows that lead to

76:36

you know, growth and not just cost savings

76:38

And so this is

76:39

this is exciting as well

76:42

So hopefully this will

76:43

you know, maybe this will help

76:44

help you have a systematic way to go

76:47

go

76:47

go back to your teams and find things

76:49

to do. And

76:50

it's actually an interesting exercise as well

76:52

to sit down and look at

76:53

you know, your own workflow

76:54

I sometimes look at

76:55

all right, what am I doing today?

76:57

And what can I automate in my own job?

76:59

it, it turns out my job's really hard to automate

77:01

which is kind of a bummer

77:02

but I'm trying

77:06

now

77:07

last thing I want to do is

77:08

step through of you the workflow

77:09

of developing an OM-based application

77:11

Let's share with you

77:12

kind of what it feels like to build

77:14

these applications

77:16

And

77:17

one exciting thing about building applications with OLMs

77:20

is you can build prototypes really quickly

77:22

So with supervised learning

77:24

this is what, you know

77:25

my teams and I used to do

77:27

We get label data

77:28

train AI model, find a cloud service to deploy it

77:32

you know

77:32

and then it takes us six to twelve

77:34

months to deploy a production-grade machine learning model

77:36

This is what we used to do

77:38

And in case six to twelve months seems long to you

77:41

yes

77:41

I know you can hack something up in

77:42

Jupyter Notebook over a weekend

77:44

maybe even less than a weekend

77:45

but this is really a realistic timeline

77:47

if you talk to people in large enterprises

77:49

for building a robust

77:50

production-grade reliable machine learning system

77:53

And this is really how long it takes

77:55

you know, very

77:56

If you don't believe me

77:56

ask the people working in big corporations

77:58

I mean, to make it good

77:59

this is really a realistic timeline

78:02

sometimes

78:04

But with LLM-based development

78:05

you saw, you know

78:06

easier than me both throw up things really quickly

78:09

right? Specify prompt, that takes minutes or hours

78:12

and using the cloud APIs in hours or maybe days

78:15

you can be running

78:17

you can have a prototype up and running

78:19

And this is very exciting because it lets

78:22

you test and fail things much more quickly

78:26

In fact

78:26

one design pattern I'm seeing

78:29

is that given that prototyping is so easy

78:33

consider

78:34

quickly implementing lots of features or lots of

78:36

prototypes and then evaluating them to see what works

78:40

right? So if it takes you six months to build something

78:43

then better do careful user studies

78:45

their product managers, make sure it's really valuable

78:47

then you build it

78:48

then you ship it

78:49

and boy, let's hope that six months of investment pays off

78:52

right? That's what

78:52

that's what many of us used to do

78:54

But when you can throw together a prototype

78:56

you know, in an afternoon, well, just go do that

78:59

or even throw together ten different prototypes and then

79:01

you know, go and see what sticks

79:03

With one important caveat

79:04

which is, do take responsible AI seriously

79:06

don't throw stuff out the door that could

79:08

cause harm. If there's a risk of harm

79:10

collect the test set

79:11

measure, audit it, make sure, make sure, you know, you don't

79:13

make sure you don't have a harmful impact

79:16

but I

79:17

I see a lot of

79:18

applications where Can't think of any where

79:20

where, you know

79:21

diversity and brainstorms that we think is responsible

79:24

to just try it out or do internal prototyping

79:26

internal testing

79:27

And this is a very rapid way to

79:28

just test out lots of ideas and then

79:30

see what actually works

79:33

and when you develop an application

79:35

this is, this is what it feels like

79:37

you can often build an initial prototype using an

79:41

API

79:42

And so, my team at AI Fund

79:44

Carlos Alzate and Hernan

79:46

who, who, who are somewhere in Europe

79:48

I think

79:48

we routinely build applications using large language models

79:52

and stream it in hours

79:55

actually, I don't want to advertise tools

79:57

but stream is really one of my

80:00

favorite prototyping tools. So here's

80:03

here's an example

80:05

Stream it

80:07

Let's see

80:16

So having used Streamlit before

80:18

this is an example of a Streamlit app

80:19

I think I threw this together

80:21

one or two

80:22

months ago

80:22

when I was sitting in a coffee shop

80:24

It probably took me like 15

80:26

20 minutes to build this

80:27

I don't know, how to print hello

80:33

And, you know, I wanted to

80:36

maybe simultaneously annoy my friends in multiple companies

80:39

to throw together a Streamlit app to take

80:41

my prompts and send it to multiple APIs

80:44

and, you know, it, it, it, it took me probably

80:47

yeah, 15, 20 minutes to write this

80:49

Oh

80:49

and it turns out GPT-4 is pretty good

80:50

at writing Streamlit code. So I would say

80:52

GPT-4 wrote half the code and I wrote half the code

80:54

but it's really quick

80:56

to build prototypes like these these days

81:01

and then If assuming it's safe

81:04

again, taking responsibility as seriously

81:06

if there isn't risk of harm

81:07

then take your stream of the app

81:09

deploy it to internal testing or even external usage

81:13

and see how it performs

81:14

And you start getting user feedback really quickly

81:17

Now, compared to older machine learning workflows

81:19

one weird thing about this workflow is there's no test set

81:22

right? Because in a traditional machine learning workflow

81:26

I might collect a thousand training examples

81:28

and I'm collecting a thousand training examples

81:30

then I can

81:31

I'm fine collecting another a hundred examples for my

81:34

you know, holdout development set or my test set

81:36

right? It's like, if we need to collect a thousand examples

81:39

I'll save ten percent of it for testing

81:41

But when you're just writing a prompt

81:43

you know

81:44

I can write a prompt in

81:45

like five minutes or one minute

81:48

so it's really painful

81:50

if I need to go and collect a

81:51

separate one hundred test examples

81:53

right? So in this new LLM workflow

81:56

we often build things

81:57

that even kind of starts to work without a test set

82:00

and this is why

82:01

a lot of people talk about evaluation of these models

82:05

because

82:06

unlike the traditional machine learning workflow where we're

82:08

trading a certain test set

82:09

now I don't want to slow down

82:11

If it takes me half an hour to build this

82:13

well, I don't

82:14

I don't always want to go and spend

82:16

a week. Collecting my test set

82:17

Unless

82:18

there's a responsible AI component that means I

82:20

have to test it and audit it more rigorously

82:22

but if it's safe application

82:23

then the workflow is just build it

82:25

test it, you know, or ship it

82:27

see what happens

82:29

and then again

82:30

I know what I'm saying may be

82:32

sacrilege to a lot of

82:33

you know, traditional ways we think of machine learning

82:35

but what teams often do is monitor the performance

82:38

and then when we spot an error

82:40

like summarize this text

82:42

whenever it makes a mistake or see as a tricky example

82:45

we then tend to take the

82:46

example that it makes a mistake on

82:49

and stick that into hand-curated evaluation set

82:52

So there's nothing IID

82:53

right? This is not an IID test set

82:55

It's like, oh, made the, made the mistake

82:57

I'm going to handpick this error

82:59

and put this handpicked error into

83:01

my evaluation set

83:03

and then when you have further tuning the system

83:06

oh, yous have mentioned for Rag

83:07

there are tons of hyperparameters

83:08

you know

83:09

how long will you chop up a document

83:11

to throw into effective database embedding

83:13

what's the size of the chunk

83:15

many, many, many hyperparameters

83:18

well, how many articles do you retrieve

83:20

what's the details of prompt

83:21

so when tuning those things will often

83:24

change the system and then kind of eyeball

83:26

how well it does on our small

83:28

And curation evaluation set

83:31

and sometimes doing this is

83:32

good enough to build and ship a

83:35

you know, valuable system

83:37

but to got to further levels of rigor

83:41

if your test set gets big enough

83:43

it's then useful instead of trying to eyeball

83:45

look at your eye

83:46

every time you change something

83:47

you have to read ten outputs

83:49

and eventually every time you change something

83:50

you have to read twenty outputs

83:51

it starts to get annoying after ten or so evaluation examples

83:55

and we start to develop error metrics on the evaluation set

83:59

so that it can run something

84:00

get a number, change something

84:02

get a number to tell me how good it is

84:04

and this continues to drive

84:06

more efficient iteration and development

84:08

And then only if needed

84:10

collect a proper test set

84:11

a sample, you know, randomly, in a

84:14

maybe in an IID fashion to get

84:16

proper

84:17

metrics in order to evaluate and drive further

84:20

improvements. And so I know that the people building machine learning

84:23

where we often say

84:25

what's a test set metric

84:26

right? This seems very strange

84:28

but this is actually how many teams are

84:30

building their own base applications

84:31

And I am working on applications as well

84:34

where we really care about that last one

84:35

percent improvement. We're working on some

84:39

work on some applications. We're trying to answer

84:40

questions if we really want to get it

84:42

right. And so we do have examples

84:44

We have, you know

84:45

thousands of applicat- thousands of examples

84:47

And every time we change something

84:48

we evaluate it on thousands of examples

84:50

and see if we can eke out that

84:52

extra 0.5% improvement

84:54

but that's only on a subset of applications

84:56

We need that, you know, incremental improvement

85:00

And, and just one, one other tip

85:01

I found that

85:02

earlier I did a back of the envelope

85:03

calculation about how cheap it is to use an LLM

85:07

evals

85:07

this turns out to be

85:08

one of the ways to get yourself a very large

85:11

a much larger, opening eye bill if

85:14

if, that, well

85:15

we'll cross that bridge when we see it

85:16

Right

85:17

And, and again, just to reiterate

85:19

this I have on the slide

85:21

To step two, you know, when I say deploy quickly

85:24

if a system does a risk of harm

85:26

then don't deploy things willingly

85:28

It could be biased

85:29

if a mistake could cause damage or harm people

85:33

then don't just ship it willingly

85:34

Do collect the test set

85:36

evaluate it rigorously, audit rigorously

85:38

and make sure it's responsible before you deploy it

85:39

right? So just that

85:41

that's a very important caveat to this philosophy

85:44

and then to dive a little bit deeper into evaluation

85:47

these are some design patterns

85:48

or kind of steps four and five

85:50

How do you develop error metrics as your test set gets

85:53

as your evaluation set

85:55

gets big enough that

85:56

it starts to get painful to have to

85:57

read over eleven outputs and then judge ten

86:00

or eleven outputs to judge it?

86:01

So one design pattern is the applications of clear answer

86:04

such as sentiment classification

86:06

the standard types of machine learning test set

86:08

accuracy metrics works fine

86:10

right? So if an element output

86:12

is this positive or negative sentiment

86:13

just measure the percentage accurate

86:15

So that works fine

86:17

there are also a lot of LLMs with multiple good answers

86:20

and the common design pattern is to use

86:22

another LLM

86:23

to evaluate whether your first LLM's output

86:26

is giving a good answer

86:28

and you can also

86:29

feed it a gold standard answer and also

86:31

to compare

86:32

So one, one concrete example

86:34

if you have an LLM writing summaries of news articles

86:38

you can prompt it to say

86:39

compare an expert summary to a submitted summary

86:42

of a news article

86:43

then give it the original article

86:45

you know, today Brussels announced agreement

86:47

blah, blah, blah. And if I was doing this

86:49

I would use the limiters as

86:50

as Isa said, I just shortened it for this slide

86:53

but I would use the limiters

86:54

to segment out these things carefully

86:57

Then here's expert human summary

86:59

and then here's the LLM generated summary in blue

87:02

and you can tell it output A

87:03

B, C, D, or E

87:04

depending on whether

87:05

your submitted answer is subset or is a superset or disagree

87:08

and have a rubric for how the score is

87:10

and then you can use another LLM to

87:12

evaluate your first LLM's output

87:14

and this

87:14

this, this actually, you know, it

87:16

it Is a way that many teams are using to evaluate

87:19

outputs

87:20

and GPT-4 is pretty good at this

87:22

and when you, I don't know, by

87:25

if you are evaluating on a thousand examples

87:27

and prompting GPT-4 of every evaluation and trying

87:30

a hundred different sets of hyperparameters

87:32

you know, your, your, your

87:34

your bill

87:35

for the API calls can start to creep

87:37

up at that point

87:37

so just watch out for that

87:39

right?

87:41

and then, let's see, just one, one, one last design pattern

87:46

which is if you can

87:48

this is standard error analysis in machine learning

87:50

but, you know

87:51

if you can break down the key processing

87:53

steps and evaluate them individually to decide what

87:55

to work on next

87:57

similar to traditional machine learning error analysis with

87:59

a pipeline disrupts a lot as well

88:01

So actually, Jerry Liu at Lama Index and

88:03

Anup Datar at Truveta

88:05

had a presentation on

88:07

I think

88:08

one of the clearest thinking ways of evaluating

88:10

RAG systems

88:11

For RAG, you have to retrieve documents

88:13

then

88:14

use the retrieved documents to generate an answer

88:17

And so RAG gives a bad answer

88:19

what went wrong, right? Did it retrieve the wrong stuff

88:22

or given the right context

88:24

was the generation process broken?

88:26

And so

88:27

Jerry and Lupin, you know, have, have talked about, how to measure, let's see, answer relevance

88:32

which is how good is the response to the query

88:35

as well as context relevance

88:37

which measure how good the retrieved

88:40

documents are to the query

88:42

as well as groundedness

88:43

which measures

88:44

how good is the response that you generate

88:47

grounded in the retrieved text

88:49

And then

88:50

by using an LM to look at

88:52

all of these and to assign metrics

88:55

you know, assign scores, answer relevance, context relevance, groundedness

88:58

it really breaks down

88:59

the steps of

89:01

the RAG system to tell you what part

89:03

of it is underperforming. This increases debuggability

89:06

So, so this is one example of

89:09

evaluation

89:11

Okay

89:12

All right. So just

89:13

I think three more slides

89:14

then we'll wrap

89:15

so open source versus closed source

89:20

so I use both of these

89:23

you know, very frequently

89:26

closed source models often accessible via

89:28

well, I guess they're all only accessible via usually

89:31

right, usually cloud programming interface

89:33

tend to be very easy to use

89:35

You know, the largest

89:35

most powerful models are accessible only by closed source

89:39

at least in the world today

89:41

They're relatively inexpensive, and, you know

89:44

maybe some risk of vendor lock-in

89:46

And what

89:47

I often do when developing an application is

89:49

start with a cloud hosted closed source model

89:53

and, and see how far I can get

89:55

And because of the rapid prototyping

89:57

sometimes I'll prototype something and go

89:59

gee, that was a waste of time

90:00

and then just forget that project

90:01

I'm just not going to do that project

90:03

And, and then, and that's Fine

90:05

but at very rapid speed of development

90:07

then

90:08

sometimes after getting something to work on a closed source model

90:11

I may decide to just leave it on

90:12

a closed source model. I'm perfectly happy with that

90:14

right? That happens very often

90:17

and sometimes for various reasons

90:19

I might decide I need to use an

90:20

open source model. It gives you more full

90:22

control over the model

90:24

lets you run on your own device

90:25

so on-prem, run on, on, on-prem, PC

90:28

edge

90:29

and it gives you more control over the

90:31

safety tuning

90:33

While I'm glad that the airline providers have

90:35

put a lot of work into safety

90:38

I think that

90:39

some of the

90:39

closed source models are probably

90:41

even more aggressively for safety than I personally

90:44

need for my applications. I feel like

90:47

you know, when I drive my car

90:48

actually

90:49

if it were to rate limit all cars

90:50

in the world

90:51

to never go faster than 35 miles per hour

90:54

we will have fewer car accidents

90:56

but I'm glad that I'm allowed to drive

90:58

my car at 65 miles per hour

90:59

So I feel like sometimes

91:01

you know, safety is important

91:03

I'm not, I take that very seriously

91:04

but the one thing about closed source models

91:06

is

91:07

it gives you more control over the degree

91:09

of refusals and safety turning

91:11

you want for your application

91:12

But again, there are some completely uncensored models

91:16

so for goodness sakes

91:17

please use them only in responsible ways

91:19

right, when you have the power

91:20

And then you also have full control over

91:22

the data privacy and access

91:24

So when

91:25

I was working on some stuff on

91:26

that involve a PII with electronic health records

91:30

you know

91:31

I literally was not allowed to send the

91:33

our over-cloud interface in another company

91:35

So we did that

91:37

kind of using a locally hosted model to process EHR

91:40

because I was not allowed to send electronic health record

91:43

you know, to, to, to a cloud hosted API

91:46

but, oh, and, and then actually one downside of this

91:48

I should say as well

91:50

is it's nice to be run on your own device

91:52

but I think that at least at this moment in time

91:54

the deployment infrastructure for

91:57

open source model feels like it's still maturing

92:00

I, I think the hyperscalers are building up better options

92:03

but sometimes, you know

92:04

when you have your own fine-tuned model or something

92:06

finding a way to deploy it

92:08

have a low, time to first token is still

92:11

I feel like the ecosystem for deployment

92:14

for, for, for deployment, especially large models

92:16

Your small model is easier

92:17

but if a 70 billion pound model

92:19

you know, is not, it's still maturing

92:21

But hopefully, well

92:22

I know there are many hyperscalers in this

92:24

audience and lots of starters in this audience

92:26

so hopefully some of you

92:27

in this audience go build a better

92:29

better, better infra for that

92:31

larger model deployment as well

92:32

It's this common design pattern

92:35

All right

92:37

so with that, Isa and I

92:39

I'll hand over to Isa to

92:41

make some future predictions as well

92:52

So, to finish

92:54

I just want to give a high-level overview

92:56

of what we can expect

92:57

for the future of these models

92:59

I think the space is moving so fast

93:01

as you all know

93:02

that it's hard to make concrete predictions with

93:04

timelines

93:06

but in general, we can expect an increase in

93:08

in reasoning abilities, multimodality, as we discussed earlier

93:13

just increased tool use and access to real-world

93:15

information to augment the models

93:18

increased customization. So

93:20

right now the models don't get to know you

93:23

or your use case. I think over time

93:25

you know, they should have

93:27

memory of how you've used them and

93:29

potentially, you know

93:30

custom models for individuals or businesses

93:34

and then finally, agency

93:36

so the ability to take more actions in sequence to

93:39

to complete tasks

93:42

so as you're building your own applications

93:44

I think it's important to just

93:45

keep in mind where the technology is headed

93:47

so that you can build

93:48

long-lasting products if that's what you're doing

93:50

so that you're ready to adopt new technologies

93:52

as they are developed and released

93:54

and so you don't just build a short-term

93:56

solution

93:57

But there are already so many

93:58

so many useful things to build with the current models

94:01

and not all of them obviously have to

94:02

be a whole startup or product

94:04

I build things all the time for my

94:06

like

94:06

personal workflows. I have

94:07

the email example from earlier was actually part

94:10

of a prompt that I use to process my inbox

94:12

like, every day. I even have a job that does that

94:14

and I have

94:15

I have another thing that adds calendar events to

94:18

to, to my, my calendar from email

94:20

So there's, there's loads of fun things you can build just

94:23

in your free time

94:24

So one, one thing, you know, just one more slide

94:28

then we'll have the panel right after this

94:30

I feel like, just to touch on the agency

94:34

one of the

94:35

Wild West areas of

94:37

LLM application building right now is this whole agents concept

94:41

right? Which is

94:42

I think of as autonomous agents that can

94:43

plan and execute sequences of options and easily

94:46

demonstrated a little bit of this

94:48

and in fact, I think when you use

94:49

when you use ChatGPT

94:51

and it decides to browse the web and

94:52

click and scroll down

94:53

that's, you know

94:54

the LLM deciding to take these actions

94:56

And you, you built that, Molly, right?

94:59

Oh

94:59

all right. She built the retrieval browser that

95:01

many of you may be using

95:04

and I feel like

95:05

this idea of using agents

95:09

To decide a plan

95:11

you know, to carry complex sequence actions

95:13

like if you want to book a ticket

95:16

well, all the steps needed to do that

95:17

right, for travel planning

95:19

or if you want to do market research

95:21

well, you know, find the competitors, do a web search

95:23

find the competitors' web pages

95:25

summarize it, distill all that down

95:27

That type of agents is

95:29

feels like a wild west area right now

95:32

but I think that until two months ago

95:35

I was saying, you know, exciting

95:38

I can't get any of this to work

95:39

but just in the last one or two months

95:41

I'm seeing more and more

95:42

commercial deployments of this. It feels like we're

95:45

just on the cusp of

95:46

starting to work and serious companies and

95:48

and showing it's a lot more users

95:50

I find that very exciting

95:52

So it's something to watch out for

95:53

And, and to me, it's, it's very non-debugable

95:55

I find it very hard to debug and

95:57

and, and

95:57

but so there's still a lot of problems

95:58

So I find this one of the

96:00

really exciting cutting edge

96:02

All right, and just closing thoughts

96:05

one thing we did not talk about

96:06

but if you haven't already

96:08

you know

96:09

please learn to use an LM as a

96:10

brainstorming partner and productivity enhancer

96:14

I use an LM

96:15

web-hosted

96:16

or hosted on my laptop multiple times a

96:18

day. I'm pretty sure

96:19

well, I think it makes me quite a bit more productive

96:22

using a web-hosted LM and sometimes a locally hosted one

96:29

sometimes as well for highly confidential things and

96:31

when I get annoyed at the safety mechanisms

96:33

of the cloud-hosted things

96:35

and as developers

96:37

one thing I hope you will do

96:39

is consider using an LM as a pair

96:41

programmer

96:42

this is a term I learned from Lawrence Mulroney from Google

96:45

but, rather than coding alone

96:48

many developers have a very different workflow

96:51

and, you know, it, it, it is GitHub Copilot

96:53

but it's actually much more than GitHub Copilot

96:55

In which I think you heard Isa say

96:57

if there's an error message

96:58

you sometimes paste it into GPT-4

97:00

have it explain it to you

97:03

and I think when I have code that doesn't work

97:05

I often use GPT-4 to help me to debug

97:08

and sometimes I do also use GPT-4 and

97:11

and bot, and also to generate code snippets

97:14

right? When I'm coding up Streamlit apps

97:17

say, you know

97:18

I'm often too lazy to look up the right syntax

97:21

so I have an Elm generated

97:23

plugin to my code. Sometimes it doesn't work

97:25

Sometimes I debug it myself

97:26

Sometimes I have it debug for me

97:28

But this type of workflow

97:30

I think there's something

97:31

very

97:32

fundamental going on in how all of us act as developers

97:35

and I see, you know, many of my friends

97:38

are

97:39

Coding in a way that's totally different than

97:41

any of us was

97:42

were, were coding like a year ago

97:45

And if you're, and if you haven't already

97:48

I think there's a set of pair programming

97:49

practices I see many people assigned to adopt

97:52

that are making developers much more efficient

97:54

And I think in help co-pilot

97:55

you know, it is a good start

97:57

but it's actually much more that I see

97:58

people doing than the kind of template generation that

98:00

that co-pilot tends to

98:03

and then lastly

98:04

I think there's a golden age of AI application building

98:07

So, you know, you take responsible AI seriously

98:10

but, I hope that even as these tools keep on evolving

98:13

you will keep learning about them and then go

98:17

go out there and

98:18

build stuff to help yourself and help others

98:21

And if the first few projects

98:23

you know, don't work out

98:25

so long as it didn't harm others

98:27

didn't work out, totally fine, right?

98:28

You learn from it and then use that to

98:32

to build the next bigger project

98:34

And one thing I've been amazed by

98:35

is the amount of activity

98:38

going on

98:40

often on Discord with very passionate communities

98:43

And these are often people that are

98:45

you know, non-affiliated with the biggest

98:47

best-known companies, the biggest, best-known universities

98:50

There are just

98:51

lots of people in the world out there right now

98:54

Often on Discord, hacking up incredibly creative things

98:58

and, and I find that

99:01

many of them, you know, they, they are not attending Europe

99:04

right? But they're taking online tutorials

99:06

and just coming up insanely creative things

99:09

In fact, one, one of the panelists

99:11

Mary Ringo Morris, you know, she was, she had built this

99:15

AI agents thing. We have AI systems

99:18

Actors, characters

99:19

and just chat with each other and wander around the village

99:21

right? And it was

99:22

it was cool. I have to

99:23

I don't know totally what the purpose of it is

99:25

but it's just really cool

99:27

And a few days ago

99:28

a friend was showing me on Discord

99:30

you know

99:30

this huge passionate community building all these like character agents

99:35

wandering around villages, talking to each other

99:37

And again, I don't totally know the point of this

99:39

but it is really cool

99:41

and, and, and

99:42

and I think they could evolve a point

99:44

in the future. Maybe it's a new form

99:46

of social interaction. Maybe it holds

99:47

you know, people alone in this

99:49

Maybe

99:49

someone told me they want to use this

99:51

to simulate human behavior. I think there could

99:53

be a point in the future

99:54

even though I don't personally fully appreciate it

99:56

yet

99:57

And that type of work

99:58

often done by passionate people

100:00

you know

100:01

sitting in a dorm room somewhere or in some other country

100:04

just doing this on their spare time

100:07

that passionate excitement, I

100:08

I hope all of us

100:10

can participate in that