# Generating text word by word
**URL:**
**Author:**  [@Eichhof](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FEichhof)%5C
**Post date:** [January 10, 2023, 5:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgenerating-text-word-by-word%2F29312%2F1 "2023-01-10T17:06:29Z")
Hello
I’m currently using GPT-J for generating text as shown below. This works well but it takes up to 5 seconds to generate the 100 tokens.
Is it possible to do the generation word by word or sentence by sentence? Similar to what ChatGPT is doing (ChatGPT seems to produce the output word by word).
```auto
import transformers
from transformers import GPTJForCausalLM
config = transformers.GPTJConfig.from_pretrained("EleutherAI/gpt-j-6B")
tokenizer = transformers.AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B", pad_token='<|endoftext|>', eos_token='<|endoftext|>', truncation_side='left')
model = GPTJForCausalLM.from_pretrained(
"EleutherAI/gpt-j-6B",
revision="float16",
torch_dtype=torch.float16,
low_cpu_mem_usage=True,
use_cache=True,
gradient_checkpointing=True,
)
model.to("cuda")
prompt = tokenizer("This is a test sentence, which should be completed", return_tensors='pt', truncation=True, max_length=2000)
prompt = {key: value.to("cuda") for key, value in prompt.items()}
out = model.generate(**prompt,
n=1,
min_length=16,
max_new_tokens=100,
do_sample=True,
top_k=15,
top_p=0.9,
batch_size=1,
temperature=1,
no_repeat_ngram_size=4,
clean_up_tokenization_spaces=True,
use_cache=True,
pad_token_id=tokenizer.eos_token_id,
)
res = tokenizer.decode(out[0])
```
---
**Author:**  [@Eichhof](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FEichhof)%5C
**Post date:** [January 13, 2023, 2:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgenerating-text-word-by-word%2F29312%2F2 "2023-01-13T14:36:57Z")
Does somebody know a solution?
---
**Author:**  [@enochlev](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fenochlev)%5C
**Post date:** [December 19, 2023, 6:29pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgenerating-text-word-by-word%2F29312%2F3 "2023-12-19T18:29:11Z")
set stream=True
```auto
out = model.generate(**prompt,
n=1,
min_length=16,
max_new_tokens=100,
do_sample=True,
top_k=15,
top_p=0.9,
batch_size=1,
temperature=1,
no_repeat_ngram_size=4,
clean_up_tokenization_spaces=True,
use_cache=True,
pad_token_id=tokenizer.eos_token_id,
stream=True
)
for i in out:
#some code to get streamed output
```
There is hundreds of example out there to do this