Repository navigation
📓 Text Generation docs rework #24575
Description
What is this?
This is an issue to discuss and track the rework of the docs for text generation. Comments and feedback are appreciated, as always 🤗
Current issues
- Our main reference for text generation is not in the docs and is quite outdated
- The docs regarding text generation are scattered, and it is not simple to navigate between them -- the reader has to know where to look for them
- We lack examples beyond the simplest forms of text generation
- We have undocumented advanced use cases, such as setting a custom stopping criteria
- We are not clear about what the user can't do
Proposed plan
EDIT:
- incorporated feedback up to this comment (including)
- Also includes this comment
I'd like to split the plan into three parts:
- Designing a simpler entry point to text generation, from which all related documentation is discoverable
- Upgrading the developer guides to cover the full potential of text generation
- Make our code more self-documenting and other code changes
1. Designing a simpler entry point for text generation docs
Tackles issues 1 and 2.
This part is further divided into two actions:
- The (blog post)[https://huggingface.co/blog/how-to-generate] is still a solid reference for the background in text generation, but it holds old examples (
tensorflow!) and focuses a bit too much ontop_p/top_k. Let's retouch it. - Create a short tutorial to serve as an entry point to the multiple forms of text generation. Like the other tutorials, it contains references to related docs throughout the text (let's see if it is enough to handle discoverability -- we can create a stand-alone related docs section in the future if needed). It would also cover a few basics like "use left-padding when doing batched generation with decoder-only models" and "double-check your generate kwargs".
Related docs:
- Tasks
- Related developer guides
- API reference
- Outside
transformers(e.g.optimum,text-generation-inference, LLM leaderboard, non-HF libs likeautogptq?)
2. Upgrading the developer guides
Tackles issues 3 and 4.
We currently have one developer guide, which writes about the API and a few basic ways to manipulate text generation. I propose we improve the existing one and add 2 new guides, preferably with examples that cover more modalities and use cases:
- 1. Improve the existing guide -- Add a section about the impact of logits processors, and another on how stopping conditions operate.
- 2. "Prompting" -- Some basic "do and don'ts" regarding prompting and how different types of models respond differently to it (encoder-decoder vs decoder, instruction-tuned vs base), the importance of prompting on chat applications
- 3. Using LLMs, with a focus on the 1st L (large) -- write about variable types, quantization, device mapping, advanced architectures (alibi, rope, MQA/GQA), flash attention
- 4. Advanced examples (name?) -- Concrete use cases that make use of many features at once, to serve as inspiration: how to control between extractive and abstraction summarization, retrival-augmented generation, and other modality-specific examples
3. Self-documenting code and other code changes
Tackles issues 3 and 5.
-
Let's be honest -- the best user experience is when no docs are needed at all. We can improve our game here, by performing parameterization validation. Currently, our validation step is very superficial, and users are allowed to do things like passing
temperaturewithdo_sample=False, ultimately resulting in GH issues. I'd suggest performing a hard validation and throwing informative exceptions, pointing to the redesigned docs 🤗 -
In parallel, our logits processors and stopping condition classes are missing docstring examples on how to use them. This should make our API reference much more robust.
Activity
SoyGema commentedon Jul 2, 2023
Hello there @gante ! 👋
First and foremost, thanks for opening a discussion about this .
Would like to glimpse a couple of vector thoughts about point 1, as point 2 and 3 seem consistent and structured enough
1. Designing a "home page" for text generation docs
As a user , the underlying pattern that I´m glimpsing from the Tutorials section make reference to how to use the library from the engineering perspective, in abstract terms ( preprocess data, Share your model, fine-tune etc ) . The tutorials seem like a library approach for a ‘HOW-TO ‘ do things. In fact, in Tutorials section, several examples about vision, audio and language are displayed.
I would think about putting a Text Generation section directly in Task guides , inside Natural Language processing, at the top , as it is related to a challenge to solve ( Text classification, Token classification ) . This doesn’t entail that one of the main ‘HOW-TOs’ related to text generation would be included inside Tutorials as a section. From what I’m taking for the guide, there is an insightful section of Search and Sampling, that could be added to the Tutorials, and a more detailed clarification added in Tasks and Developer guides.
The thing is that following this schema, at first sight ( abstracting main challenge from guide in Tutorials and add a robust example or some "home-page" references in Tasks with link to developer guides ) seems more coherent with your current structure.
On the other hand, and tangential, why not adding a LLMs leaderboard link somewhere (maybe in point 2) so the users can be mindful about the state of the art of the models in terms of perf for Text generation tasks ?
Hope I explained myself clearly enough 🧭
Thanks again for the open discussion! And for making the library! 👐
patrickvonplaten commentedon Jul 3, 2023
Big +1 on the 3.) point. I think self-explanatory code with better doc string examples and arg/kwarg checking would take us a long way!
RE: 1.) Yes this makes sense, but I think a single concise page is probably better than a homepage that links to everything we already have as this might be too difficult to keep up to date and might also be too complex. A single concise page under "Tutorials" which is a strong iteration on the blog post "how to generate" (which is still one of our most read blog posts) could be a good starting point. The blog post does a very good job at explaining how LLMs fundamentally work. It is however not up to date anymore and also puts too much focus on things like top_k and top_p. So a strong incremental improvement (more tailored to new LLMs) could serve as the main point of introduction to text-generation and be put under "Tutorials".
RE: 2.) Yes I think these could all go into Developer guides and be a nice iterative improvement to "Customize Text generation strategy"
Vaibhavs10 commentedon Jul 3, 2023
Hi @gante - I love the plan.
Here are a couple of quick suggestions:
togglebased snippets, how to generate withpipelineand how to generate withprocessor + generate-> this will help cater to our rough user base.yet, link it to a gh repo or gist. These are examples like generate withggml,gptqintegration and so on.Cheers!
NielsRogge commentedon Jul 3, 2023
Adding here that it would be nice to include a section on batched generation (I made a PR for GPT-2 here). This is not that intuitive for people as you need to pad from the left, set the padding token appropriately in the tokenizer, etc.
gante commentedon Jul 3, 2023
Thank you for the feedback folks 🙌
I've incorporated your feedback in the plan, which got edited. Main differences:
stevhliu commentedon Jul 6, 2023
Thanks for taking the time to outline this detailed rework! Big +1 for the additional developer guides and tutorial. ❤️
I would consider leaving the blog post as is also to reduce long-term maintenance and instead keep all the relevant text generation content in the docs. In general, I feel like a blog post is more appropriate for "timely" content like announcements/news or explaining why something was designed the way it was. Content that needs to be maintained and updated is better in the docs I think. As you mentioned, the how-to-generate blog post still contains super useful background info about text generation so I think we should definitely find a way to preserve that info. My suggestions would be to:
MKhalusova commentedon Sep 13, 2023
As discussed with @gante, I'll start working on an LLM prompting guide (part 2.2 ("Prompting" )).
github-actions commentedon Jan 1, 2024
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.