Read the slides version
How structured is the problem
Structured: Don’t use an LLM
- Computational
- Information retreival
How structured is the problem
Unstrcutured: Degrees of “unstructuredness”
- Almost structured
- Semi-structured
- Unstructured
How structured is the problem
The closer the problem is to being structured the more likely an LLM can help.
How structured is the problem
- Almost structured
- Fix grammar
- Rephrase text
- Semi-structured
- Extract entities
- Determine sentiment
- Unstructured
- Solve unemployment
- Create world peace
Does it have an objective solution?
- Objectively verifiable
- Extracting countries mentioned in an article
- Checking if an article includes a price
- Approximate
- Does the title describe the content well?
- Highly opinionated
- Will the reader’s life change after reading this article?
Does it have an objective solution?
How repeatable is it?
The more repeatable the more you want to use the API and/or finetuning.
How much harm can occur form being wrong?
Medical diagnosis (harmful): Making definitive diagnoses, and actually producing prescriptions for which drugs and treatments a patient should get.
Medical diagnosis (harmless): Sending alerts to patients and doctors, “Your behavior indiates that you might be developing condition X. It would be good to check with your doctor.” A false positive would be OK, and being correct in early detection would be of great value.
How much harm can occur form being wrong?
Should you make an LLM app?
Yes, and the ideal situation is if it is:
Example LLM apps using these criteria
- Entity extraction app
- App that evaluates content accoring to a set of criteria
- Creating content in bulk, using rich structured data
- Creating a vector store for semantic search
- Extract FAQs from customer call transcripts
Example apps
| app | structured | verifiable | repeatable | harm |
|---|---|---|---|---|
| Entity extraction | medium | medium | high | low |
| Content evaluation | high | medium | high | low |
| Content creation | medium | high* | high | high |
| Creating a vector store | medium | medium | high | low |
| Extract FAQs from customer calls | medium | high | high | low |
| Transcribe from audio | high | high | high | low |
| Reformat text (to a blog post) | high | high | high | low |
- structured: How structured is the problem and its solution?
- verifiable: Does an objective solution exist for this problem?
- repeatable: How highly repeatable is this problem?
- harm: How harmful would it be if the proposed solution is wrong?
* Assuming you are using clean structured data in your prompt