Should You Make an LLM App? - Slides

Elias Dabbas

2024-06-29

How structured is the problem

Structured: Don’t use an LLM

• Computational
• Information retreival

How structured is the problem

Unstrcutured: Degrees of “unstructuredness”

• Almost structured
• Semi-structured
• Unstructured

How structured is the problem

Is the problem structured

The closer the problem is to being structured the more likely an LLM can help.

How structured is the problem

• Almost structured
  • Fix grammar
  • Rephrase text
• Semi-structured
  • Extract entities
  • Determine sentiment
• Unstructured
  • Solve unemployment
  • Create world peace

Does it have an objective solution?

• Objectively verifiable
  • Extracting countries mentioned in an article
  • Checking if an article includes a price
• Approximate
  • Does the title describe the content well?
• Highly opinionated
  • Will the reader’s life change after reading this article?

Does it have an objective solution?

How repeatable is it?

The more repeatable the more you want to use the API and/or finetuning.

How much harm can occur form being wrong?

• Medical diagnosis (harmful): Making definitive diagnoses, and actually producing prescriptions for which drugs and treatments a patient should get.

• Medical diagnosis (harmless): Sending alerts to patients and doctors, “Your behavior indiates that you might be developing condition X. It would be good to check with your doctor.” A false positive would be OK, and being correct in early detection would be of great value.

How much harm can occur form being wrong?

Should you make an LLM app?

Yes, and the ideal situation is if it is:

Example LLM apps using these criteria

• Entity extraction app
• App that evaluates content accoring to a set of criteria
• Creating content in bulk, using rich structured data
• Creating a vector store for semantic search
• Extract FAQs from customer call transcripts

Example apps

app	structured	verifiable	repeatable	harm
Entity extraction	medium	medium	high	low
Content evaluation	high	medium	high	low
Content creation	medium	high*	high	high
Creating a vector store	medium	medium	high	low
Extract FAQs from customer calls	medium	high	high	low
Transcribe from audio	high	high	high	low
Reformat text (to a blog post)	high	high	high	low

• structured: How structured is the problem and its solution?
• verifiable: Does an objective solution exist for this problem?
• repeatable: How highly repeatable is this problem?
• harm: How harmful would it be if the proposed solution is wrong?

* Assuming you are using clean structured data in your prompt

Share a suggested app, or change to the guidelines

• LinkedIn.com/in/EliasDabbas

• @EliasDabbas