Should You Make an LLM App?

An attempt to create simple guidelines to help decide which are the ideal situations to create an LLM app, using a programmatic approach (API and/or finetuning).
Author

Elias Dabbas

Published

June 29, 2024

Read the slides version

How structured is the problem

Structured: Don’t use an LLM

  • Computational
  • Information retreival

How structured is the problem

Unstrcutured: Degrees of “unstructuredness”

  • Almost structured
  • Semi-structured
  • Unstructured

How structured is the problem

Is the problem structured

The closer the problem is to being structured the more likely an LLM can help.

How structured is the problem

  • Almost structured
    • Fix grammar
    • Rephrase text
  • Semi-structured
    • Extract entities
    • Determine sentiment
  • Unstructured
    • Solve unemployment
    • Create world peace

Does it have an objective solution?

  • Objectively verifiable
    • Extracting countries mentioned in an article
    • Checking if an article includes a price
  • Approximate
    • Does the title describe the content well?
  • Highly opinionated
    • Will the reader’s life change after reading this article?

Does it have an objective solution?

How repeatable is it?

The more repeatable the more you want to use the API and/or finetuning.

How much harm can occur form being wrong?

  • Medical diagnosis (harmful): Making definitive diagnoses, and actually producing prescriptions for which drugs and treatments a patient should get.

  • Medical diagnosis (harmless): Sending alerts to patients and doctors, “Your behavior indiates that you might be developing condition X. It would be good to check with your doctor.” A false positive would be OK, and being correct in early detection would be of great value.

How much harm can occur form being wrong?

Should you make an LLM app?

Yes, and the ideal situation is if it is:

Example LLM apps using these criteria

Example apps

app structured verifiable repeatable harm
Entity extraction medium medium high low
Content evaluation high medium high low
Content creation medium high* high high
Creating a vector store medium medium high low
Extract FAQs from customer calls medium high high low
Transcribe from audio high high high low
Reformat text (to a blog post) high high high low
  • structured: How structured is the problem and its solution?
  • verifiable: Does an objective solution exist for this problem?
  • repeatable: How highly repeatable is this problem?
  • harm: How harmful would it be if the proposed solution is wrong?

* Assuming you are using clean structured data in your prompt

Share a suggested app, or change to the guidelines