Customer support tooling

Support ticket routing

I wanted a small project that went beyond a notebook. This one takes a customer message through an API, saves the prediction, and leaves a place for a reviewer to correct it when the model is unsure.

GitHub repositoryPython · FastAPI · Streamlit · scikit-learn · SQLite · Docker

The basic loop

What happens after a message comes in

The dashboard sends the message to FastAPI. The API chooses a route, calculates a confidence score, writes the result to SQLite, and returns the top alternatives. When the score is below 0.60, the row stays in the review queue instead of being treated as settled.

Local support ticket review page showing a customer message, prediction, and review queue — This is the local setup: a simple review page on top of the same API used by the prediction request.

Model choice

I kept the first model deliberately boring

The serving model is a word-and-character TF-IDF pipeline with logistic regression. Word features handle common phrases; character features help with spelling differences and short messages. A few explicit rules cover requests that are easy to recognize, such as cancelling an order.

Held-out split: 31,496 training rows and 8,454 test rows. Accuracy was 0.9838 and Macro-F1 was 0.9780 on this public-data benchmark. That score should not be read as expected accuracy on a real support queue.

The shared router has ten coarse routes, including account, card, order delivery, payment, refund, transfer, and an `other` bucket. The two source datasets use different original labels, so I mapped them into this smaller set rather than pretending the taxonomies were identical.

Review queue

The uncertain cases are still part of the system

A prediction is not the same thing as a confirmed label. The API returns the route, confidence, alternatives, and review status. A reviewer can submit a corrected route through the dashboard or POST /review/:prediction_id.

POST /predict
{
  "text": "I need to cancel my order"
}

→ { "route": "refund", "confidence": 0.5124,
    "needs_review": true, "review_status": "pending" }

There is also a 40-row challenge set that is kept separate from training and rule changes. It is small, but it gives me a fixed place to check whether a new experiment actually helps.

Experiments

The bigger experiment did not help this dataset

I tried frozen BERT embeddings and a confidence-calibration step. Both stayed in the repository as comparisons; neither replaced the smaller serving model.

Experiment	Challenge Macro-F1	Decision
Word + character TF-IDF	0.7922	kept
Frozen BERT embeddings	0.7277	not kept
Confidence calibration	0.7642	not kept

The main reason is simple: the alternatives were worse on the frozen challenge set. Keeping that result visible is more useful than presenting only the model that made it into the API.

Before real use

What I would change before connecting real tickets

Replace the public and synthetic data with a reviewed sample from the actual support queue.
Move from local SQLite to a managed database if more than one API instance is needed.
Measure correction rates by route instead of relying on one overall score.
Add authentication and audit permissions before exposing the review endpoint.

Read the code