Build a Text Classification Model with Python: From Labeled Data to Error Analysis
NLPPythontext-classificationscikit-learnhands-on-tutorial

Build a Text Classification Model with Python: From Labeled Data to Error Analysis

SSkilling.pro Editorial Team
2026-11-18
7 min read

A reusable Python workflow for text classification, from labeled data and TF-IDF baselines to evaluation, error analysis, and safe iteration.

Build a practical text classifier in Python by moving through the complete workflow: define labels, inspect and split data, establish a baseline, train a scikit-learn pipeline, evaluate predictions, and use error analysis to decide what to improve next. The checklist below is designed for a reusable NLP project rather than a one-off demo.

Overview

Text classification assigns one or more categories to a piece of text. Common examples include routing support messages, identifying sentiment, sorting news topics, flagging unwanted content, and detecting the language of a document. The underlying workflow is consistent even when the business problem changes.

For a first text classification Python tutorial, a linear model with TF-IDF features is a strong baseline. It is quick to train, relatively easy to inspect, and often performs well when the labels depend on words or short phrases. A baseline also gives you a reference point before you introduce embeddings, pretrained language models, or more complex preprocessing.

A useful project has five clearly separated parts:

  1. Data: labeled text, a stable label definition, and a record of how examples were collected.
  2. Preparation: validation, deduplication, text normalization, and train-validation-test splitting.
  3. Modeling: a reproducible feature and classifier pipeline.
  4. Evaluation: metrics that reflect the cost of different errors.
  5. Iteration: error analysis that turns incorrect predictions into specific improvements.

Keep these stages distinct. If you change labels, preprocessing, and model architecture at the same time, it becomes difficult to tell which decision affected the result.

Checklist by scenario

1. When you are starting with a new dataset

  • Write a one-sentence definition for every class. Include examples and borderline cases.
  • Check whether each row has usable text and exactly one expected label, unless the task is multilabel classification.
  • Look for duplicate messages, copied templates, empty records, and labels that differ only because of spelling or capitalization.
  • Count examples per class. Severe imbalance may require stratified splitting, class weighting, resampling, or a different metric.
  • Remove or mask information that would not be available at prediction time, such as a resolution code added after a support ticket was handled.

Use a dataframe with at least two explicit columns, such as text and label. Keep a separate data dictionary describing the source, collection period, labeling process, and permitted use. Do not place secrets, private customer information, or unnecessary personal data in a notebook or public repository.

2. When you need a reproducible baseline

A compact baseline can be built with a TF-IDF vectorizer and a linear classifier. The pipeline ensures that vocabulary learning and feature transformation happen inside each training fold rather than leaking information from the evaluation data.

from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

X_train, X_test, y_train, y_test = train_test_split(
    df["text"],
    df["label"],
    test_size=0.2,
    random_state=42,
    stratify=df["label"]
)

model = Pipeline([
    ("tfidf", TfidfVectorizer(
        lowercase=True,
        ngram_range=(1, 2),
        min_df=2,
        max_df=0.95,
        sublinear_tf=True
    )),
    ("classifier", LogisticRegression(
        max_iter=1000,
        class_weight="balanced"
    ))
])

model.fit(X_train, y_train)
predictions = model.predict(X_test)
print(classification_report(y_test, predictions))

Use class_weight="balanced" only as a starting experiment; it is not automatically the best choice. Compare it with an unweighted model and select using validation data. Set a random seed where the library supports one, record the Python and package environment, and save the exact preprocessing settings with the model.

3. When evaluation needs more than accuracy

Accuracy can conceal poor performance on a minority class. Review precision, recall, and F1 score for each label, along with the confusion matrix. Precision asks how many predicted positives were correct; recall asks how many actual positives were found. Choose the emphasis based on the consequence of each error.

  • Prioritize precision when false positives create costly manual reviews or inappropriate actions.
  • Prioritize recall when missing a relevant case is more harmful than reviewing extra cases.
  • Use macro-averaged metrics when each class deserves equal attention, even if class sizes differ.
  • Use weighted averages to summarize performance while reflecting class frequency, but do not let them replace per-class results.

For a deeper treatment of validation and error analysis, see Machine Learning Model Evaluation: Metrics, Validation Methods, and Error Analysis.

4. When the baseline is not enough

Improve the simplest part of the system first. Inspect misclassified examples, then test one change at a time: word and character n-grams, minimum document frequency, class weights, a different linear classifier, clearer labels, or more representative training examples. Character n-grams can help with spelling variation, product codes, and short text. Word n-grams may capture phrases that single words miss.

If meaning depends heavily on context, long-range relationships, or domain-specific language, compare the baseline with pretrained embeddings or a pretrained transformer. Keep the baseline in your experiment log so that added complexity has to justify itself through better validation performance, clearer error behavior, or an operational benefit.

What to double-check

Data splitting and leakage

Do not tune the model repeatedly against the test set. Split once into training and test data, use a validation split or cross-validation for decisions, and reserve the test set for a final check. If several rows come from the same user, document, conversation, or template, a random split may put near-duplicates in both sets. Group-aware or time-based splitting may provide a more realistic estimate.

Labels and ambiguity

A model cannot reliably learn inconsistent labels. Review a sample from every class, resolve overlapping definitions, and record an “uncertain” policy if humans would disagree. If the task is genuinely multilabel, do not force each example into one mutually exclusive category just to fit a simpler algorithm.

Text processing

Avoid deleting punctuation, numbers, URLs, or case distinctions without testing the effect. Those features may carry meaning in sentiment, spam detection, technical support, or product identification. Keep normalization conservative and reproducible. Never use a preprocessing step that depends on information from the full dataset when it should be fitted only on training data.

Operational and security concerns

Validate input length and encoding before inference. Treat uploaded text as untrusted data, and avoid logging raw text if it may contain personal, confidential, or credential information. Store model artifacts with access controls, pin or record dependencies, and test the loading path before deployment. A model that scores well but exposes sensitive text through logs is not a successful production machine learning workflow.

Common mistakes

  • Starting with a complex model: Without a baseline, you cannot measure whether complexity helped.
  • Using only accuracy: Imbalanced classes can make a weak classifier appear reliable.
  • Leaking vocabulary or labels: Fitting transformations before the split produces an optimistic evaluation.
  • Randomly splitting related records: Near-duplicates can make test performance look unrealistically high.
  • Changing several variables at once: Run controlled experiments and record data, settings, metrics, and observations.
  • Ignoring confidence and abstention: Some workflows should route low-confidence predictions to a person instead of forcing a class.
  • Stopping at a score: Read actual false positives and false negatives. Error patterns usually suggest the next useful action.

For a broader project structure, connect this tutorial to the End-to-End ML Project Checklist and Reusable Template. If the classifier will run continuously, plan monitoring for data drift, label changes, latency, and changes in class distribution; the model monitoring guide covers that next stage.

When to revisit

Revisit the model whenever the input process, label policy, or user behavior changes—not only when a new algorithm is released. Set a review point after the first meaningful batch of real predictions, then repeat on a schedule appropriate to the application. Seasonal planning cycles are a useful prompt for checking whether language, topics, and class frequencies have shifted.

Use this return checklist:

  1. Sample recent inputs and compare them with the training data.
  2. Check per-class precision, recall, confusion patterns, and abstention rates.
  3. Review new false positives and false negatives with a qualified labeler.
  4. Confirm that preprocessing and library behavior still match the recorded experiment.
  5. Retrain only after deciding whether the problem is data quality, label quality, thresholding, or model capacity.
  6. Re-run the untouched test evaluation and document the reason for the change.

Once the baseline is stable, package the notebook into a small, testable project with separate modules for loading data, training, evaluation, and inference. That structure makes future NLP tutorials, portfolio projects, and production changes easier to reproduce. The practical goal is not merely to train a text classifier; it is to create a workflow that explains its decisions, exposes its weaknesses, and can be improved safely.

Related Topics

#NLP#Python#text-classification#scikit-learn#hands-on-tutorial
S

Skilling.pro Editorial Team

AI and Machine Learning Education Editors

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.