User Guide: Data Cleaning Pattern Checker

This web app converts the original Google Colab data-cleaning script into a server-based tool. It accepts a CSV file, converts values to numeric data when possible, drops rows with no numeric values, checks suspicious response patterns, and exports a new CSV file with a Warning column.

1. Recommended input format

Prepare your file as a CSV table. The first row should usually contain column names.

Example:

StudentID,Q1,Q2,Q3,Q4,Q5,Q6
S001,1,2,3,4,5,6
S002,4,4,4,4,4,4

Non-numeric cells are treated as missing values. This is useful when your file contains ID columns, names, or blank cells. The warning calculation uses numeric cells only.

2. What the app checks

WarningMeaningDefault rule
OutlierThe whole row is statistically unusual compared with other rows.Mahalanobis distance > 24.32
SD < 0.5The row has very little variation.Sample SD below 0.5
ConstantA long sequence has exactly the same answer.36 identical values in a row
IncreasingA row contains a strictly increasing response pattern.6 increasing values in a row
DecreasingA row contains a strictly decreasing response pattern.6 decreasing values in a row

3. How to use the app

  1. Open index.php in your browser.
  2. Upload your CSV file.
  3. Choose the delimiter. Auto detect works for most comma, semicolon, tab, and pipe-separated files.
  4. Adjust thresholds if needed.
  5. Click Clean Data.
  6. Download the generated CSV file.

4. How to interpret the output

The output file contains a new column called Warning. Rows marked OK did not trigger the selected rules. Rows with warnings are not automatically wrong; they should be reviewed manually.

5. Mahalanobis distance notes

Mahalanobis distance requires a covariance matrix that can be inverted. If your dataset has too few complete cases, duplicated columns, or highly correlated columns, the app will turn this check off and continue with the other warning rules.

6. Important limitations