pandas, NumPy, statistics, visualisation and notebooks. Open a question to read the answer.
Broadcasting is NumPy’s rule for doing element-wise arithmetic between arrays of different shapes without copying data. The smaller array is virtually stretched to match the larger one, so vectorised code replaces Python loops.
In data science this is the idiom behind feature standardisation, centring, and pairwise distance matrices.
Apply. In a customer-segmentation job I would normalise a feature matrix per column and compute distances from every customer to every centroid with X[:, None,:] - C[None,:,:], which gives an (n, k, d) array that is then reduced over the last axis. Using keepdims=True on reductions keeps shapes predictable, for example X / X.sum(axis=1, keepdims=True) for row-wise proportions.
Test. Each vectorised function gets a unit test that compares it with a slow, obviously correct loop version using np.testing.assert_allclose on small random inputs. Shape assertions run on inputs where n, k and d are all different, because square inputs hide axis mistakes.
Pandas offers label-based, position-based, and boolean selection, and mixing them up is the main source of bugs.
Assign through one loc call rather than chained df[mask]["col"] =.... From pandas 3.0, Copy-onWrite is the default, so chained assignment never modifies the original frame.
Apply. In a retail reporting pipeline I set a meaningful index (such as order_date as a DatetimeIndex) so that df.loc["2025-03"] slices a month and joins align automatically. Row filters use loc with masks, and all writes go through a single df.loc[mask, "col"] = value.
Test. Tests assert index properties explicitly: df.index.is_unique, is_monotonic_increasing before label slicing, and expected shape after each filter. A small fixture with a deliberately non-default index (for instance [10, 20, 30]) catches code that confuses labels with positions, since a default RangeIndex makes loc and iloc look identical.
groupby implements split-apply-combine: rows are split by key, a function is applied to each group, and the results are combined. merge is a SQL-style join of two DataFrames on key columns or indexes.
Apply. For a churn model I would aggregate an orders table to one row per customer (recency, frequency, total spend) with groupby().agg(), then left-merge it onto the customer table so customers without orders are kept and filled with zeros.
Test. Row-count assertions guard every join: after a m:1 left merge, len(result) == len(left) must hold, and validate="m:1" makes pandas raise MergeError if the right side has duplicate keys. I also test aggregates on a tiny hand-computed frame and check that group totals reconcile with the raw total, for example summary["total"].sum() == orders["amount"].sum().
Data cleaning makes raw data consistent and valid enough to analyse: fixing types, removing duplicates, standardising categories and units, and handling missing or impossible values. Imputation is the part that fills in missing values with estimates.
Imputation statistics must be learned on training data only, otherwise the test set leaks into the model.
Apply. For a loan-default dataset I would first profile nulls per column and by segment, fix types and units, then build the cleaning as code: a ColumnTransformer with median imputation for numeric columns and a constant for categoricals, placed inside the modelling pipeline so it is fitted per crossvalidation fold. Domain rules (age between 18 and 100, no negative balances) become explicit filters, not silent fixes.
Test. I write assertions on the cleaned output: no nulls remaining in required columns, no duplicate primary keys, values inside valid ranges, and unchanged row counts except where rows are deliberately dropped. A before-and-after comparison of each column’s distribution (mean, quantiles) shows whether imputation distorted it.
An outlier is an observation far from the bulk of the data. It may be an error, a rare but genuine event, or a sign of a different population, so detection comes before any decision to remove it.
Handling options are to correct, cap (winsorise), transform (for instance log1p), model separately, or keep; the decision should be justified by the business context.
Apply. In a payments-fraud project I would use IQR on log-transformed transaction amounts for quick screening and an IsolationForest on multivariate behaviour features to rank unusual accounts. Flagged rows go to a review table instead of being deleted, so analysts can tell fraud from data-entry errors.
Test. I check the detector on data with known anomalies: inject synthetic extreme values and confirm recall, and verify that the share of flagged rows stays close to the expected contamination rate.
Exploratory data analysis (EDA) is the structured first look at a dataset to understand its quality, structure and relationships before modelling. Its purpose is to form and challenge hypotheses, find data problems early, and decide what features and methods are appropriate.
In pandas this starts with df.info(), df.describe(), df.isna().mean(), df.duplicated().sum(), then histograms, box plots, groupby summaries, and a correlation matrix. Findings should be written down as decisions: which columns to drop, which to transform, and which relationships to test properly.
Apply. For a subscription-churn project I would begin by agreeing the target definition and observation window, then profile the data: grain of each table, null rates, cardinality, class balance, and distributions of key features split by the churn label. I would keep the work in a notebook with a fixed random seed, summarise conclusions in a short findings section, and move reusable checks into functions or a profiling report.
Test. EDA conclusions need verification: reproduce a surprising pattern on a second time period or a random split before trusting it, and check that row counts reconcile with the source system. Where a relationship looks strong, test it formally or with a simple baseline model instead of relying on a chart.
A probability distribution describes how likely each value of a random variable is. Knowing which distribution fits a variable guides the choice of test, model, and error metric.
The Central Limit Theorem explains why sample means are approximately normal for large samples even when the data is not, which underpins t-tests and confidence intervals. Heavy tails and skew break normal-based assumptions, so medians, transformations, or non-parametric methods are used instead.
Apply. In a capacity-planning project I would model hourly request counts as Poisson, check for overdispersion (variance larger than the mean), and move to a negative binomial if needed. For order values I would work on a log scale because they are log-normal. For simulation, I draw from the fitted distribution with a seeded numpy.random.Generator to estimate tail risk.
Test. Visual checks come first: histogram against the fitted density and a Q-Q plot (scipy.stats.probplot). Formal tests such as Shapiro-Wilk or Kolmogorov-Smirnov add evidence, but with large samples they reject tiny, irrelevant deviations, so effect size and the plot matter more than the p-value. Parameters estimated from the same data need a parametric bootstrap for valid KS pvalues.
Hypothesis testing is a procedure for deciding whether observed data is consistent with a default claim, the null hypothesis (H0, usually “no effect”), or whether there is enough evidence to favour the alternative (H1).
The p-value is the probability of seeing a result at least as extreme as the one observed, assuming H0 is true. It is not the probability that H0 is true, and it says nothing about the size or importance of the effect. A result is called significant when p falls below a threshold chosen in advance, the significance level alpha (commonly 0.05), which is the accepted rate of false positives (Type I errors). A Type II error is missing a real effect; power is 1 minus its probability.
Apply. For a pricing change on a checkout page I would write down H0, the metric, alpha, and the minimum effect worth detecting before looking at data. A two-proportion z-test (statsmodels.stats.proportion.proportions_ztest) suits conversion; Welch’s t-test or MannWhitney suits revenue per user. I would report the effect size and a confidence interval next to the pvalue.
Test. The test procedure itself is verified by simulation: generate data under H0 many times and confirm that roughly 5% of runs give p < 0.05, then generate data with a known effect to check power. Assumptions (independence, sample size, variance) are checked before trusting the output.
A confidence interval (CI) gives a range of plausible values for an unknown population parameter, such as a mean or conversion rate, based on a sample. A 95% CI means that if the same sampling procedure were repeated many times, about 95% of the intervals built this way would contain the true value. It does not mean there is a 95% probability that this particular interval contains it.
Width depends on variability, confidence level, and sample size: more data narrows it roughly with 1/ √n, and a higher confidence level widens it. For a mean, the usual form is estimate ± t* × standard error. For skewed or awkward statistics such as medians or ratios, the bootstrap is a practical alternative.
Apply. In a conversion-rate report I would publish the lift together with its 95% CI, so that stakeholders see that “+1.2% (CI -0.3% to +2.7%)” is inconclusive. For proportions I would use a Wilson interval (statsmodels.stats.proportion.proportion_confint(method="wilson")) rather than the naive normal approximation, and a bootstrap for medians or percentiles of latency.
Test. Coverage is checked by simulation: draw many samples from a known distribution, compute the interval each time, and confirm that about 95% contain the true parameter. This quickly exposes intervals that are too narrow for skewed data or small samples. A second test compares the analytic and bootstrap intervals on the real data; large disagreement suggests assumptions are violated.
A/B testing is a randomised controlled experiment that compares a control and a variant on a predefined metric to estimate the causal effect of a change. Good design is decided before launch.
Apply. For a new recommendation widget on an e-commerce site, I would define click-through to purchase as the primary metric, set refund rate and page latency as guardrails, and compute the sample size for the smallest lift the business cares about. Assignment would use a hash of the user ID and experiment name so that users stay in their group across visits.
Test. Before launch I run an A/A test to confirm that the pipeline reports no difference and that false positives are near alpha. During the run I check SRM with a chi-square test on assignment counts, and verify that exposure logging fires only when the user actually sees the feature.
Feature engineering is turning raw data into inputs that make a model’s job easier, using domain knowledge. Good features often improve performance more than a more complex algorithm.
Every feature must be computable at prediction time with the information available then.
Apply. For a credit-risk model I would build behavioural features such as utilisation ratio, payment delays in the last 3, 6 and 12 months, and counts of missed payments, all computed as of the application date. The transformations are packaged in a scikit-learn ColumnTransformer or reusable functions so that training and serving share one definition.
Test. I unit-test each feature function on a small frame with hand-computed expected values, including edge cases: zero denominators, nulls, and new categories. Point-in-time correctness is tested by recomputing features for a past date and confirming that no data after that date is used. Value comes from ablation: compare cross-validated metrics with and without a feature group and look at permutation importance.
Scaling matters for algorithms that depend on distances or gradient steps and on regularisation penalties: k-NN, k-means, SVM, PCA, logistic and linear regression with regularisation, and neural networks. Tree-based models (random forest, gradient boosting) are insensitive to monotonic feature scaling, so they do not need it.
Apply. In a customer-clustering project with k-means, features such as annual income (tens of thousands) and visit count (single digits) would let income dominate distances, so I standardise numeric columns inside a Pipeline with StandardScaler before the clusterer. Skewed columns get a log transform first, and heavy-outlier columns use RobustScaler.
Test. I assert on the training output that each column has mean near 0 and standard deviation near 1 (np.allclose(Xs.mean(0), 0)). A test confirms that the stored scaler.mean_ and scale_ come from the training data only and that the same object scales validation data. Comparing cross-validated scores with and without scaling shows whether it helps the chosen model.
Matplotlib is the foundational Python plotting library, offering full control over every element of a figure. Seaborn is a higher-level statistical plotting library built on top of Matplotlib; it works directly with DataFrames and produces attractive defaults with little code.
Seaborn does the statistics and grouping; Matplotlib is used for the final adjustments. Both are normally used together.
Apply. For exploratory work in a churn analysis I use Seaborn for quick views (histplot, boxplot, heatmap of correlations, pairplot) because grouping by hue takes one argument. For the final stakeholder report I build the figure with Matplotlib’s object-oriented API (fig, ax = plt.subplots()), adjust annotations and layout, and save with fig.savefig("chart.png", dpi=200, bbox_inches="tight").
Test. Plot code is checked in CI by running it on a small fixture and asserting it completes without exceptions, for instance with the Agg backend. Visual regression tools such as pytest-mpl compare generated images with approved baselines. Beyond that, I verify the numbers behind the chart, not just its appearance: aggregate in pandas and assert the plotted values.
Window functions compute a value for each row using a related set of rows (a window) without collapsing the rows as GROUP BY does. The syntax is function() OVER (PARTITION BY... ORDER BY... frame).
Windows run after WHERE and GROUP BY, so filtering on a window result needs a subquery or CTE (or QUALIFY in Snowflake, BigQuery, and Databricks).
Apply. In a subscription analytics warehouse I would use ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY event_ts DESC) to pick each customer’s latest status, LAG on payment dates to compute days between payments, and a 28-day moving average of active users for dashboards.
Test. I validate on a handful of customers whose results I can compute by hand, include ties and gaps in the test data, and reconcile totals: the final value of a running total must equal the plain SUM. A uniqueness check on the dedup output (COUNT(*) = COUNT(DISTINCT customer_id)) verifies ROW_NUMBER logic.
A Jupyter notebook is an interactive document that mixes code, output, charts, and narrative, backed by a kernel that holds variables in memory. That convenience is also the reproducibility risk: cells can run out of order, and the kernel’s hidden state may not match what the saved file shows.
Parameterised, scheduled runs use papermill, and nbconvert exports the result to HTML or PDF.
Apply. On a forecasting project the team keeps exploration notebooks in a notebooks/ folder and production logic in a versioned src/ package installed into the notebook environment. Each notebook starts with a cell that fixes seeds and prints library versions, loads data through one function that references a dataset version, and ends with conclusions. A pre-commit hook runs nbstripout so outputs do not pollute Git history.
Test. CI executes notebooks top to bottom on a small data sample, using pytest --nbmake or papermill, and fails on any exception. A second check reruns the key notebook twice and compares the final metrics; they must match within a tolerance. Logic that moved into src/ gets ordinary unit tests.
A scikit-learn Pipeline chains preprocessing steps and a final estimator into a single object with the usual fit, predict, and score interface. ColumnTransformer applies different transformations to different column groups within it.
The main benefit is correctness: when you call fit, each transformer is fitted only on the data passed in, so inside cross-validation or a grid search, preprocessing statistics come from the training fold only. That prevents leakage and ensures identical transformations at training and prediction time. A pipeline can also be saved as one artefact with joblib, and its hyperparameters are tuned together using step__param names.
Apply. For a loan-approval model I would wrap imputation, scaling, encoding, and a gradient-boosted classifier in one pipeline, tune it with GridSearchCV or RandomizedSearchCV using parameter names like model__learning_rate, and persist the fitted object with joblib.dump for the scoring service. set_output(transform="pandas") keeps column names through the steps, which helps debugging.
Test. I run a smoke test that fits the pipeline on a tiny frame and checks predictions have the right shape and no NaNs. Contract tests feed rows with missing columns, unseen categories, and new null patterns. A persistence test saves, reloads, and checks that predictions are identical. Custom transformers must pass sklearn.utils.estimator_checks.check_estimator or at least a fit-transform consistency test.
Cross-validation estimates how well a model generalises by training and evaluating it on several different splits of the data. In k-fold CV the data is divided into k folds; each fold serves once as validation while the model is trained on the other k-1, and the k scores are averaged. The mean shows expected performance and the spread shows stability.
Final reporting should still use a separate test set that was never used for tuning.
Apply. For a hospital readmission model with several admissions per patient, I would use StratifiedGroupKFold with the patient ID as the group, run the entire pipeline inside the folds, and use cross_validate with several metrics (ROC-AUC, recall, and a calibration measure). Hyperparameter search runs inside an inner loop, and a final holdout set is touched once.
Test. I verify the split itself: assert that no group appears in both train and validation indices, that class ratios per fold are close to the overall ratio, and that time-based splits never validate on earlier data than they train on. Fixed seeds make runs repeatable, and repeated CV (RepeatedStratifiedKFold) shows how much the score depends on the particular split.
Classification metrics summarise how well predicted classes or probabilities match the true labels. Most derive from the confusion matrix (TP, FP, TN, FN).
Accuracy misleads on imbalanced data: a model that always predicts “no fraud” is 99.8% accurate at a 0.2% fraud rate and useless. Precision and recall trade off through the decision threshold, which should be picked from business costs rather than defaulting to 0.5. Log loss and the Brier score evaluate whether predicted probabilities are well calibrated.
Apply. For a fraud model I would agree with the business on the cost of a missed fraud against the cost of a manual review, then choose precision at a fixed review capacity (for example top 1% of scores) and PR-AUC as the headline metrics, not accuracy. The operating threshold is picked from the precision-recall curve and written to configuration.
Test. Metric code is tested on tiny hand-built label arrays where precision and recall are easy to compute, including degenerate cases such as no predicted positives. Evaluation always uses data the model has not seen, ideally a time-based holdout, and reports confidence intervals by bootstrapping so that small differences between models are not over-read.
Regression diagnostics check whether a fitted model’s assumptions hold and whether individual points distort it. They matter most for inference (coefficients, p-values, intervals) but also reveal weaknesses in predictive models.
Apply. In a pricing-elasticity model for a retailer, I would fit OLS with statsmodels, review the summary, plot residuals against fitted values and each predictor, and compute VIF for the pricerelated features. Any decision based on coefficients (for example “a 1% price rise cuts demand by 1.4%”) is only reported after diagnostics pass or are addressed.
Test. The diagnostic suite is scripted so it reruns on each refit: Breusch-Pagan, Durbin-Watson, VIF table, top Cook’s distance rows, and holdout RMSE and MAE. I test on simulated data with a known linear relationship and known violations, such as an injected heteroscedastic noise, to confirm each check flags what it should.
Time series forecasting predicts future values of a variable observed at regular time intervals, using its own history and optionally external drivers such as holidays or prices.
Evaluation must respect time order, using a rolling-origin backtest rather than random splits, and metrics such as MAE, RMSE, or MASE; MAPE fails when actual values are near zero.
rolling means, price, promotion, and holiday flags, and forecast a 28-day horizon. Forecasts are produced on a schedule and written to a table that planners consume.
Test. A rolling-origin backtest with several cutoffs mirrors production: train on data up to the cutoff, forecast the horizon, and score against what happened. I compare each model against seasonal naive, check bias (mean error) as well as MAE, and inspect accuracy by product segment and horizon step. Feature code is tested to ensure that lags never use data after the forecast origin.
Data leakage is when information that would not be available at prediction time influences model training or evaluation, producing performance estimates that are far better than real-world results.
Time makes it worse: using future data to predict the past, or random splits of time-ordered data, leaks the future.
Apply. All preprocessing sits inside a scikit-learn pipeline, and splits are by time and grouped by customer so that one person’s rows never straddle the boundary.
must fall to chance, or something leaks. Duplicate detection checks that no key appears in both train and test.
An imbalanced dataset has a target where one class is far rarer than the others, as in fraud (0.2% positive), churn, or equipment failure. Standard training optimises overall error, so a model can ignore the minority class and still look accurate.
training folds only, using an imblearn Pipeline so that cross-validation stays honest.
Apply. In a predictive-maintenance project with 0.5% failure rate, I would first try class weights and threshold tuning on a gradient-boosting model, since they change nothing in the data. Only if recall stays inadequate would I add resampling, inside the training folds. The final threshold is selected from the precision-recall curve to match the number of inspections the maintenance team can perform.
Test. Evaluation uses stratified or time-based splits with the untouched natural class ratio in the test set, so metrics reflect production. I report PR-AUC, recall at a fixed precision, and the confusion matrix with confidence intervals. A dummy classifier that always predicts the majority class sets the floor for comparison.
Data quality validation is the automated checking of datasets against explicit expectations before they are used for analysis or modelling. It catches problems at the point of ingestion or transformation, not in a stakeholder’s dashboard.
Apply. In a daily sales pipeline I would put validation after ingestion and again before publishing to the reporting layer. Hard failures (duplicate order IDs, negative amounts, missing keys) stop the load and alert the data owner; softer anomalies (row count 30% below the 7-day average) raise warnings. Rules live in version control next to the transformation code, as dbt tests or Great Expectations suites.
Test. The checks themselves are tested with fixture datasets that contain known violations, confirming each rule fails when it should and passes on clean data. Thresholds for volume or distribution checks are backtested on historical data to see how many false alarms they would have produced. Pipeline runs record check results so that trends are visible.
Data versioning means tracking immutable snapshots of datasets so that any model, report, or experiment can be tied to the exact data it used. Git is poor at large files, so data is stored elsewhere and only a small pointer is versioned.
Apply. Small reference files in the repo are tracked with DVC. A model card then states exactly which data snapshot it was trained on.
Test. I test reproducibility by checking out an old commit, restoring its data version, retraining with fixed seeds, and comparing metrics with the logged ones. Another check confirms that pointer files and remote storage agree (dvc status -c), and that a pipeline run fails when the data hash does not match the one recorded.
Experiment tracking is the systematic recording of everything about each model training run so that results can be compared, reproduced, and audited. Without it, teams cannot answer which settings produced the model currently in production.
A tracking system also provides a UI or API to sort and filter runs, tags to mark the purpose of a run, and links to a model registry for promotion. Common tools are MLflow, Weights & Biases, Neptune, Comet, and DVC experiments. The key habit is to log automatically from the training script instead of writing numbers in a spreadsheet, so that every run, including failed or abandoned ones, is stored.
Apply. On a team of five building a demand model, each training script wraps the run in a tracker context, logs the parameters, the data version, the Git SHA, cross-validated metrics, and the fitted pipeline as an artifact, and tags runs with the ticket number. A shared tracking server with a naming convention for experiments lets anyone compare runs and promote the best one to the registry.
Test. A CI job runs a short training on a sample and asserts that the run was created with the required fields: params, metrics, tags, and model artifact. Reproducibility is spot-checked by rerunning a logged run from its recorded commit and data version and confirming the metric matches within tolerance.
Add your resume and the job description in CrackItX's Settings. When the interviewer asks a Data Science Technologies question, Claude suggests an answer using your real projects and the role's requirements, in real time.
See pricing →