Final Project - Classic Machine Learning - Machine Learning in the Geosciences

Objective¶

This assignment guides you through classic machine learning tasks for your AI-ready dataset. You will compare model families, evaluate performance honestly, and assess computational cost. The goal is to practice selecting and optimizing a model for your project, preparing you for deeper model development.

Structure of the Assignment¶

Research Project Relevance
- Task: Write a brief essay that discusses the relevance of machine learning to your project and outlines your approach.
- Requirements:
  - A concise, ½-page essay (in Research_Relevance.md) that includes:
    - The problem’s connection to classification or regression.
    - Justification of whether the work will use a supervised, unsupervised, self-supervised, or semi-supervised approach.
    - Expected outcomes and potential impact of applying ML to your dataset.
    - An AI-use disclosure: state which parts of the code and text were produced with AI assistance, and how you verified them.
Clustering Analysis for Classification Projects
- Task: If your project involves classification, perform a clustering analysis on the features of your data, especially after dimensionality reduction.
- Requirements:
  - Apply clustering algorithms (e.g., K-means) to understand feature clusters or patterns.
  - Conduct best practices for clustering analysis in a notebook (notebooks/Clustering_Analysis.ipynb):
    - Use silhouette analysis and elbow curve for optimal K identification.
    - Evaluate clusters with homogeneity or the Fowlkes-Mallows Index.
    - Test robustness using repeated runs of K-means with random seeds.
  - Visualize the clusters and discuss how this analysis informs your approach to classification.
Model Comparison and Hyperparameter Tuning
- Task: Compare model families systematically and tune the promising ones. (Earlier editions required pycaret; that tool is retired.)
- Requirements:
  - In a notebook (notebooks/AutoML_Hyperparameter_Tuning.ipynb):
    - Establish a trivial baseline first (majority class or mean/climatology prediction). Every model is judged against it.
    - Compare at least four classifier or regressor families, including a gradient-boosting model (scikit-learn HistGradientBoosting* or LightGBM).
    - Tune the best candidates with an Optuna study or a grid search wrapped in cross-validation.
    - Describe the algorithms evaluated and the hyperparameters optimized.
    - Identify the most promising models based on performance, interpretability, and computational cost.
Training Engineering and Model Assessment
- Task: Perform a thorough analysis of training strategies and model assessment.
- Requirements:
  - In a notebook (notebooks/Model_Training_Assessment.ipynb), demonstrate:
    - A clean train-validation-test protocol: all model selection happens on training/validation data; the test set is used once.
    - A cross-validation scheme chosen for your data’s correlation structure, with justification: if your samples are correlated in time or space (most geoscience data are), use TimeSeriesSplit, blocked, or GroupKFold splits and explain your choice; if you use random K-fold, justify why leakage is not a concern.
    - Performance generality by testing on diverse subsets of the data.
    - Bootstrapping and bagging techniques across different model architectures and data variations.
    - Visualize model performance (e.g., learning curves, error metrics) across training rounds.
    - A discussion of results, including how these methods affect model generality and performance.
  - All reported metrics must come from your own held-out validation and test sets, never from training data. The instructor holds hidden test data for the course datasets and will spot-check claimed performance; a large gap between claimed and spot-checked scores will cost credit.
Computational Time Analysis
- Task: Analyze the computational time needed for model training and deployment.
- Requirements:
  - In a notebook (notebooks/Computational_Time_Analysis.ipynb), include:
    - Metrics on training time for each model architecture, detailing how various parameters affect speed.
    - An exploration of time vs. accuracy trade-offs for different configurations.
    - An assessment of expected time requirements for model deployment in real-world scenarios.
  - Summarize findings in a short conclusion, focusing on any computational challenges or optimizations relevant to your model.

Deliverables¶

A GitHub repository with the following structure:

- data/
  - ai_ready/
- notebooks/
  - Clustering_Analysis.ipynb
  - AutoML_Hyperparameter_Tuning.ipynb
  - Model_Training_Assessment.ipynb
  - Computational_Time_Analysis.ipynb
- Research_Relevance.md
- README.md

Update the README.md file from the previous assignment with an additional section about CML, key findings, and clear instructions to reproduce the analyses by naming which notebook to run. Please add the version of the repository in a text file as the assignment.

Grading Criteria¶

Relevance Essay (10%): Clarity of problem framing, appropriate ML approach, impact explanation, and AI-use disclosure.
Clustering Analysis (20%) (only if classification project): Depth of clustering analysis, best practices in evaluation, and clarity of insights.
Model Comparison and Hyperparameter Tuning (20%): Baseline established, range of model families tested (including gradient boosting), quality of the Optuna or grid search, and clarity in reporting.
Training Engineering and Model Assessment (30%): Robustness of training strategy, a correlation-aware and justified cross-validation choice, use of bootstrapping, and clear results discussion.
Computational Time Analysis (10%): Thorough analysis of training and deployment times, insights on computational efficiency.
Documentation and Code Clarity (10%): Clear explanations, code readability, and adherence to best practices.

This assignment applies classic machine learning methods to real geoscientific data, deepening your practical understanding of model selection, tuning, and honest assessment.