Modeling

.gitignore

@@ -5,3 +5,4 @@ venv/
 *.db
 *.idea
 *.ruff_cache
+.DS_Store

0_domain_study/README.md

@@ -1 +1,106 @@
-# Domain Research
+# Domain Study: Peer-to-Peer (P2P) Lending Risk
+
+This document outlines our foundational research into the domain of Peer-to-Peer
+(P2P) lending and the critical challenge of modeling loan default risk. It also
+provides an overview of the contents of this folder.
+
+## Folder Contents
+
+This `0_domain_study` folder is organized to document our research process and
+findings.
+
+- **`README.md` (This File):** Provides a high-level summary of our research
+  domain, the evolution of our research question, and the overall folder
+  structure.
+- **`guide.md`:** Offers a detailed orientation to the folder's contents,
+  explaining the purpose of each file and sub-directory.
+- **`research_questions/`:** Contains the history of our research questions.
+  - `old/`: Documents our initial research on "Buy Now, Pay Later" (BNPL).
+  - `new/`: Details our current, refined research questions on P2P lending.
+- **`sources/`:** Lists the data and literature that support our research.
+  - `old/`: Contains sources related to our initial BNPL research.
+  - `new/`: Contains sources for our current P2P lending research.
+
+## Evolution of Our Research
+
+Our initial research focused on the risks associated with "Buy Now, Pay Later"
+(BNPL) services. However, due to the limited availability of public datasets
+required to rigorously investigate our initial questions, we pivoted our
+research.
+
+Our new focus is on **Peer-to-Peer (P2P) lending**, a domain with rich, publicly
+available data that allows for robust modeling and analysis of credit risk.
+
+## The P2P Lending Landscape
+
+Peer-to-Peer (P2P) lending platforms have emerged as a significant alternative
+to traditional banking, connecting individual borrowers with investors directly.
+These platforms offer greater access to credit for borrowers and potentially
+higher returns for investors. However, the decentralized nature of P2P lending
+introduces unique challenges in assessing borrower creditworthiness and
+predicting the likelihood of default, which is crucial for sustainable growth
+and investor confidence.
+
+## Problem Statement
+
+While P2P platforms provide extensive data on loans and borrowers, accurately
+predicting which loans will default remains a complex problem. Investors face
+the risk of capital loss due to insufficient or ineffective risk assessment
+models. Therefore, there is a critical need to develop robust predictive models
+that can identify the key drivers of default risk, enabling investors to make
+more informed decisions and platforms to refine their underwriting standards.
+
+## Research Question
+
+_What are the key borrower and loan characteristics that best predict default
+risk in peer-to-peer (P2P) lending platforms in the United States?_
+
+### Secondary Questions
+
+1. Which machine learning approaches most accurately model default risk in P2P
+   lending data?
+2. How do features such as credit grade, interest rate, debt-to-income ratio,
+   income, loan term, and loan purpose contribute to risk prediction?
+3. How do default risk patterns change across time, different regions, or
+   borrower segments?
+4. In what ways can advanced risk modeling support investor decisions and
+   improve P2P platform underwriting?
+
+## Key Focus Areas
+
+Our research is structured around three core areas:
+
+- **Technical Focus**: We will concentrate on advanced data cleaning, feature
+  engineering, and benchmarking machine learning models (Logistic Regression,
+  Random Forest, XGBoost). We will also use explainability techniques like SHAP
+  to interpret model predictions.
+- **Business Focus**: The insights from our models will be framed to improve
+  loan pricing, enhance underwriting processes, and develop actionable risk
+  management tools for investors.
+- **User Focus**: We aim to identify risk signals across different borrower
+  types and investigate factors related to financial health, while ensuring
+  fairness in risk assessment.
+
+## Methodology and Dataset
+
+Our study will utilize a quantitative approach, applying machine learning
+techniques to a large-scale dataset.
+
+### Dataset
+
+The primary dataset for this research is the **Lending Club Loan Data** from
+Kaggle, which contains comprehensive information on loans issued in the U.S.
+
+### Modeling Approach
+
+We will employ a systems thinking lens to understand the interconnected factors
+influencing default risk. Our modeling process will involve:
+
+- **Data Preparation**: Cleaning and preparing the Lending Club dataset.
+- **Model Training**: Building and training predictive models.
+- **Evaluation**: Assessing model performance using metrics like AUC-ROC.
+- **Interpretation**: Using SHAP to understand the key features driving
+  predictions.
+
+This structured approach will ensure our findings are both statistically robust
+and practically applicable for investors and P2P platforms.

0_domain_study/guide.md

@@ -1,15 +1,64 @@
 # Domain Study: Guide
 
-To do meaningful research in a domain, you need to learn what others already do
-and don't understand in this area. Use this folder to organize your group's
-understanding of your research domain including: your own summaries, helpful
-PDFs, links you found helpful, ...
+This `guide.md` file provides an orientation to the contents and structure of
+the `0_domain_study` folder. This folder contains our group’s domain-level
+understanding of the research question we are pursuing—centered around
+financial inclusion, credit systems, and BNPL (Buy Now Pay Later).
 
-This folder is different from `/notes` because it contains _only_ information
-about your research domain. When deciding what goes here, ask yourself this
-question: _Would someone need to know this to understand our research?_
+Our research process is guided by systems thinking, where we understand each
+topic not in isolation but as an interconnected part of the broader financial
+and socio-economic system. This has shaped our analysis of how data, credit
+scoring models, regulatory structures, user behaviors, and technological
+interventions all influence financial outcomes.
 
-## README.md
+## Folder Structure and Contents
 
-Use this folder's README to document all the notes and resources in this folder.
-Someone shouldn't need to read through _everything_ to find what they need.
+### 1. `research_questions/`
+
+This folder documents the evolution of our central inquiry. It contains:
+
+- `initial_questions.md`: A record of early-stage questions and themes we
+  explored through our literature review.
+
+- `final_questions.md`: The final, actionable research questions that emerged
+  from both our literature review and group discussions. These are the
+  questions we are now pursuing using data science methods.
+
+### 2. `sources/`
+
+This folder contains the foundational materials we have studied, including:
+
+- `sources.md`: A list of key academic papers, datasets, and articles that
+  informed our domain understanding. Each source includes a brief summary
+  and notes on how it influenced our thinking.
+
+### 3. `README.md`
+
+This file provides an overview of the entire `0_domain_study` folder. It
+explains the research domain, links to relevant subfolders (questions,
+sources), and gives an at-a-glance summary of our direction.
+
+### 4. `guide.md` (this file)
+
+You’re reading it! This file explains how to navigate the domain study
+materials and how each file contributes to understanding our research domain.
+
+## Our Approach
+
+We apply systems thinking to structure our understanding of BNPL and its
+intersection with financial inclusion. We focus on identifying:
+
+- **Structural components**: Actors in the system (users, lenders, data
+  brokers, regulators)
+
+- **Behavioral patterns**: How usage behaviors like BNPL adoption relate to
+  credit health
+
+- **Purpose dynamics**: Who benefits from the system as currently designed?
+
+Our research is rooted in real-world patterns but also reflects a personal
+lens on financial vulnerability, particularly among young or underbanked
+populations.
+
+The deliverables from this folder will support Milestone 1 and guide the next
+steps of our modeling, data acquisition, and analysis phases.

0_domain_study/research_questions/new/final_questions.md

@@ -0,0 +1,57 @@
+# Problem Overview and Final Research Framework
+
+## Primary Research Question
+
+What are the key borrower and loan characteristics that best predict default
+risk in peer-to-peer (P2P) lending platforms in the United States?
+
+## Refined Secondary Questions
+
+1. Which machine learning approaches most accurately model default risk in P2P
+   lending data?
+2. How do features such as credit grade, interest rate, debt-to-income ratio,
+   income, loan term, and loan purpose contribute to risk prediction?
+3. How do default risk patterns change across time, different regions, or
+   borrower segments?
+4. In what ways can advanced risk modeling support investor decisions and
+   improve P2P platform underwriting?
+
+## Research Focus Areas
+
+### Technical Focus
+
+- Advanced data cleaning and feature selection tailored to the Lending Club
+  dataset
+- Model benchmarking using logistic regression, random forest, and
+  gradient-boosted trees
+- Application of SHAP/LIME for model explainability and feature impact analysis
+- Temporal and geographic slicing for advanced pattern discovery
+
+### Business Focus
+
+- Leveraging model insights to improve loan pricing, selection, and underwriting
+  processes
+- Developing actionable investor tools for portfolio risk management
+
+### User Focus
+
+- Identifying risk signals across borrower types
+- Investigating financial health factors and demographic variance in risk
+  profiles
+- Ensuring fairness in risk assessment across borrower populations
+
+## Anticipated Outcomes
+
+- Empirically validated predictors of P2P loan default risk
+- Interpretable machine learning models with operational use-cases for investors
+  and platforms
+- Recommendations for data-driven, transparent credit risk evaluation
+- Guidelines for ongoing research and practical application
+
+## Constraints and Ethics
+
+- Data limitations, such as missing or incomplete records
+- Ensuring results are actionable and interpretable for business and regulatory
+  purposes
+- Attention to modeling bias, fairness, privacy, and responsible model
+  deployment

0_domain_study/research_questions/new/initial_questions.md

@@ -0,0 +1,48 @@
+# Initial Research Questions
+
+This document summarizes the initial exploratory research questions that
+informed the early direction of our study into credit risk and default
+prediction in U.S. peer-to-peer (P2P) lending.
+
+## Main Questions
+
+1. What are the key borrower and loan characteristics that best predict default
+   risk in P2P lending platforms in the United States?
+2. How effectively can machine learning models leverage these variables to
+   predict loan defaults on P2P platforms?
+3. Which modeling techniques and data engineering approaches yield the most
+   robust and interpretable risk predictions?
+4. How do macroeconomic, demographic, and geographic factors influence default
+   risk patterns among P2P borrowers?
+
+## Supporting Questions
+
+### Technical
+
+- Which data cleaning and feature engineering steps are critical for preparing
+  P2P loan datasets?
+- How do models such as logistic regression, random forest, and gradient
+  boosting compare for default prediction?
+- What role do explainability frameworks (e.g., SHAP, LIME) play in model
+  transparency?
+
+### Business
+
+- How can P2P platforms enhance loan origination and pricing using predictive
+  risk analytics?
+- What characteristics should investors prioritize when selecting loans for
+  their portfolios?
+
+### User
+
+- Which borrower demographics and financial behaviors most strongly correlate
+  with default risk?
+- What patterns emerge when examining default outcomes by borrower or loan
+  attribute?
+
+## Next Steps
+
+- [ ] Obtain and preprocess the Lending Club dataset from Kaggle
+- [ ] Perform exploratory data analysis (EDA) to identify predictive patterns
+- [ ] Fit baseline and advanced machine learning models for risk prediction
+- [ ] Analyze feature importance, refine models, and document findings