Merge pull request #1 from RichardScottOZ/copilot/create-python-package

Create geoscience_data_quality Python package from notebook code

Author: RichardScottOZ

README.md

@@ -1,15 +1,108 @@
 # Geoscience-Data-Quality-for-Machine-Learning
 
+A Python package for assessing geoscience data quality for machine learning.
+
 A problem exists when building broad scale models, for example, Australia.
+Disparate datasets from many domains need to be assessed for quality before
+being combined into machine learning pipelines. This package provides tools
+to quantify and map data quality across geoscience datasets.
+
+## Installation
+
+```bash
+pip install -e .
+```
+
+With optional dependencies:
+
+```bash
+# For Excel file support
+pip install -e ".[excel]"
+
+# For gravity point-density analysis (verde, xarray, pooch)
+pip install -e ".[gravity]"
+
+# For visualization (matplotlib)
+pip install -e ".[viz]"
+
+# Everything
+pip install -e ".[all]"
+
+# Development (includes tests)
+pip install -e ".[dev]"
+```
+
+## Package Modules
+
+### `geoscience_data_quality.quality_model`
+Quality scoring model for geoscience datasets. Load quality models from
+CSV/Excel, compute resolution scores, final quality scores, and filter
+by domain or sub-domain.
+
+```python
+from geoscience_data_quality import load_quality_model, compute_final_score, compute_resolution_score
+
+model = load_quality_model("DataQuality_Models.csv")
+res_score = compute_resolution_score(90.0)   # finer resolution → higher score
+final = compute_final_score(score=3, presence=1.0, resolution_score=res_score)
+```
 
-## Disparate datasets, breaking them down into broad domains:
+### `geoscience_data_quality.vector`
+Analyze quality fields (confidence, observation method, positional accuracy,
+metadata) in geological vector datasets.
+
+```python
+from geoscience_data_quality import analyze_quality_fields, get_quality_summary
+
+results = analyze_quality_fields(geology_gdf, fields=["confidence", "obsmethod"])
+summary = get_quality_summary(geology_gdf)
+```
+
+### `geoscience_data_quality.survey`
+Fetch, filter, and fix geophysical survey metadata from WFS services such
+as Geoscience Australia's GADDS.
+
+```python
+from geoscience_data_quality import fetch_ga_survey_metadata, filter_surveys, fix_survey_geometry
+
+surveys = fetch_ga_survey_metadata()
+mag_line = filter_surveys(surveys, measure_type="magnetic", dataset_type="line")
+gdf = fix_survey_geometry(mag_line, swap_coordinates=True)
+```
+
+### `geoscience_data_quality.rasterize`
+Rasterize vector quality attributes onto reference grids or new grids
+defined by bounds and resolution.
+
+```python
+from geoscience_data_quality import rasterize_vector_attribute
+
+array = rasterize_vector_attribute(
+    gdf, column="max_line_spacing_m",
+    reference_raster="model_raster.tif",
+    output_path="survey_quality.tif",
+    sort_ascending=False,  # smallest (best) value wins in overlaps
+)
+```
+
+### `geoscience_data_quality.point_density`
+Compute observation point density for datasets like gravity stations
+(requires the `gravity` optional dependencies).
+
+```python
+from geoscience_data_quality.point_density import compute_point_density
+
+coords, counts = compute_point_density((longitude, latitude), spacing=0.1)
+```
+
+## Disparate datasets, breaking them down into broad domains
 
 - Geophysics (Gravity, Magnetics, Radiometrics, Seismic, Electromagnetic, Induced Polarisation, Magnetotelluric...)
 - Geology (Lithology, Stratigraphy, Structure, Hydro..)
 - Remote Sensing (Landsat, ASTER, Sentinel...)
 - Geochemistry (Rock, Soil, Water, Assay techniques...)
 
-## Variety of data layers:
+## Variety of data layers
 
 - Direct observations
 - Gridded Data
@@ -57,8 +150,9 @@ How, thinking in a raster fashion, to get a combined per-pixel Data Quality rati
 - Simple qualitative (3/2/1, Good/Average/Bad, High/Medium/Low or other ordinals).
 - Exists / Missing
 
-  # Reference
-  - [https://www.researchgate.net/profile/Alan_Aitken/publication/326193704/figure/fig1/AS:646297606443016@1531100765653/](https://www.researchgate.net/publication/326193704_A_role_for_data_richness_mapping_in_exploration_decision_making)
+## Reference
+
+- [A role for data richness mapping in exploration decision making (Aitken et al)](https://www.researchgate.net/publication/326193704_A_role_for_data_richness_mapping_in_exploration_decision_making)
 
 ![sample map output](https://github.com/RichardScottOZ/Geoscience-Data-Quality-for-Machine-Learning/blob/main/reliability_index.png "Sample Quality Map - derived from Leonardo Uieda's Australia Gravity Data repository work")
 

pyproject.toml

@@ -0,0 +1,65 @@
+[build-system]
+requires = ["setuptools>=64", "setuptools-scm>=8"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "geoscience-data-quality"
+version = "0.1.0"
+description = "Tools for assessing geoscience data quality for machine learning"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.9"
+authors = [
+    {name = "Richard Scott"},
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: GIS",
+    "Topic :: Scientific/Engineering",
+]
+dependencies = [
+    "geopandas>=0.12",
+    "numpy>=1.22",
+    "pandas>=1.4",
+    "rasterio>=1.3",
+    "shapely>=2.0",
+]
+
+[project.optional-dependencies]
+excel = [
+    "openpyxl>=3.0",
+]
+gravity = [
+    "verde>=1.7",
+    "xarray>=2022.3",
+    "pooch>=1.6",
+]
+viz = [
+    "matplotlib>=3.5",
+]
+all = [
+    "geoscience-data-quality[excel,gravity,viz]",
+]
+dev = [
+    "geoscience-data-quality[all]",
+    "pytest>=7.0",
+    "pytest-cov>=4.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/RichardScottOZ/Geoscience-Data-Quality-for-Machine-Learning"
+Repository = "https://github.com/RichardScottOZ/Geoscience-Data-Quality-for-Machine-Learning"
+Issues = "https://github.com/RichardScottOZ/Geoscience-Data-Quality-for-Machine-Learning/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

src/geoscience_data_quality/__init__.py

@@ -0,0 +1,32 @@
+"""Tools for assessing geoscience data quality for machine learning."""
+
+__version__ = "0.1.0"
+
+from geoscience_data_quality.quality_model import (
+    compute_final_score,
+    compute_resolution_score,
+    load_quality_model,
+)
+from geoscience_data_quality.rasterize import rasterize_vector_attribute
+from geoscience_data_quality.survey import (
+    fetch_ga_survey_metadata,
+    filter_surveys,
+    fix_survey_geometry,
+)
+from geoscience_data_quality.vector import (
+    analyze_quality_fields,
+    get_quality_summary,
+)
+
+__all__ = [
+    "__version__",
+    "analyze_quality_fields",
+    "compute_final_score",
+    "compute_resolution_score",
+    "fetch_ga_survey_metadata",
+    "filter_surveys",
+    "fix_survey_geometry",
+    "get_quality_summary",
+    "load_quality_model",
+    "rasterize_vector_attribute",
+]

src/geoscience_data_quality/point_density.py

@@ -0,0 +1,74 @@
+"""Point density analysis for geophysical observations.
+
+Functions for computing observation density from point data such as
+gravity station locations, providing a spatial measure of data quality.
+
+Based on the Gravity-Survey-Quality notebook which uses verde for block
+reduction to compute points-per-pixel.
+
+These functions require the optional ``verde`` dependency.  Install it
+with::
+
+    pip install geoscience-data-quality[gravity]
+"""
+
+from __future__ import annotations
+
+from typing import Optional
+
+import numpy as np
+
+
+def compute_point_density(
+    coordinates: tuple[np.ndarray, np.ndarray],
+    spacing: float = 0.1,
+    center_coordinates: bool = True,
+) -> tuple[tuple[np.ndarray, np.ndarray], np.ndarray]:
+    """Compute point density using block reduction.
+
+    Divides the area into blocks of the given *spacing* and counts the
+    number of points in each block.
+
+    Parameters
+    ----------
+    coordinates : tuple of ndarray
+        ``(longitude, latitude)`` arrays of observation locations.
+    spacing : float
+        Block size in degrees. Default ``0.1``.
+    center_coordinates : bool
+        If ``True``, return the centre of each block as the
+        coordinates. Default ``True``.
+
+    Returns
+    -------
+    coords : tuple of ndarray
+        ``(longitude, latitude)`` of block centres.
+    counts : ndarray
+        Number of points in each block.
+
+    Raises
+    ------
+    ImportError
+        If ``verde`` is not installed.
+    """
+    try:
+        import verde as vd
+    except ImportError as exc:
+        raise ImportError(
+            "The 'verde' package is required for point density analysis. "
+            "Install it with: pip install geoscience-data-quality[gravity]"
+        ) from exc
+
+    def _count(array: np.ndarray) -> int:
+        return array.size
+
+    # Create dummy data matching the coordinate arrays
+    dummy_data = np.ones(coordinates[0].shape)
+
+    coords, counts = vd.BlockReduce(
+        _count,
+        center_coordinates=center_coordinates,
+        spacing=spacing,
+    ).filter(coordinates, data=dummy_data)
+
+    return coords, counts