Merge pull request #49 from ncbo/feat/run_umls_pipeline

 Automate the full umls2rdf pipeline

Author: alexskr

.gitignore

@@ -1,6 +1,7 @@
 conf.py
 *.pyc
 output/*
+data/*
 
 *.swp
 sync*

README.md

@@ -1,11 +1,18 @@
 This project takes a MySQL Unified Medical Language System (UMLS) database and converts the ontologies to RDF using OWL and SKOS as the main schemas.
 
-Virtual Appliance users can review the [documentation in the OntoPortal Administration Guide}(https://ontoportal.github.io/documentation/administration/ontologies/handling_umls).
+Virtual Appliance users can review the [documentation in the OntoPortal Administration Guide](https://ontoportal.github.io/documentation/administration/ontologies/handling_umls).
 
-To use it:
+Recommended workflow:
 
-* Specify your database connection conf.py
-* Specify the SAB ontologies to export in umls.conf
+* Install Python dependencies with <code>pip install -r requirements.txt</code>
+* Configure <code>conf.py</code>
+* Specify the SAB ontologies to export in <code>umls.conf</code>
+* Run the full resumable import/export pipeline with <code>python run_umls_pipeline.py</code>
+
+Generated TTL files are written under a versioned output directory based on
+<code>OUTPUT_FOLDER</code> from <code>conf.py</code>. A common pattern is
+<code>OUTPUT_FOLDER = "output/%s" % UMLS_VERSION.upper()</code>, which writes to
+<code>output/2025AB</code>.
 
 The umls.conf configuration file must contain one ontology per line. The lines are comma separated tuples where the elements are:
 
@@ -23,11 +30,59 @@ umls2rdf.py is designed to be an offline, run-once process.
 It's memory intensive and exports all of the default ontologies in umls.conf in 3h 30min. 
 The ontologies listed in umls.conf are the UMLS ontologies accessible in [BioPortal](https://bioportal.bioontology.org/).
 
-If you get an error when installing the MySQL-python python library, https://stackoverflow.com/questions/12218229/my-config-h-file-not-found-when-intall-mysql-python-on-osx-10-8 may be of help.
+To download the full UMLS release archive outside the full pipeline, run:
+
+<pre>
+python download_umls.py
+</pre>
+
+The downloader returns the local path to the downloaded archive. This step only
+fetches and extracts the pre-built UMLS release; you still need to load the
+UMLS tables into MySQL before running <code>umls2rdf.py</code>. The script uses
+<code>UMLS_VERSION</code> and <code>UMLS_API_KEY</code> from <code>conf.py</code>.
+If <code>UMLS_DOWNLOAD_DIR</code> is set, the zip archive is stored under that
+directory. If it is not set, the library default <code>~/.data/bio/umls</code>
+is used. By default, the archive is extracted into an
+<code>extracted</code> subdirectory next to the downloaded zip. You can override
+that location with <code>UMLS_EXTRACT_DIR</code>.
+
+To create the target MySQL database with explicit UTF-8 settings outside the
+full pipeline, run:
+
+<pre>
+python create_mysql_db.py
+</pre>
+
+The script creates or updates <code>DB_NAME</code> from <code>conf.py</code>
+with <code>utf8mb4</code> character set and
+<code>utf8mb4_unicode_ci</code> collation.
+
+To run the full UMLS pipeline end-to-end, use:
+
+<pre>
+python run_umls_pipeline.py
+</pre>
+
+The pipeline performs these stages:
+
+* Download the configured UMLS full release archive
+* Extract the release only when the extracted <code>META</code> directory is not already present
+* Recreate the configured <code>DB_NAME</code> and load it with the extracted <code>META/populate_mysql_db.sh</code> script
+* Run <code>umls2rdf.py</code>
 
-If running a Windows 10 OS with MySQL, the following tips may be of help.
+The pipeline patches loader settings from <code>conf.py</code> into a generated
+copy of <code>populate_mysql_db.sh</code>, and it patches
+<code>META/mysql_tables.sql</code> in place to replace
+<code>@LINE_TERMINATION@</code>. Pipeline state is stored under
+<code>PIPELINE_WORK_DIR</code> (default:
+<code>data/pipeline/&lt;UMLS_VERSION&gt;</code>) and reruns skip completed steps
+after validating the extracted files, MySQL tables, and RDF output. Add
+<code>MYSQL_HOME</code> to <code>conf.py</code>; if your MySQL client is at
+<code>/usr/bin/mysql</code>, set <code>MYSQL_HOME = "/usr"</code>. Pipeline
+stdout and stderr are appended to <code>PIPELINE_LOG_FILE</code> when set, or
+to <code>data/pipeline/&lt;UMLS_VERSION&gt;/pipeline.log</code> by default.
 
-- Install [MySQL 5.5](https://dev.mysql.com/downloads/mysql/5.5.html#downloads) to avoid the InnoDB space [disclaimer](https://www.nlm.nih.gov/research/umls/implementation_resources/scripts/README_RRF_MySQL_Output_Stream.html) by NLM. 
-- [Python 2.7.x](https://www.python.org/downloads/) should be used to avoid syntax errors on 'raise Attribute'
-- For installtion of the MySQLdb module <pre>python -m pip install MySQLdb</pre> is error prone. Install with executable [MySQL-python-1.2.3.win-amd64-py2.7](http://www.codegood.com/archives/129) (last known location).
-- Create your RRF subset(s) using mmsys with the MySQL load option, load your database, edit conf.py and umls.py to specifications, run umsl2rdf.py
+If <code>PROCESS_ONLY_CURRENT_UMLS_VERSION</code> is set to <code>True</code>,
+the exporter only processes ontologies whose <code>MRSAB.IMETA</code> exactly
+matches <code>UMLS_VERSION</code>. Ontologies with a different value are skipped
+and logged.

conf_sample.py

@@ -1,19 +1,35 @@
-#Folder to dump the RDF files.
-OUTPUT_FOLDER = "output"
-
 UMLS_VERSION = "2025AB"
 
-#DB Config
+# Folder to dump the RDF files.
+OUTPUT_FOLDER = "output/%s" % UMLS_VERSION.upper()
+
+# DB Config
 DB_HOST = "localhost"
 DB_NAME = "umls%s" % UMLS_VERSION.lower()
 DB_USER = "umls"
 DB_PASS = "umls"
-
-# Define the base URI used to generate the concepts URI
-UMLS_BASE_URI = "http://purl.bioontology.org/ontology/"
+MYSQL_HOME = "/usr"
 
 # Include the semantic type concepts for each Ontology file generated
 INCLUDE_SEMANTIC_TYPES = True
 
+# Define the base URI used to generate the concepts URI
+UMLS_BASE_URI = "http://purl.bioontology.org/ontology/"
+
 # Only process ontologies updated in this UMLS release (MRSAB.IMETA == UMLS_VERSION)
 PROCESS_ONLY_CURRENT_UMLS_VERSION = False
+
+# Pipeline config
+UMLS_API_KEY = "your umls api key"
+
+# Optional: final umls download directory, e.g. data/umls
+# UMLS_DOWNLOAD_DIR = "data/umls"
+
+# Optional: directory for extracted full UMLS contents
+# UMLS_EXTRACT_DIR = "data/umls-extracted"
+
+# Optional: working directory for pipeline state and patched loader script
+# PIPELINE_WORK_DIR = "data/pipeline"
+
+# Optional: pipeline log file path
+# PIPELINE_LOG_FILE = "data/pipeline/pipeline.log"

create_mysql_db.py

@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+
+import argparse
+
+import pymysql
+
+import conf
+
+
+def connect_server():
+    return pymysql.connect(
+        host=conf.DB_HOST,
+        user=conf.DB_USER,
+        passwd=conf.DB_PASS,
+        charset="utf8mb4",
+        autocommit=True,
+    )
+
+
+def database_exists(connection, db_name):
+    with connection.cursor() as cursor:
+        cursor.execute("SHOW DATABASES LIKE %s", (db_name,))
+        return cursor.fetchone() is not None
+
+
+def create_database(connection, db_name):
+    with connection.cursor() as cursor:
+        cursor.execute(
+            """
+            CREATE DATABASE IF NOT EXISTS `{db_name}`
+            CHARACTER SET utf8mb4
+            COLLATE utf8mb4_unicode_ci
+            """.format(db_name=db_name)
+        )
+        cursor.execute(
+            "ALTER DATABASE `{db_name}` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci".format(
+                db_name=db_name
+            )
+        )
+
+
+def drop_database(connection, db_name):
+    with connection.cursor() as cursor:
+        cursor.execute("DROP DATABASE IF EXISTS `{db_name}`".format(db_name=db_name))
+
+
+def ensure_database(db_name, recreate=False):
+    connection = connect_server()
+    try:
+        if recreate:
+            drop_database(connection, db_name)
+        create_database(connection, db_name)
+    finally:
+        connection.close()
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--recreate",
+        action="store_true",
+        help="Drop the configured database before creating it.",
+    )
+    return parser.parse_args()
+
+
+def main():
+    args = parse_args()
+    ensure_database(conf.DB_NAME, recreate=args.recreate)
+    print(conf.DB_NAME)
+
+
+if __name__ == "__main__":
+    main()

download_umls.py

@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+
+import os
+from pathlib import Path
+import zipfile
+
+import conf
+
+
+def get_extract_dir(zip_path):
+    extract_dir = getattr(conf, "UMLS_EXTRACT_DIR", None)
+    if extract_dir:
+        return Path(extract_dir).expanduser().resolve()
+    return zip_path.parent / "extracted"
+
+
+def main():
+    from umls_downloader import download_umls_full
+
+    download_dir = getattr(conf, "UMLS_DOWNLOAD_DIR", None)
+    if download_dir:
+        download_dir = Path(download_dir).expanduser().resolve()
+        os.environ["BIO_HOME"] = download_dir.parent.as_posix()
+
+    path = download_umls_full(
+        version=conf.UMLS_VERSION.upper(),
+        api_key=conf.UMLS_API_KEY,
+    )
+    extract_dir = get_extract_dir(path)
+    extract_dir.mkdir(parents=True, exist_ok=True)
+
+    with zipfile.ZipFile(path) as zip_file:
+        zip_file.extractall(extract_dir)
+
+    print(extract_dir)
+
+
+if __name__ == "__main__":
+    main()

requirements.txt

@@ -0,0 +1,2 @@
+pymysql
+umls_downloader