Skip to content

3DHISTECH-1.zip crashes ingest with OOM #226

Description

@dmoore247

Environment
AWS

Cluster configuration JSON:
Serverless CPU Job

Fully reproducible code snippet

Full error message

  src/dbx/pixels/utils.py:113-115 reads each inner zip entry whole into RAM
  (f.write(file_object.read())), OOM-killing the serverless worker on 3DHISTECH-1.zip (361 MB) and
  CMU-1-JP2K-33005.zip (127 MB) which were uploaded to s3://hls-eng-data-public/dicom/landing_zone/
   on 2026-06-04. Suggested fix: replace with shutil.copyfileobj. Until then, install must filter
  or re-skip those zips

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions