Skip to content

feat(S3UriPath): Add S3UriPath class to override __fspath__() for pandas competibility#209

Open
SafetyMary wants to merge 3 commits intoliormizr:masterfrom
SafetyMary:dev_pd_path
Open

feat(S3UriPath): Add S3UriPath class to override __fspath__() for pandas competibility#209
SafetyMary wants to merge 3 commits intoliormizr:masterfrom
SafetyMary:dev_pd_path

Conversation

@SafetyMary
Copy link
Copy Markdown

@SafetyMary SafetyMary commented Feb 13, 2026

Add S3UriPath class for pandas competibility

Add a new S3UriPath class to override __fspath__ for pandas read methods competitibility

Implementation

  • Based on Pandas compatibility #204
  • Inherited S3Path to create the new class S3UriPath
    • Added S3UriPath class in __init__.py for python 3.12 or above only
  • Prepended "s3:/" to __fspath__ return
    • Did not use to_uri() due to URL encoding
  • Updated docs in interface.rst
    • I tried editing the s3path_graph.svg file with gimp but it doesnt work out. Might need some help on this
    • Please let me know if any more documentations needed to be updated regarding to this PR
  • Did not add any new test cases since testing pd.read from s3 would require connection to s3 buckets
    • Only tested on my personal s3 bucket

Testing issues

Boto3 versioning

The stated boto3 version 1.16.35 in setup.py cannot be imported with the following error. So I used 1.42.48 to run testing instead

Traceback (most recent call last):
  File "<python-input-0>", line 1, in <module>
    import boto3
  File "/home/safetymary/Desktop/safety_git/s3path/.venv/lib/python3.14/site-packages/boto3/__init__.py", line 16, in <module>
    from boto3.session import Session
  File "/home/safetymary/Desktop/safety_git/s3path/.venv/lib/python3.14/site-packages/boto3/session.py", line 17, in <module>
    import botocore.session
  File "/home/safetymary/Desktop/safety_git/s3path/.venv/lib/python3.14/site-packages/botocore/session.py", line 29, in <module>
    import botocore.configloader
  File "/home/safetymary/Desktop/safety_git/s3path/.venv/lib/python3.14/site-packages/botocore/configloader.py", line 19, in <module>
    from botocore.compat import six
  File "/home/safetymary/Desktop/safety_git/s3path/.venv/lib/python3.14/site-packages/botocore/compat.py", line 33, in <module>
    from botocore.vendored.six.moves import http_client
ModuleNotFoundError: No module named 'botocore.vendored.six.moves'

Failed case

All test cases passed except the following. This test case also failed at the master branch so i believe it is unrelated to this PR.

======================================================== test session starts ========================================================
platform linux -- Python 3.14.1, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/safetymary/Desktop/safety_git/s3path
configfile: pyproject.toml
collected 108 items

tests/test_not_supported.py .................                                                                                 [ 15%]
tests/test_path_operations.py ................................................                                                [ 60%]
tests/test_pure_path_operations.py ...............................F...                                                        [ 92%]
tests/test_s3path_configuration.py ........                                                                                   [100%]

============================================================= FAILURES ==============================================================
____________________________________________________________ test_match _____________________________________________________________

    def test_match():
        assert PureS3Path('a/b.py').match('*.py')
        assert PureS3Path('/a/b/c.py').match('b/*.py')
        assert not PureS3Path('/a/b/c.py').match('a/*.py')
        assert PureS3Path('/a.py').match('/*.py')
        assert not PureS3Path('a/b.py').match('/*.py')
>       assert not PureS3Path('a/b.py').match('*.Py')
E       AssertionError: assert not True
E        +  where True = match('*.Py')
E        +    where match = PureS3Path('a/b.py').match
E        +      where PureS3Path('a/b.py') = PureS3Path('a/b.py')

tests/test_pure_path_operations.py:179: AssertionError
========================================================= warnings summary ==========================================================
tests/test_pure_path_operations.py::test_from_uri_issue_150
tests/test_pure_path_operations.py::test_uri
tests/test_pure_path_operations.py::test_uri
tests/test_pure_path_operations.py::test_uri
  /home/safetymary/Desktop/safety_git/s3path/s3path/current_version.py:136: DeprecationWarning: pathlib.PurePath.as_uri() is deprecated and scheduled for removal in Python 3.19. Use pathlib.Path.as_uri().
    uri = super().as_uri()

tests/test_pure_path_operations.py::test_reserved
  /home/safetymary/Desktop/safety_git/s3path/tests/test_pure_path_operations.py:163: DeprecationWarning: pathlib.PurePath.is_reserved() is deprecated and scheduled for removal in Python 3.15. Use os.path.isreserved() to detect reserved paths on Windows.
    assert not PureS3Path('/a/b').is_reserved()

tests/test_pure_path_operations.py::test_reserved
  /home/safetymary/Desktop/safety_git/s3path/tests/test_pure_path_operations.py:164: DeprecationWarning: pathlib.PurePath.is_reserved() is deprecated and scheduled for removal in Python 3.15. Use os.path.isreserved() to detect reserved paths on Windows.
    assert not PureS3Path('a/b').is_reserved()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
====================================================== short test summary info ======================================================
FAILED tests/test_pure_path_operations.py::test_match - AssertionError: assert not True
============================================ 1 failed, 107 passed, 6 warnings in 32.04s =============================================

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant