Feature/refactoring

[ypspy]

Note

Medium Risk
Medium risk because it introduces a large amount of new parsing/merging code and refactors the scraper entrypoint, which could change output datasets and scraping behavior if file patterns/column assumptions are off.

Overview
Refactors the DART scraping workflow into a more structured, config-driven pipeline: app.py now calls scraper/dart_scraper.py, reads config.yaml, and replaces the recursive retry logic with a loop using a configurable delay.

Adds a shared parsing utility module (parsers/common.py) plus many new parser scripts to extract audit/business-report fields into standardized TSV outputs, along with merge scripts (merge/e*.py) that combine those outputs and apply filters (year-end, industry exclusion, minimum audit hours).

Introduces project hygiene and tooling: new requirements.txt, pytest setup with unit tests for parsers/common.py, .gitignore, expanded README.md, and removes legacy/test artifacts (changeLog.md, test.py, archive/index.md).

^{Reviewed by Cursor Bugbot for commit 0d3d4fd. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 4 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0d3d4fd. Configure here.}

[cursor]

Parser output filenames don't match merge script expectations

High Severity

Several parsers write output files using an old naming convention (wp001_data_NNN_output.csv) that doesn't match the names the merge scripts expect (wp01.dataNNN.output.csv). For example, d4_1_time_information.py saves to wp001_data_007_output.csv, but all four merge scripts (e1, e3, e4, e5) try to read wp01.data06.output.csv. Similarly, d2_1 writes wp001_data_002_output.csv instead of wp01.data02.output.csv, d3_2 writes wp001_data_009_output.csv, and d5_1 writes wp001_data_008_output.csv. This breaks the entire pipeline since merge steps will fail with FileNotFoundError.

Additional Locations (2)

• parsers/d2_1_audit_cover_period.py#L71-L72
• parsers/d3_2_audit_opinion.py#L78-L79

 

^{Reviewed by Cursor Bugbot for commit 0d3d4fd. Configure here.}

[cursor]

Using find() > 0 misses matches at position zero

Medium Severity

find("감사위원회") > 0 fails to detect a match when the search term appears at the very start of the text (position 0), since 0 > 0 is False. The correct check is != -1 (or using in). The same pattern appears in d5_1_total_asset.py and d5_3_inv_rec.py for "단위", ":원", "부채", and "자산" lookups, potentially causing incorrect unit detection or missed balance sheet tables.

Additional Locations (1)

• parsers/d5_1_total_asset.py#L45-L46

 

^{Reviewed by Cursor Bugbot for commit 0d3d4fd. Configure here.}

[cursor]

Personal filesystem paths committed in config.yaml

Medium Severity

config.yaml contains hardcoded personal paths like C:/Users/ckpys/Desktop/output and E:/workingDirectory. This file appears to be a developer's local configuration that was committed to the repository. It won't work for any other contributor and exposes a username. This file would typically be .gitignored with a config.yaml.example template provided instead.

 

^{Reviewed by Cursor Bugbot for commit 0d3d4fd. Configure here.}

[cursor]

Duplicated data loading logic across all merge scripts

Low Severity

The identical block of code for loading wp01.data06.output.csv, renaming columns, filtering by December year-end, removing financial industry rows via industry.xlsx, and filtering by audit hours ≥ 100 is copy-pasted verbatim across e1_merge_period.py, e3_merge_reportdate.py, e4_merge_financials.py, and e5_audit_committee.py. Extracting this into a shared utility function would reduce maintenance burden and risk of inconsistent changes.

Additional Locations (2)

• merge/e3_merge_reportdate.py#L13-L42
• merge/e5_audit_committee.py#L13-L51

 

^{Reviewed by Cursor Bugbot for commit 0d3d4fd. Configure here.}