dp

0 Followers · 1 Following

• Joined on 2026-04-18

Repositories Projects Packages Public Activity Starred Repositories

dp created pull request OFRA/foxtrot#8 2026-04-21 22:23:03 +00:00

advanced_analyzer

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 21:56:30 +00:00

64e7ff0b0a Enhance error reporting in load_folder.py and load_sas.py for better debugging

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 21:48:42 +00:00

eff82c73ce Add all_nullable configuration option in load_folder.py and load_sas.py for flexible schema management

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 20:55:28 +00:00

c283b42876 Add safe numeric to datetime conversion in load_sas.py to handle edge cases

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 18:40:40 +00:00

a46f0518f6 Suppress PerformanceWarning in load_sas.py to reduce noise during processing of wide SAS files. This change filters out warnings related to DataFrame fragmentation, which are irrelevant for our pipeline as we directly convert DataFrames to pyarrow tables.

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 18:17:04 +00:00

969a442775 Refactor numeric column type inference in load_sas.py for improved data handling

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 17:54:08 +00:00

212218fb67 Enhance error handling and abort functionality in load_folder.py for parallel file loading

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 17:14:46 +00:00

ae65140390 Add column type overrides in load_folder.py and load_sas.py for enhanced schema control

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 15:46:57 +00:00

0c5e6e31f0 Enhance memory management in load_folder.py and load_sas.py for improved performance

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 15:05:23 +00:00

9afb52aecb Add --chunk-rows option to load_folder.py for customizable memory management

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 13:31:50 +00:00

eac75cbb26 Refactor load_cluster function in load_folder.py for improved parallel file loading

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 13:16:18 +00:00

1265489276 Enhance date and timestamp handling in _prepare_for_copy function in load_sas.py

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 13:12:41 +00:00

2dd247b067 Add --no-prescan option to load_folder.py for skipping metadata scan

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 03:43:05 +00:00

052fb0e087 Refactor pre-scan process in load_folder.py to utilize ThreadPoolExecutor for improved performance

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 03:02:57 +00:00

fe7dc4d5a1 Enhance load_cluster function for parallel processing and progress tracking

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 02:44:52 +00:00

96f2d6fe79 Update requirements and enhance SAS file processing with progress tracking

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 02:32:58 +00:00

7beb44ac4d Add pyarrow dependency and optimize DataFrame serialization in load_sas.py

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 02:08:16 +00:00

5e347f50ef Add widening compatibility checks in load_sas.py for type inference

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 01:43:31 +00:00

f84e127796 Update type inference behavior in load_sas.py to scan entire files by default

dp pushed to advanced_analyzer at OFRA/foxtrot 2026-04-21 00:27:04 +00:00

a94ab68f4d Refine partition name patterns in sas_profiler.py