David Peterson 0c5e6e31f0 Enhance memory management in load_folder.py and load_sas.py for improved performance ... Added memory management optimizations in the _worker_load_append_file function to release unused memory from pyarrow's pool and trigger Python's garbage collection. Implemented explicit memory trimming using glibc's malloc_trim to ensure efficient memory usage during long-running processes. Updated the copy_dataframes function in load_sas.py to release pyarrow's memory pool between chunks, preventing high memory usage in long-lived workers. These changes aim to reduce memory footprint and improve overall performance during large dataset processing.		2026-04-21 10:46:54 -05:00
generic_loader	Enhance memory management in load_folder.py and load_sas.py for improved performance	2026-04-21 10:46:54 -05:00
utils	Refine partition name patterns in sas_profiler.py	2026-04-20 19:27:01 -05:00
.gitignore	adding explorer	2026-04-20 16:27:54 -05:00
requirements.txt	Update requirements and enhance SAS file processing with progress tracking	2026-04-20 21:44:49 -05:00