foxtrot/generic_loader
David Peterson 052fb0e087 Refactor pre-scan process in load_folder.py to utilize ThreadPoolExecutor for improved performance
Updated the main function to replace sequential file processing with a threaded approach using ThreadPoolExecutor. This change enhances the efficiency of reading row counts from SAS files, particularly for large datasets, by allowing concurrent I/O operations. Added progress tracking with tqdm for better user feedback during the pre-scan phase.
2026-04-20 22:43:02 -05:00
..
samples adding explorer 2026-04-20 16:27:54 -05:00
.env.example Adding generic loader 2026-04-18 09:34:48 -05:00
generate_sample_folder.py Add generate_sample_folder.py and load_folder.py for clustered SAS file generation and loading 2026-04-18 11:25:04 -05:00
generate_sample_sas.py Adding generic loader 2026-04-18 09:34:48 -05:00
load_folder.py Refactor pre-scan process in load_folder.py to utilize ThreadPoolExecutor for improved performance 2026-04-20 22:43:02 -05:00
load_sas.py Enhance load_cluster function for parallel processing and progress tracking 2026-04-20 22:02:55 -05:00
sample_config.yaml adding index field 2026-04-20 10:18:09 -05:00
sample_folder_config.yaml Enhance file clustering by implementing numeric sorting for last digit groups in stems and updating documentation for embedded-digit handling in auto-detection. 2026-04-20 11:48:22 -05:00