David Peterson dd83f58412 Implement parallel processing for partition discovery in load_folder.py and enhance column filtering in load_sas.py ... Added support for parallel processing using ProcessPoolExecutor in the _discover_cluster_partitions function, allowing for efficient partition value discovery across multiple files. This change significantly reduces I/O overhead by reading only necessary columns during scans. Additionally, updated iter_sas_chunks and iter_text_chunks functions to accept a usecols parameter, enabling selective column parsing for improved performance during data loading. These enhancements aim to optimize resource usage and speed up the data processing pipeline.		2026-04-21 21:43:42 -05:00
..
samples	adding explorer	2026-04-20 16:27:54 -05:00
.env.example	Adding generic loader	2026-04-18 09:34:48 -05:00
generate_sample_folder.py	Add generate_sample_folder.py and load_folder.py for clustered SAS file generation and loading	2026-04-18 11:25:04 -05:00
generate_sample_sas.py	Adding generic loader	2026-04-18 09:34:48 -05:00
load_folder.py	Implement parallel processing for partition discovery in load_folder.py and enhance column filtering in load_sas.py	2026-04-21 21:43:42 -05:00
load_sas.py	Implement parallel processing for partition discovery in load_folder.py and enhance column filtering in load_sas.py	2026-04-21 21:43:42 -05:00
sample_config.yaml	adding text file support	2026-04-21 20:05:26 -05:00
sample_folder_config.yaml	adding text file support	2026-04-21 20:05:26 -05:00