foxtrot/generic_loader
David Peterson 857f696305 Optimize datetime parsing in load_sas.py by implementing a sample-based format detection approach
Introduced a new mechanism to sample non-null values for determining the appropriate datetime parsing strategy, significantly reducing processing time for large datasets. This change replaces the previous full row-walk method with a more efficient sampling technique, enhancing performance while maintaining robust handling of various date formats. Updated comments for clarity on the new approach.
2026-04-22 12:54:19 -05:00
..
samples adding explorer 2026-04-20 16:27:54 -05:00
.env.example moving to env file 2026-04-22 15:37:35 +00:00
generate_sample_folder.py Add generate_sample_folder.py and load_folder.py for clustered SAS file generation and loading 2026-04-18 11:25:04 -05:00
generate_sample_sas.py Adding generic loader 2026-04-18 09:34:48 -05:00
load_folder.py Implement parallel processing for partition discovery in load_folder.py and enhance column filtering in load_sas.py 2026-04-22 15:35:19 +00:00
load_sas.py Optimize datetime parsing in load_sas.py by implementing a sample-based format detection approach 2026-04-22 12:54:19 -05:00
sample_config.yaml adding text file support 2026-04-21 20:05:26 -05:00
sample_folder_config.yaml adding text file support 2026-04-21 20:05:26 -05:00