6 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
michael-corey	3b913b2ca6	adding user prompt for db creds	2026-04-18 12:37:22 -05:00
David Peterson	5b48872dd7	Add generate_sample_folder.py and load_folder.py for clustered SAS file generation and loading ... Introduce generate_sample_folder.py to create a test folder with clustered SAS XPORT files, including configurations for schema compatibility checks. Implement load_folder.py to facilitate loading entire directories of SAS files into Postgres, supporting explicit and auto-detect clustering. Update sample_folder_config.yaml for usage examples and configuration structure. Enhance load_sas.py with a public schema compatibility check function for orchestrators.	2026-04-18 11:25:04 -05:00
David Peterson	5645ff5597	Update load_sas.py to support streaming data loads with iter_sas_chunks and copy_dataframes. Enhance documentation for schema inference and type detection, clarifying the use of read_sas_preview and the implications of sampling. Add __pycache__ to .gitignore.	2026-04-18 10:44:32 -05:00
David Peterson	3a0537270c	Implement type inference sampling in load_sas.py to improve performance on large SAS files. Introduce TYPE_INFERENCE_SAMPLE_ROWS to limit the number of rows scanned for type detection while ensuring nullability checks cover the entire column. Update documentation to reflect these changes.	2026-04-18 10:28:37 -05:00
David Peterson	4f7ded09c6	Enhance load_sas.py with detailed usage instructions, YAML config structure, and command-line interface documentation for loading SAS files.	2026-04-18 10:20:07 -05:00
michael-corey	f681f1012a	Adding generic loader	2026-04-18 09:34:48 -05:00