batch_folder_processing #1

Merged

dp merged 7 commits from batch_folder_processing into main 2026-04-18 18:23:52 +00:00

Conversation 0 Commits 7 Files Changed 6 +1222 -44

dp commented 2026-04-18 16:26:27 +00:00

Contributor

Copy Link

No description provided.

dp added 4 commits 2026-04-18 16:26:28 +00:00

Enhance load_sas.py with detailed usage instructions, YAML config structure, and command-line interface documentation for loading SAS files. 4f7ded09c6

Implement type inference sampling in load_sas.py to improve performance on large SAS files. Introduce TYPE_INFERENCE_SAMPLE_ROWS to limit the number of rows scanned for type detection while ensuring nullability checks cover the entire column. Update documentation to reflect these changes. 3a0537270c

Update load_sas.py to support streaming data loads with iter_sas_chunks and copy_dataframes. Enhance documentation for schema inference and type detection, clarifying the use of read_sas_preview and the implications of sampling. Add __pycache__ to .gitignore. 5645ff5597

Add generate_sample_folder.py and load_folder.py for clustered SAS file generation and loading ... 5b48872dd7

Introduce generate_sample_folder.py to create a test folder with clustered SAS XPORT files, including configurations for schema compatibility checks. Implement load_folder.py to facilitate loading entire directories of SAS files into Postgres, supporting explicit and auto-detect clustering. Update sample_folder_config.yaml for usage examples and configuration structure. Enhance load_sas.py with a public schema compatibility check function for orchestrators.

dp requested review from mc 2026-04-18 16:26:35 +00:00

mc added 1 commit 2026-04-18 17:37:28 +00:00

adding user prompt for db creds 3b913b2ca6

dp added 1 commit 2026-04-18 17:41:10 +00:00

Update requirements.txt to support new package versions and add boto3 dependency c1e1fec10b

mc added 1 commit 2026-04-18 18:06:09 +00:00

removing latin encoding, adding usage notes 1bbe0d4cd6

mc approved these changes 2026-04-18 18:07:01 +00:00

dp merged commit 1bbe0d4cd6 into main 2026-04-18 18:23:52 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

mc

Labels

Clear labels

No items

No Label

Milestone

Clear milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

No Assignees

2 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: OFRA/foxtrot#1

No description provided.