Commit Graph

5 Commits

Author SHA1 Message Date
David Peterson
eff82c73ce Add all_nullable configuration option in load_folder.py and load_sas.py for flexible schema management
Introduced an `all_nullable` boolean option in both `load_folder.py` and `load_sas.py`, allowing users to specify whether all columns should be treated as nullable during schema inference. This feature addresses scenarios where the data sampling may incorrectly suggest that columns are non-nullable, preventing potential errors during data loading. Updated YAML configuration files to include examples of this new option, enhancing usability and providing clearer documentation for users.
2026-04-21 16:48:37 -05:00
David Peterson
ae65140390 Add column type overrides in load_folder.py and load_sas.py for enhanced schema control
Implemented a new feature allowing users to specify explicit column type mappings via a `column_types` configuration in both `load_folder.py` and `load_sas.py`. This addition enables users to bypass automatic type inference for specific columns, ensuring correct data types are used when loading datasets. Updated the YAML configuration files to include examples of the new `column_types` option, enhancing usability and flexibility in handling varying data formats across files.
2026-04-21 12:14:44 -05:00
michael-corey
b3d7a9d440 adding index field 2026-04-20 10:18:09 -05:00
michael-corey
0d955eeab1 adding partition flag 2026-04-20 09:56:00 -05:00
michael-corey
f681f1012a Adding generic loader 2026-04-18 09:34:48 -05:00