0c5e6e31f0
Added memory management optimizations in the _worker_load_append_file function to release unused memory from pyarrow's pool and trigger Python's garbage collection. Implemented explicit memory trimming using glibc's malloc_trim to ensure efficient memory usage during long-running processes. Updated the copy_dataframes function in load_sas.py to release pyarrow's memory pool between chunks, preventing high memory usage in long-lived workers. These changes aim to reduce memory footprint and improve overall performance during large dataset processing. |
||
|---|---|---|
| .. | ||
| samples | ||
| .env.example | ||
| generate_sample_folder.py | ||
| generate_sample_sas.py | ||
| load_folder.py | ||
| load_sas.py | ||
| sample_config.yaml | ||
| sample_folder_config.yaml | ||