Processing huge tables with pandas and multiprocessing
A practical pattern for splitting a giant table into chunks and using every CPU core without running out of memory.
Data Engineer @ Infinyty Tech
Small beginnings. Unbreakable foundations.
I design and run the pipelines that move billions of records from legacy systems into modern databases, accurately, quickly and without drama.
A practical pattern for splitting a giant table into chunks and using every CPU core without running out of memory.
From intern to full-time data engineer in nine months, and why I'm going to document the journey one post at a time.