I'm writing a program that uses dynamic programming to solve a difficult problem. The DP solution requires storing a large table. The full table occupies approximately 300 Gb. Physically it is stored in 40 ~7Gb files. I'm marking unused table entries with the byte
\xFF. I'd like to allocate space for this table quickly. The program will have to run both under Windows and Linux.
In short, I want to efficiently create large files filled with a specific byte in a cross-platform manner.
Here is the code I'm currently using:
def reset_storage(self, path): fill = b'\xFF' with open(path, 'wb') as f: for _ in range(3715948544 * 2): f.write(fill)
It takes it about 40 minutes to create one 7 Gb file. How do I speed it up?
I've taken a look at other questions, but none of them seem to be relevant:
- Allocate a file of particular size in Linux with python — no answer
- create file of particular size in python — file is filled with
\0or solution is Windows-only
- How to create a file with a given size in Linux? — all solutions are Linux-specific