vendredi 30 octobre 2015

How to efficiently allocate a file of predefined size and fill it with a non-zero value using Python?

I'm writing a program that uses dynamic programming to solve a difficult problem. The DP solution requires storing a large table. The full table occupies approximately 300 Gb. Physically it is stored in 40 ~7Gb files. I'm marking unused table entries with the byte \xFF. I'd like to allocate space for this table quickly. The program will have to run both under Windows and Linux.

In short, I want to efficiently create large files filled with a specific byte in a cross-platform manner.

Here is the code I'm currently using:

def reset_storage(self, path):
    fill = b'\xFF'

    with open(path, 'wb') as f:
        for _ in range(3715948544 * 2):
            f.write(fill)

It takes it about 40 minutes to create one 7 Gb file. How do I speed it up?

I've taken a look at other questions, but none of them seem to be relevant:

Aucun commentaire:

Enregistrer un commentaire