jeudi 30 avril 2015

Python different operations on the same resources

I'm trying to analyze a large python file with different tasks on it. I've already read and preprocessed the file and it's now in memory. The thing is, the tasks I have, they pretty much have to go through the whole list of the records. It's something similar like:

resourceList = [..] #list of records from the file (say, 2GB)
def taskA():
    for i in resourceList:
        #doSthA()

def taskB():
    for i in resourceList:
        #doSthB()

If I do taskA() then taskB() it's going through the 2GB file twice and it's really slow. Is that a way that taskA and taskB can do their job simultaneously at the same time so that I don't have to go through the task twice?

I read about something which involved python threads and Queue, is that the only (and right) way to do it? If so, what if the "resourceList" is a generator instead of a list?

Thanks!

Aucun commentaire:

Enregistrer un commentaire