I'm trying to analyze a large python file with different tasks on it. I've already read and preprocessed the file and it's now in memory. The thing is, the tasks I have, they pretty much have to go through the whole list of the records. It's something similar like:
resourceList = [..] #list of records from the file (say, 2GB) def taskA(): for i in resourceList: #doSthA() def taskB(): for i in resourceList: #doSthB()
If I do taskA() then taskB() it's going through the 2GB file twice and it's really slow. Is that a way that taskA and taskB can do their job simultaneously at the same time so that I don't have to go through the task twice?
I read about something which involved python threads and Queue, is that the only (and right) way to do it? If so, what if the "resourceList" is a generator instead of a list?