jeudi 5 novembre 2015

How to find similar records of one csv file into another csv file write all records into a result.csv file by python

I would like to search and then copy / merge two CSV files based on the common records of adv_id and user_id in both files based on conv_id, and store records in a new CSV file. That means I will take the records from both CSV files if the adv_id and user_id is same in both files. But when I use nested for loops for to search same adv_id and user_id of fileOne into fileTwo, it takes very long execution times. This is because, fileOne contains 8 Million records, fileTwo contain 50 million records. Below is an example of the files and expected result file. fileOne and fileTwo contain the fillowing columns. Your kind suggestions in highly appreciated.

fileOne: adv_id, conv_id, user_id, dwell, session, pv, elapsed, cvflg.

fileTwo: adv_id, user_id, dwell, session, pv, elapsed, cvflg

Image of fileOne, fileTwo, and Result file

Python "List assignment index out of range"

I am a novice student in Python (and programming in general).

I am supposed to make a python program that opens two files with random numbers in them and creates a new file with the numbers ordered from lowest to highest.

So I made this code that iterates using two for loops through all the numbers, searching for the lowest, very basic stuff, than stores the number and its position, appends to a Lmix list that will be saved on the final file and stores the numbers position to delete it from that list so it won't be found again.

The variables are in Portuguese, but I translated them in the comments, the rest are self-explanatory.

arq1 = open("nums1.txt","r")
arq2 = open("nums2.txt","r")

arqmix = open("numsord.txt","w")

L1 = arq1.readlines()
L2 = arq2.readlines()
Lmix = []

L1 = list(map(int,L1)) # converts lists into int
L2 = list(map(int,L2))

cont = 0

menor = L1[0]  # "Menor" is the variable that stores the lowest number it finds
menorpos = 0   # "Menorpos" is the position of that variable in the list, so it can delete later
listdec = 0    # "listdec" just stores which list the number was from to delete.

while cont != (len(L1)+len(L2)):   

# while loops that finds the lowest number, stores the number and position, appends to the Lmix and deletes from the list so it won't be found on next   iteration

    n = 0
    for n,x in enumarate(L1):
        m = 0
        for m,y in enumarate(L2):
            if x<menor:
                menor = x
                menorpos = n
                listdec = 0
            elif y<menor:
                menor = y
                menorpos = m
                listdec = 1
            m += 1
        n += 1

    Lmix.append(menor)
    if listdec == 0:
        del L1[menorpos]
    elif listdec == 1:
        del L2[menorpos]
    cont += 1

for x in Lmix:
    arqmix.write("%d\n"%x)

arq1.close()
arq2.close()
arqmix.close()

But everytime I run it, this error appears:

Traceback (most recent call last): File "C:/Users/Danzmann-Notebook/PycharmProjects/untitled/aula18.py", line 41, in del L2[menorpos] IndexError: list assignment index out of range

I know what it means but I just can't understand why it happens, and how can I solve it.

Any help would be appreciated.

Thanks in advance, sorry for any grammar error, english is not my native language.

C++ function prototyping input file into vector

create a function which reads a text file & returns its contents in a vector; its prototype is sdd::vector<std::string> SDI::readTextFile(const std::string );

The initial specification is to read a text file the name of the file is the parameter, which can include a path. The return value will be a vector of strings.

What is the second function asking me to do? Would std::ifstream File("textFile.txt");meet the prototype?

Getting information from a file

can anyone help me how this file created?

i want to get the data of this file.

is it compressed or encrypted?

the file is created in Delphi compiled program.

file composed of numbers and colors

Concantenate user input Strings to convert into a complete file path (Java)

I wrote a short script to create a file to my Desktop, and the file appeared. I just did it all in main, like so:

    import java.io.*;
    import java.util.Scanner;
public class FilePractice {
  public static void main(String[] args) {

    //create a new File object
    File myFile = new File("/home/christopher/Desktop/myFile");

    try{
        System.out.println("Would you like to create a new file? Y or N: ");
        Scanner input = new Scanner(System.in);
        String choice = input.nextLine();
        if(choice.equalsIgnoreCase("Y"))
        {
            myFile.createNewFile();
        }
        else
        {
            //do nothing
        }
    }catch(IOException e) {
        System.out.println("Error while creating file " + e);
    }
    System.out.println("'myFile' " + myFile.getPath() + " created.");
  }
}

I just wanted to make sure the code worked, which it did. After that, I wanted to expand by creating a file with user input, as well as define which directory the user wished to send the file to. I'm on a Linux machine, and I wanted to send it to my Desktop again, so my user input was "/home/christopher/Desktop" for the userPath. Nothing happened. I even cd'd to my Desktop via terminal to "ls" everything there, and still nothing.

Perhaps my syntax is wrong?

If this is a duplicate of anything, my apologies. I tried to do a thorough search before coming here, but I only found info on creating files and sending files to directories that are already defined as a string (e.g. File myFile = new File("/home/User/Desktop/myFileName")).

Here is the expanded attempt:

try {
       System.out.println("Alright. You chose to create a new file.\nWhat would you like to name the file?");
            String fileName = input.nextLine();
            input.nextLine();
            System.out.println("Please enter the directory where you would like to save this file.\nFor example: C:\\Users\\YourUserName\\Documents\\");
            String userFilePath = input.nextLine();
            File userFile = new File(userFilePath, fileName);
            System.out.println("Is this the file path you wish to save to? ----> " + userFile.getPath()+"\nY or N: ");
            String userChoice = input.nextLine();

            if (userChoice.equalsIgnoreCase("Y")) {
                userFile.createNewFile();
                //print for debug 
                System.out.println(userFile.getPath());
               }
            }catch(IOException e) {
                System.out.println("Error while attempting to create file " + e);
            }
            System.out.println("File created successfully");

My print statement for a debug attempt outputs "/home/christopher/Desktop", but not the file name appended to the directory.

Thanks for any help offered. This is just for experimentation while learning Java I/O. Since a hypothetical user may not be on the same OS as me, I can work on those methods later. I'm keeping it on my home machine, hence the Unix filepath names.

How to iterate through set of files based on file names?

I have a set of files named like this:

qd-p64-dZP-d64-z8-8nn.q         
qd-p8-dPZ-d8-z1-1nn.q             qq-p8-dZP-d8-z1-2nn.q
qd-p8-dPZ-d8-z1-2nn.q             qq-p8-dZP-d8-z1-4nn.q
qd-p8-dPZ-d8-z1-4nn.q             qq-p8-dZP-d8-z16-1nn.q
qd-p8-dPZ-d8-z16-1nn.q            qq-p8-dZP-d8-z16-2nn.q
qd-p8-dPZ-d8-z16-2nn.q            qq-p8-dZP-d8-z16-4nn.q
qd-p8-dPZ-d8-z16-4nn.q            qq-p8-dZP-d8-z16-8nn.q
qd-p8-dPZ-d8-z16-8nn.q            qq-p8-dZP-d8-z1-8nn.q
qd-p8-dPZ-d8-z1-8nn.q             qq-p8-dZP-d8-z2-1nn.q
qd-p8-dPZ-d8-z2-1nn.q             qq-p8-dZP-d8-z2-2nn.q
qd-p8-dPZ-d8-z2-2nn.q             qq-p8-dZP-d8-z2-4nn.q 
qd-p8-dPZ-d8-z2-4nn.q             qq-p8-dZP-d8-z2-8nn.q
qd-p8-dPZ-d8-z2-8nn.q             qq-p8-dZP-d8-z32-1nn.q
qd-p8-dPZ-d8-z32-1nn.q            qq-p8-dZP-d8-z32-2nn.q
qd-p8-dPZ-d8-z32-2nn.q            qq-p8-dZP-d8-z32-4nn.q
qd-p8-dPZ-d8-z32-4nn.q            qq-p8-dZP-d8-z32-8nn.q
qd-p8-dPZ-d8-z32-8nn.q            qq-p8-dZP-d8-z4-1nn.q
qd-p8-dPZ-d8-z4-1nn.q             qq-p8-dZP-d8-z4-2nn.q
qd-p8-dPZ-d8-z4-2nn.q             qq-p8-dZP-d8-z4-4nn.q

The information to iterate is given in the file names, for example:

Fix

dZP, 1nn, z2,  

and vary

d 

with values

{d8, d16, d32 d64}

Then, increase z value to get

dZP, 1nn, z4

and vary d again

{d8, d16, d32 d64}   

Once I'm able to iterate like this I need to do some information processing from the files.

Clojure lazily read random line from file

I have a sample data set in a txt file. The data file is extremely large so loading it in memory is not an option. I need to be able to read the file lazily. Furthermore, I need the lines to be read in a random order. And there might be cases where I don't need to read all the lines. This is what I found so far -

(defn read-lazy [in-file]
        (letfn [(helper [rdr]
                            (if-let [line (.readLine rdr)]
                                (cons line (helper rdr))
                                (do (.close rdr) nil)))]
            (helper (io/reader in-file))))

which returns a lazy-seq of the file. How can I loop through random lines in the lazy-seq when I need to? I think using a go block could help here. Go blocks could put a random line in a channel and await for something to consume it. Once the data gets read it puts another line in the channel awaits for the next read. How can I implement that?

Here's how I've worked it out (not random) -

(def lazy-ch (chan))
(defn async-fetch-set [in-file]
    (go
        (with-open [reader (io/reader in-file)]
            (doseq [line (line-seq reader)]
                (>! lazy-ch line)))
        (close! lazy-ch)))

(println "got: " (<!! lazy-ch))

Is this a good way to approach the problem? Is there a better solution? I might not need to read all the lines so I'd like to be able to close the reader if whenever I need to.