[FIXED] Why is BufferedReader readLine reading past EOF

Issue

I have a very large file (~6GB) that has fixed-width text separated by \r\n, and so I'm using buffered reader to read line by line. This process can be interrupted or stopped and if it is, it uses a checkpoint "lastProcessedLineNbr" to fast forward to the correct place to resume reading. This is how the reader is initialized.

private void initializeBufferedReader(Integer lastProcessedLineNbr) throws IOException {
    reader = new BufferedReader(new InputStreamReader(getInputStream(), "UTF-8"));
    if(lastProcessedLineNbr==null){lastProcessedLineNbr=0;}

    for(int i=0; i<lastProcessedLineNbr;i++){
        reader.readLine();
    }
    currentLineNumber = lastProcessedLineNbr;
}

This seems to work fine, and I read and process the data in this method:

public Object readItem() throws Exception {
    if((currentLine = reader.readLine())==null){
        return null;
    }
    currentLineNumber++;
    return parse(currentLine);
}

And again, everything works fine until I reach the last line in the document. readLine() in the latter method throws an error:

17:06:49,980 ERROR [org.jberet] (Batch Thread - 1) JBERET000007: Failed to run job ProdFileRead, parse, org.jberet.job.model.Chunk@3965dcc8: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
    at java.util.Arrays.copyOf(Arrays.java:3332)
    at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:569)
    at java.lang.StringBuffer.append(StringBuffer.java:369)
    at java.io.BufferedReader.readLine(BufferedReader.java:370)
    at java.io.BufferedReader.readLine(BufferedReader.java:389)
    at com.rational.batch.reader.TextLineReader.readItem(TextLineReader.java:55)

Curiously, it seems to be reading past the end of the file and allocating so much space that it runs out of memory. I tried looking at the contents of the file using Cygwin and "tail file.txt" and in the console it gave me the expected 10 lines. But when I did "tail file.txt > output.txt" output.txt ended up being like 1.8GB, much larger than the 10 lines I expected. So it seems Cygwin is doing the same thing. As far as I can tell there is no special EOF character. It's just the last byte of data and it ends abruptly.

Anyone have any idea on how I can get this working? I'm thinking I could resort to counting the number of bytes read until I get the full size of the file, but I was hoping there was a better way.

Solution

But when I did tail file.txt > output.txt output.txt ended up being like 1.8GB, much larger than the 10 lines I expected

What this indicates to me is that the file is padded with 1.8GB of binary zeroes, which Cygwin's tail command ignored when writing to the terminal, but which Java is not ignoring. This would explain your OutOfMemoryError as well, as the BufferedReader continued reading data looking for the next \r\n, never finding it before overflowing memory.

Answered By - Jim Garrison

Answer Checked By - David Goodson (PHPFixing Volunteer)

Read More

[FIXED] How to read a textual HTTP response into a String exactly as-is?

Issue

The following code uses BufferedReader to read from an HTTP response stream:

final StringBuilder responseBuilder = new StringBuilder();
String line = bufferedReader.readLine();
while (line != null) {
    responseBuilder.append(line);
    responseBuilder.append('\n');
    line = bufferedReader.readLine();
}
response = responseBuilder.toString();

But appending '\n' to each line seems a bit flawed. I want to return the HTTP response exactly as-is so what if it doesn't have a return character after the last line? One would get added anyway using the code above - is there a better way?

Solution

I want to return the HTTP response exactly as-is

Don't use readLine() then - it's as simple as that. I'd suggest using a StringWriter instead:

StringWriter writer = new StringWriter();
char[] buffer = new char[8192];
int charsRead;
while ((charsRead = bufferedReader.read(buffer)) > 0) {
    writer.write(buffer, 0, charsRead);
}
response = writer.toString();

Note that even this won't work if you get the encoding wrong. To preserve the exact HTTP response, you'd need to read (and write) it as a binary stream.

Answered By - Jon Skeet

Answer Checked By - Katrina (PHPFixing Volunteer)

Read More

[FIXED] How do I convert a BufferedReader-Input (String) to Integer and save it in an Integer List in java?

Issue

I would like to search for an Integer in a String given by a BufferedReader. The Integers have to be saved inside an Integer-List and be returned. My Idea was splitting the String in a String [ ] and save the Integers with Integer.parseInt directly inside the Array-List, but unfortunatelly i only get NumberFormatExceptions, although the String [ ] is filled. Could someone give me some advice?

    public List<Integer> getIntList(BufferedReader br) {
        List <Integer> List = new ArrayList<>();
        try{
            while(br.ready()){
                try{
                    String line = (br.readLine());
                    String [] arr = line.split("\\s");
                    for (String s : arr) {
                        System.out.println(s);
                    }
                    if(line.equals("end")){
                        return List;
                    }
                    for (String s : arr) {
                        List.add(Integer.parseInt(s));

                    }
                }
                catch(IOException e){
                    System.err.println("IOException");
                }
                catch(NumberFormatException e){
                    System.out.println("Number");
                }
            }
            return List;
        }
        catch(IOException e){
            System.err.println("IOException");
        }
    return null;
    }

Solution

You catch NumberFormatException in a wrong place so that you cannot continue number searching loop. You have to wrap this line List.add(Integer.parseInt(s)); into try catch block. Also never start variable name with capital letter.

Answered By - Dmitry Pukhov

Answer Checked By - David Marino (PHPFixing Volunteer)

Read More

[FIXED] How to convert a string number into an integer in Java?

Issue

I have imported a text file into my program that contains several scores from a gamer. I am in need of adding the scores from each gamer - while attempting to do so, I realized that those scores are coming from a text file, therefore, they are actually strings, not numbers. I am wanting to convert the string number into an integer but have not been able to do so by using the parseInt() method, I have attempted to use the following method in my code:

String myString = "1234";

int foo = Integer.parseInt(myString);

But have not been able to do it successfully. Here is my current work:

import java.io.File;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.io.BufferedReader;


public class luis_ramirez_GamesReport {

    private static String gamer;

    public static void main(String[] args) throws IOException
    {
        File fileName = new File("/Users/luisramirez/eclipse-workspace/GameScores.txt");
        if (fileName.exists())
        {
            BufferedReader br = null;
            String line = "";
            String cvsSplitBy = ",";
            int recordCount = 0;
            String number ="167";
            int result = Integer.parseInt(number);
            
            br = new BufferedReader(new FileReader(fileName));
        
        System.out.println("-----------------------------------------------------------------------------------------------");
        System.out.println("Games Report");
        System.out.println("-----------------------------------------------------------------------------------------------");
        System.out.println("Gamer    1       2       3       4       5       6       7       8       9       10       Total");
        System.out.println("-----------------------------------------------------------------------------------------------");
        while ((line = br.readLine()) != null)
        {
        String[] record = line.split(cvsSplitBy);
        System.out.println(record[0] + "\t"
            + record[1] + (record[2].length() > 7 ? "\t" : "\t")
            + record[2] + (record[2].length() > 7 ? "\t" : "\t")
            + record[3] + (record[3].length() > 7 ? "\t" : "\t")
            + record[4] + (record[4].length() > 7 ? "\t" : "\t")
            + record[5] + (record[5].length() > 7 ? "\t" : "\t")
            + record[6] + (record[6].length() > 7 ? "\t" : "\t")
            + record[7] + (record[7].length() > 7 ? "\t" : "\t")
            + record[8] + (record[8].length() > 7 ? "\t" : "\t")
            + record[9] + (record[9].length() > 7 ? "\t" : "\t")
            + record[10] + (record[10].length() > 7 ? "\t" : "\t")
            + record[1] + record[2] + record[3] + record[4] + record[5] + record[6] + record[7] + record[8] + record[9] + record[10]);
            recordCount++;

        }
        System.out.println("----------------------------------------------------------------------------------------------");
        System.out.printf("# of Gamers: %d%n",recordCount);
        System.out.println("Top Gamer: Adelie");
        System.out.println("----------------------------------------------------------------------------------------------");
            br.close();
        }

This is my output:

-----------------------------------------------------------------------------------------------
Games Report
-----------------------------------------------------------------------------------------------
Gamer    1       2       3       4       5       6       7       8       9       10       Total
-----------------------------------------------------------------------------------------------
Bob 167 123 159 102 102 189 183 173 197 148 167123159102102189183173197148
Sally   189 130 138 113 159 116 134 196 150 144 189130138113159116134196150144
Mario   104 106 120 188 143 189 149 174 163 100 104106120188143189149174163100
Lev 152 159 195 140 154 176 107 128 166 181 152159195140154176107128166181
Carden  158 200 175 114 117 150 176 181 131 132 158200175114117150176181131132
Adelie  175 199 122 104 198 182 175 153 120 165 175199122104198182175153120165
Lada    161 108 102 193 151 197 115 137 126 186 161108102193151197115137126186
Xavier  178 171 147 113 107 129 128 189 165 195 178171147113107129128189165195
Raffi   176 144 151 124 149 112 158 159 119 177 176144151124149112158159119177
Chang   135 144 177 153 143 125 145 140 117 158 135144177153143125145140117158
Mich    156 105 178 137 165 180 128 115 139 157 156105178137165180128115139157
Mason   162 185 108 106 113 135 139 135 197 160 162185108106113135139135197160
Cora    186 115 106 126 135 108 157 156 187 120 186115106126135108157156187120
Sergio  117 105 115 116 193 200 176 134 122 153 117105115116193200176134122153
Jonas   132 163 196 101 134 159 131 104 135 168 132163196101134159131104135168
----------------------------------------------------------------------------------------------
# of Gamers: 15
Top Gamer: Adelie
----------------------------------------------------------------------------------------------

Please note that the output of my scores appears correctly when the program is ran.

In addition, here is the information from the text file:

Bob,167,123,159,102,102,189,183,173,197,148 Sally,189,130,138,113,159,116,134,196,150,144 Mario,104,106,120,188,143,189,149,174,163,100 Lev,152,159,195,140,154,176,107,128,166,181 Carden,158,200,175,114,117,150,176,181,131,132 Adelie,175,199,122,104,198,182,175,153,120,165 Lada,161,108,102,193,151,197,115,137,126,186 Xavier,178,171,147,113,107,129,128,189,165,195 Raffi,176,144,151,124,149,112,158,159,119,177 Chang,135,144,177,153,143,125,145,140,117,158 Mich,156,105,178,137,165,180,128,115,139,157 Mason,162,185,108,106,113,135,139,135,197,160 Cora,186,115,106,126,135,108,157,156,187,120 Sergio,117,105,115,116,193,200,176,134,122,153 Jonas,132,163,196,101,134,159,131,104,135,168

Any insight you can kindly provide would be greatly appreciated.

Solution

I'm assuming from your original code that you want to parseInt because you want the final column to display the sum properly.

To accomplish this, I remove the first element from your String array record (since this is the name of the person), and then I use Java's Streams API in order to convert each element into an integer and get the sum. There was also some whitespace around some of the integers that were Strings, so I stripped the whitespace before using parseInt.

Here is my implementation:

package com.sandbox;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;

public class ScoresDisplay {

    public static void main(String[] args) throws IOException
    {
        File fileName = new File("path/to/scores.txt");
        if (fileName.exists())
        {
            BufferedReader br = null;
            String line = "";
            String cvsSplitBy = ",";
            int recordCount = 0;
            String number ="167";
            int result = Integer.parseInt(number);
            
            br = new BufferedReader(new FileReader(fileName));
        
        System.out.println("-----------------------------------------------------------------------------------------------");
        System.out.println("Games Report");
        System.out.println("-----------------------------------------------------------------------------------------------");
        System.out.println("Gamer    1       2       3       4       5       6       7       8       9       10       Total");
        System.out.println("-----------------------------------------------------------------------------------------------");
        while ((line = br.readLine()) != null)
        {
        String[] record = line.split(cvsSplitBy);
        String[] ints_only = Arrays.copyOfRange(record, 1, record.length);
        List<Integer> recordAsInts = Arrays.stream(ints_only)
            .map(str -> str.strip())
            .map(Integer::parseInt)
            .collect(Collectors.toList());
        int sum = recordAsInts.stream().reduce(Integer::sum).orElse(0);
        System.out.println(record[0] + "\t"
            + record[1] + (record[2].length() > 7 ? "\t" : "\t")
            + record[2] + (record[2].length() > 7 ? "\t" : "\t")
            + record[3] + (record[3].length() > 7 ? "\t" : "\t")
            + record[4] + (record[4].length() > 7 ? "\t" : "\t")
            + record[5] + (record[5].length() > 7 ? "\t" : "\t")
            + record[6] + (record[6].length() > 7 ? "\t" : "\t")
            + record[7] + (record[7].length() > 7 ? "\t" : "\t")
            + record[8] + (record[8].length() > 7 ? "\t" : "\t")
            + record[9] + (record[9].length() > 7 ? "\t" : "\t")
            + record[10] + (record[10].length() > 7 ? "\t" : "\t")
            + sum);
            recordCount++;

        }
        System.out.println("----------------------------------------------------------------------------------------------");
        System.out.printf("# of Gamers: %d%n",recordCount);
        System.out.println("Top Gamer: Adelie");
        System.out.println("----------------------------------------------------------------------------------------------");
            br.close();
        }
    }

}

Output:

I'm using a picture instead here because the formatting looks weird in text.

Answered By - Richard K Yu

Answer Checked By - Dawn Plyler (PHPFixing Volunteer)

Read More