PHPFixing
  • Privacy Policy
  • TOS
  • Ask Question
  • Contact Us
  • Home
  • PHP
  • Programming
  • SQL Injection
  • Web3.0

Monday, August 29, 2022

[FIXED] How to read and turn a 13Gbite text in python?

 August 29, 2022     csv, dataset, pandas, python, text     No comments   

Issue

I have a 1.5Gbite rar file that when I open it it gives me 13 Gbite .txt file and I can't load it with anything. I used:

import pandas as pd
read_file = pd.read_csv (r'Path where the Text file is stored\File name.txt', sep='\t')
read_file.to_csv (r'Path where the CSV will be saved\File name.csv', index=None)

I used it in Pycharm and it takes forever and gives

MemoryError

I used colab in it crashed and I think it's because my Drive was almost full because I only have 15G of drive. What should I do?


Solution

The chunksize parameter specifies the number of rows per chunk.

chunksize = 10 ** 6
with pd.read_csv(filename, chunksize=chunksize, on_bad_lines='skip') as reader:
    for chunk in reader:
        # you can use chunk to process the data in parts


Answered By - Himanshuman
Answer Checked By - Willingham (PHPFixing Volunteer)
  • Share This:  
  •  Facebook
  •  Twitter
  •  Stumble
  •  Digg
Newer Post Older Post Home

0 Comments:

Post a Comment

Note: Only a member of this blog may post a comment.

Total Pageviews

Featured Post

Why Learn PHP Programming

Why Learn PHP Programming A widely-used open source scripting language PHP is one of the most popular programming languages in the world. It...

Subscribe To

Posts
Atom
Posts
Comments
Atom
Comments

Copyright © PHPFixing