[FIXED] How to keep the same datatype after saving data to CSV file?

Issue

I'm saving data to CSV files using pandas. I have one important column with dtype: datetime64[ns].Somehow the datatype is changed to object when I read the data back from CSV file. How can I write, read while keeping the same datatype? Is this related to the encoding type?

df = pd.io.sql.read_sql(sql, cnxn)
df.to_csv(fileName)
df.TimeSeries

Name: TimeSeries, Length: 10000, dtype: datetime64[ns]

DF = pd.read_csv(fileName, sep=',')
DF.TimeSeries

Name: TimeSeries, Length: 10000, dtype: object

Solution

CSV files do not store data types. Data in CSV files is stored as text.

Your best options are:

1. Store in a serialized or other type-aware format (pickle, HDF5) if this is appropriate for your use case.
2. Use the parse_dates argument of pd.read_csv, e.g. df = pd.read_csv(filename, sep=',', parse_dates=['Date']). See pd.read_csv documentation for more details.

The second option is just a workaround. It will convert text back to datetime when you read the data into a dataframe.

Answered By - jpp

Answer Checked By - Dawn Plyler (PHPFixing Volunteer)