[FIXED] How do I merge multiple cvs files into one DataFrame in pandas?

Issue

I have a folder with a lot of csv-files each containing messurements of signal data. They have the following structure:

Frequency [kHz],Power [dbm]
852000,-135.812845793404
852008,-142.13849097071088
852016,-138.21218081816156
852024,-137.32593610384734
852032,-139.464539680863

I want to merge these files into a DataFrame with Frequency as the key column, because the frequency is the same in every file. So it should look something like this in the DataFrame:

Frequency [kHz] | Power [dbm] | Power [dbm] | Power [dbm] | ...

So I wrote the following code:

df = pd.DataFrame()
for f in csv_files:
    csv = pd.read_csv(f)
    df = pd.merge(df, csv, on='Frequency [kHz]', sort=False)

But the only thing I get is an KeyError: 'Frequency [kHz]'

The closest I came to my desired result was through pd.concat([pd.read_csv(f) for f in csv_files], axis=0, sort=False) but then there are still those Frequency columns in between.

Solution

I think you can collect them all as dfs, and then merge, like so:

import pandas as pd
from functools import reduce

data_frames = []
for f in csv_files:
    df = pd.read_csv(f)
    data_frames.append(df)

df_merged = reduce(lambda left, right: pd.merge(left, right, on=['Frequency [kHz]'],
                                            how='outer'), data_frames)

Answered By - Jasmin Heifa

Answer Checked By - Katrina (PHPFixing Volunteer)