Concatenate a list of pandas dataframes together

How to Concatenate a List of Pandas DataFrames

So you have a list of Pandas DataFrames and you want to combine them into one mighty DataFrame? You've come to the right place! 🐼

Problem: Combining Multiple DataFrames

Let's say you have a list of Pandas DataFrames like this:

mydfs = [d1, d2, d3]

And you want to concatenate them into one single DataFrame. How do you do it? 🤔

Solution: The concat Function

Fear not, because the solution is as simple as using the concat function from the Pandas library. 🙌

Here's how you can concatenate your DataFrames:

import pandas as pd

result_df = pd.concat(mydfs)

And voilà! 🎉 You now have your concatenated DataFrame stored in result_df.

Example

To make it even clearer, let's see an example using the provided sample dataframes:

# sample dataframes
d1 = pd.DataFrame({'one': [1., 2., 3., 4.], 'two': [4., 3., 2., 1.]})
d2 = pd.DataFrame({'one': [5., 6., 7., 8.], 'two': [9., 10., 11., 12.]})
d3 = pd.DataFrame({'one': [15., 16., 17., 18.], 'two': [19., 10., 11., 12.]})

# list of dataframes
mydfs = [d1, d2, d3]

# concatenate DataFrames
result_df = pd.concat(mydfs)

By using the concat function, you've successfully combined d1, d2, and d3 into one pandas DataFrame named result_df. 🎊

Advanced Tip: Handling Large Tables with chunksize

If you're dealing with large tables and using the chunksize option when reading data from a database, you can still leverage the concat function.

For example, let's assume you have already created your list of dataframes using the chunksize option:

# create empty list
dfs = []
sqlall = "select * from mytable"

# iteratively read data from database
for chunk in pd.read_sql_query(sqlall, cnxn, chunksize=10000):
    dfs.append(chunk)

Once you have your list of dataframes filled, you can concatenate them in the same manner as before:

result_df = pd.concat(dfs)

The chunksize option allows you to read a large-ish table in chunks, making it memory-efficient. And when it's time to combine those chunks into one DataFrame, concat comes to the rescue!

Call-to-Action: Share Your Experiences

Now that you've mastered the art of concatenating Pandas DataFrames, it's time to put your skills to the test. 🚀

Go ahead and use this newfound knowledge to solve your own data manipulation challenges. And make sure to share your experiences with us in the comments below!

If you found this guide helpful, don't forget to share it with fellow data enthusiasts. Let's spread the joy of concatenating DataFrames together! 📈🔄

Happy coding! 😄

Note: This blog post assumes you have a basic understanding of Pandas and Python. If you're new to Pandas, it's always a good idea to start with the official documentation and tutorials.