Pandas is a popular data analysis library in Python programming language that provides high-performance data structures and tools for data manipulation. One of the key functions of Pandas is to combine multiple datasets into a single one. Pandas provides various functions for concatenating, merging and joining pandas dataframes. In this article, we will discuss ‘Pandas Concat’ and learn how to use it to combine multiple pandas dataframes with code examples.
Pandas Concat:
In Pandas, the concat function is used to concatenate multiple pandas dataframes either vertically or horizontally. The vertical concatenation is used when we need to stack the dataframes one over another, whereas horizontal concatenation is used when we need to concatenate the dataframes side-by-side. Pandas concat function consists of several parameters that allow us to control the output or customize it according to our requirements.
The syntax of pandas concat is as follows:
pd.concat(objs, axis=0, join='outer', ignore_index=False, keys=None, levels=None, names=None, verify_integrity=False, sort=False, copy=True)
Parameters:
objs: List or dictionary of pandas dataframes that needs to be concatenated
axis: Concatenation axis, use 0 for vertical concatenation, and 1 for horizontal concatenation (default is 0)
join: Join method for dataframe columns, use ‘inner’ to use only common columns or ‘outer’ to use all columns (default is ‘outer’)
ignore_index: Set to True if the concatenated axis should be re-indexed (default is False)
keys: List of keys to be used to create a hierarchical index (default is None)
levels: Lists of key labels for the hierarchy levels (default is None)
names: List of names for the levels of the resulting hierarchical index (default is None)
verify_integrity: Check to see if the new dataframe contains duplicate rows or indexes (default is False)
sort: Sort the final dataframe (default is False)
copy: Copy the data instead of using reference (default is True)
Concatenating Multiple Dataframes with Pandas:
Let's combine multiple dataframes vertically i.e. rows stacked over each other. We have two dataframes named 'df1' and 'df2' containing the same columns and indexes.
import pandas as pd
data1 = {'A': [1,2,3], 'B': [4,5,6]}
df1 = pd.DataFrame(data1, index=['a', 'b', 'c'])
data2 = {'A': [7,8,9], 'B': [10,11,12]}
df2 = pd.DataFrame(data2, index=['d', 'e', 'f'])
print(df1)
print(df2)
Output:
A B
a 1 4
b 2 5
c 3 6
A B
d 7 10
e 8 11
f 9 12
Now we will combine these two dataframes vertically using the 'concat' function as shown below:
vertical_concat = pd.concat([df1, df2])
print(vertical_concat)
Output:
A B
a 1 4
b 2 5
c 3 6
d 7 10
e 8 11
f 9 12
As you can see, the two dataframes are stacked vertically, and the resulting dataframe has the same columns as input dataframes with rows concatenated.
Let's concatenate these dataframes horizontally i.e. columns side-by-side. We have two dataframes named 'df3' and 'df4', containing different columns and indexes.
data3 = {'C': [13,14,15], 'D': [16,17,18]}
df3 = pd.DataFrame(data3, index=['a', 'b', 'c'])
data4 = {'E': [19,20,21], 'F': [22,23,24]}
df4 = pd.DataFrame(data4, index=['a', 'b', 'c'])
print(df3)
print(df4)
Output:
C D
a 13 16
b 14 17
c 15 18
E F
a 19 22
b 20 23
c 21 24
Now we will combine these two dataframes horizontally using the 'concat' function as shown below:
horizontal_concat = pd.concat([df3, df4], axis=1)
print(horizontal_concat)
Output:
C D E F
a 13 16 19 22
b 14 17 20 23
c 15 18 21 24
As you can see, the two dataframes are concatenated horizontally, and the resulting dataframe has all columns from input dataframes with rows side-by-side.
Using Join in Pandas Concat:
In Pandas Concat function, join parameter is used to control how the new dataframe will handle columns that are not common in input dataframes. The default value is ‘outer’ which means that all columns are taken from all input dataframes. If the column is missing from input dataframes, the column will be filled with NaN values.
Let's say we have two dataframes where one contains additional columns that are not present in the other dataframe.
data5 = {'A': [1,2,3], 'B': [4,5,6], 'C': [7,8,9]}
df5 = pd.DataFrame(data5, index=['a', 'b', 'c'])
data6 = {'A': [10,11,12], 'B': [13,14,15]}
df6 = pd.DataFrame(data6, index=['d', 'e', 'f'])
print(df5)
print(df6)
Output:
A B C
a 1 4 7
b 2 5 8
c 3 6 9
A B
d 10 13
e 11 14
f 12 15
Now we will use the 'join' parameter to only take the common columns from the input dataframes.
joined = pd.concat([df5, df6], join='inner', axis=1)
print(joined)
Output:
A B A B
a 1 4 NaN NaN
b 2 5 NaN NaN
c 3 6 NaN NaN
d NaN NaN 10 13
e NaN NaN 11 14
f NaN NaN 12 15
As you can see, only the common columns are taken from input dataframes, and the resulting dataframe has NaN values for the columns that are not present in the input dataframes.
Using Hierarchical Index in Pandas Concat:
In Pandas Concat function, we can use a hierarchical index to differentiate the input dataframes. We can use the 'keys' parameter to give names to the dataframe(s) that will be concatenated.
Let's create two dataframes with different columns and indexes and concatenate them using the 'keys' parameter.
data7 = {'A': [1,2,3], 'B': [4,5,6]}
df7 = pd.DataFrame(data7, index=['a', 'b', 'c'])
data8 = {'C': [7,8,9], 'D': [10,11,12]}
df8 = pd.DataFrame(data8, index=['d', 'e', 'f'])
print(df7)
print(df8)
Output:
A B
a 1 4
b 2 5
c 3 6
C D
d 7 10
e 8 11
f 9 12
Now we will use the 'keys' parameter to concatenate the two dataframes using a hierarchical index. We will use the keys 'first' and 'second' to name the input dataframes.
keys_concat = pd.concat([df7, df8], axis=0, keys=['first', 'second'])
print(keys_concat)
Output:
A B C D
first a 1.0 4.0 NaN NaN
b 2.0 5.0 NaN NaN
c 3.0 6.0 NaN NaN
second d NaN NaN 7.0 10.0
e NaN NaN 8.0 11.0
f NaN NaN 9.0 12.0
As you can see, a hierarchical index is created, and the input dataframes are named 'first' and 'second.'
Conclusion:
Pandas Concat is a powerful function that allows us to combine multiple dataframes into a single one either vertically or horizontally. In this article, we learned how to use the Pandas Concat function and its various parameters to control output and customize it according to our requirements. We hope this article helped you understand the concept of Pandas Concat and how to use it with code examples.
let me provide some more details about the topics mentioned in the article.
Pandas DataFrames:
A Pandas DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. It can be thought of as a spreadsheet or SQL table. It is similar to a two-dimensional array or a dictionary of Series, where each column is a Series object. It is one of the most commonly used data structures in data analysis using Python.
In Pandas DataFrame, rows and columns are both labeled with their index and column names, respectively. It provides a variety of functions to perform operations on data, such as filtering, sorting, merging, joining, grouping, aggregating, and many more.
Pandas Concat:
Pandas Concat is a function provided by the Pandas library that allows us to combine multiple Pandas DataFrames either vertically or horizontally. It is a very powerful function that allows us to concatenate multiple data frames with a variety of options to control the output.
Vertical Concatenation:
In Pandas Concat, vertical concatenation is used when we need to stack the dataframes one over another. We can pass multiple data frames as a list to the pd.concat() function to perform vertical concatenation. The stacked data frames will have their indexes preserved in the final result, which can cause issues if the indexes of the input data frames overlap.
Horizontal Concatenation:
In Pandas Concat, horizontal concatenation is used when we need to concatenate the data frames side-by-side. We can pass multiple data frames as a list to the pd.concat() function to perform horizontal concatenation. The columns of the resulting data frame are sorted lexicographically by their name.
Joining in Pandas Concat:
In Pandas Concat, join parameter is used to control how the new data frame will handle columns that are not common in input data frames. We can set the join parameter to 'inner' to use only common columns or 'outer' to use all columns. If the column is missing from input data frames, the column will be filled with NaN values.
Hierarchical Indexing in Pandas Concat:
In Pandas Concat, we can use hierarchical indexing to differentiate the input data frames. We can use the 'keys' parameter to give names to the data frames that will be concatenated. A hierarchical index is created, and the input data frames are named using the 'keys' parameter.
In conclusion, Pandas Concat is a very powerful function that allows us to combine multiple Pandas DataFrames vertically or horizontally. It provides many options to control the output and customize it according to our requirements. Pandas DataFrames are one of the most commonly used data structures in data analysis using Python, and they provide a variety of functions to perform operations on data.
Popular questions
Sure, here are 5 questions related to Pandas Concat with code examples and their answers.
Q1. What is Pandas Concat, and how is it used in Python?
A1. Pandas Concat is a function provided by the Pandas library that allows us to combine multiple Pandas DataFrames either vertically or horizontally. It is used to concatenate multiple data frames with a variety of options to control the output. Here is an example:
import pandas as pd
df1 = pd.DataFrame({'A': [1,2,3], 'B': [4,5,6]})
df2 = pd.DataFrame({'A': [7,8,9], 'B': [10,11,12]})
df3 = pd.concat([df1, df2])
Q2. What is the use of the axis parameter in Pandas Concat, and how is it used?
A2. The axis parameter is used to specify the axis along which the concat operation will be performed. For vertical concatenation, axis = 0 is used, and for horizontal concatenation, axis = 1 is used. Here is an example:
import pandas as pd
df1 = pd.DataFrame({'A': [1,2,3], 'B': [4,5,6]})
df2 = pd.DataFrame({'C': [7,8,9], 'D': [10,11,12]})
df3 = pd.concat([df1, df2], axis=1)
Q3. What is the use of the join parameter in Pandas Concat, and how is it used?
A3. The join parameter is used to specify how to handle columns that are not present in both data frames. It can take the values "inner" (use only common columns) or "outer" (use all columns). Here is an example:
import pandas as pd
df1 = pd.DataFrame({'A': [1,2,3], 'B': [4,5,6]})
df2 = pd.DataFrame({'C': [7,8,9], 'D': [10,11,12]})
df3 = pd.concat([df1, df2], axis=1, join='inner')
Q4. What is the use of the keys parameter in Pandas Concat, and how is it used?
A4. The keys parameter is used to give a hierarchical index to the concatenated data frames. It can take a list of strings to name the input data frames. Here is an example:
import pandas as pd
df1 = pd.DataFrame({'A': [1,2,3], 'B': [4,5,6]})
df2 = pd.DataFrame({'A': [7,8,9], 'B': [10,11,12]})
df3 = pd.concat([df1, df2], keys=['df1', 'df2'])
Q5. What happens if the indexes of the input data frames overlap during the Pandas Concat operation?
A5. If the indexes of the input data frames overlap during the Pandas Concat operation, it may cause issues. The stacked data frames will have their indexes preserved in the final result, which can cause problems if the indexes overlap. To avoid this, we can use the ignore_index parameter, which will reset the index of the resulting data frame. Here is an example:
import pandas as pd
df1 = pd.DataFrame({'A': [1,2,3], 'B': [4,5,6]})
df2 = pd.DataFrame({'A': [7,8,9], 'B': [10,11,12]})
df3 = pd.concat([df1, df2], ignore_index=True)
Tag
"PandasConcat"
