Learning Objectives – after this chapter you will be able to:
- Create DataFrames from dictionaries and lists.
- Select columns, rows and cells with
[],locandiloc. - Filter rows with conditions.
- Add, modify, drop and sort columns and rows, and summarise with
groupby().
Creating a DataFrame
A DataFrame is a 2-D labelled table. The most common way to create one is from a dictionary of lists: keys become column names and lists become column values.
import pandas as pd
data = {
"Name": ["Asha", "Ravi", "Meera", "John", "Sara"],
"Course": ["BCA", "BBA", "BCA", "MCA", "BBA"],
"Age": [20, 21, 19, 23, 20],
"Marks": [82, 67, 91, 74, 58],
}
df = pd.DataFrame(data)
print(df)
print(df.shape)
print(df.columns.tolist())Output
Name Course Age Marks 0 Asha BCA 20 82 1 Ravi BBA 21 67 2 Meera BCA 19 91 3 John MCA 23 74 4 Sara BBA 20 58 (5, 4) ['Name', 'Course', 'Age', 'Marks']
From a list of rows
import pandas as pd rows = [["Pen", 10, 50], ["Book", 250, 4], ["Bag", 700, 1]] items = pd.DataFrame(rows, columns=["Item", "Price", "Qty"]) print(items)
Output
Item Price Qty 0 Pen 10 50 1 Book 250 4 2 Bag 700 1
Selecting Columns
import pandas as pd
df = pd.DataFrame({"Name": ["Asha", "Ravi", "Meera"], "Age": [20, 21, 19], "Marks": [82, 67, 91]})
print(df["Marks"].tolist()) # one column -> Series
print(df[["Name", "Marks"]]) # several columns -> DataFrameOutput
[82, 67, 91]
Name Marks
0 Asha 82
1 Ravi 67
2 Meera 91Selecting Rows and Cells: loc and iloc
| Expression | Selects |
|---|---|
df.loc[label] | Row with that index label |
df.iloc[pos] | Row at that integer position |
df.loc[rows, cols] | Rows and columns by label (slice ends included) |
df.iloc[rows, cols] | Rows and columns by position (slice ends excluded) |
import pandas as pd
df = pd.DataFrame({"Name": ["Asha", "Ravi", "Meera", "John"],
"Age": [20, 21, 19, 23],
"Marks": [82, 67, 91, 74]})
print(df.loc[2, "Name"], df.iloc[0, 2])
print(df.loc[1:2, ["Name", "Marks"]]) # rows 1 to 2 (included)
print(df.iloc[:2, :2]) # first 2 rows, first 2 columns
df2 = df.set_index("Name") # use names as row labels
print(df2.loc["John", "Marks"])Output
Meera 82
Name Marks
1 Ravi 67
2 Meera 91
Name Age
0 Asha 20
1 Ravi 21
74Filtering Rows
Use a boolean condition on a column. Combine conditions with & and |, each in parentheses.
import pandas as pd
df = pd.DataFrame({"Name": ["Asha", "Ravi", "Meera", "John", "Sara"],
"Course": ["BCA", "BBA", "BCA", "MCA", "BBA"],
"Marks": [82, 67, 91, 74, 58]})
print(df[df["Marks"] >= 75])
print(df[(df["Course"] == "BBA") & (df["Marks"] > 60)])
print(df[df["Course"].isin(["MCA", "BBA"])]["Name"].tolist())Output
Name Course Marks 0 Asha BCA 82 2 Meera BCA 91 Name Course Marks 1 Ravi BBA 67 ['Ravi', 'John', 'Sara']
Adding, Modifying and Dropping
import pandas as pd
df = pd.DataFrame({"Name": ["Asha", "Ravi", "Meera"], "Marks": [82, 67, 91]})
df["Bonus"] = df["Marks"] + 5 # new calculated column
df["Result"] = ["Pass" if m >= 70 else "Retry" for m in df["Marks"]]
df.loc[1, "Marks"] = 72 # change one cell
print(df)
df = df.drop(columns=["Bonus"]) # remove a column
df = df.drop(index=0) # remove a row
print(df)Output
Name Marks Bonus Result
0 Asha 82 87 Pass
1 Ravi 72 72 Retry
2 Meera 91 96 Pass
Name Marks Result
1 Ravi 72 Retry
2 Meera 91 Pass Most DataFrame methods (
drop, sort_values, rename …) return a new DataFrame. Assign the result (df = df.drop(...)) to keep the change.Sorting and Grouping
import pandas as pd
df = pd.DataFrame({"Name": ["Asha", "Ravi", "Meera", "John", "Sara"],
"Course": ["BCA", "BBA", "BCA", "MCA", "BBA"],
"Marks": [82, 67, 91, 74, 58]})
print(df.sort_values("Marks", ascending=False).head(3))
print(df.groupby("Course")["Marks"].mean())Output
Name Course Marks 2 Meera BCA 91 0 Asha BCA 82 3 John MCA 74 Course BBA 62.5 BCA 86.5 MCA 74.0 Name: Marks, dtype: float64
Key Takeaways
df["col"]gives a Series;df[["a", "b"]]gives a DataFrame.loc= labels (ends included),iloc= positions (ends excluded).- Filter with boolean conditions; most methods return a new DataFrame;
groupby()summarises by category.
Video : Pandas DataFrame
Question / Answer (15 questions)
Answer : A DataFrame is a two-dimensional, labelled data structure in pandas, made of rows and columns like a spreadsheet or SQL table. Each column is a Series and columns may have different data types.
Answer : Pass a dictionary of lists to pd.DataFrame(): the keys become column names and the lists become column values, e.g. pd.DataFrame({"Name": ["Asha", "Ravi"], "Age": [20, 21]}). All lists must have the same length.
Answer : Pass a list of lists and the column names: pd.DataFrame(rows, columns=["Item", "Price", "Qty"]).
Answer : df["Marks"] (a single name) returns a Series. df[["Marks"]] (a list of names) returns a DataFrame with one column.
Answer : shape returns (rows, columns). columns returns the column labels, which can be converted to a list with df.columns.tolist().
Answer : loc selects rows and columns by labels and includes the end label in slices. iloc selects by integer positions and excludes the end position in slices.
Answer : df.loc[2, "Name"]. Using positions, df.iloc[2, 0] would give the same cell if Name is the first column.
Answer : set_index("Name") returns a new DataFrame in which the Name column is used as the row index (labels), allowing access such as df.loc["John", "Marks"].
Answer : Index the DataFrame with a boolean condition on a column: df[df["Marks"] >= 75]. Combine conditions with & and |, each condition in parentheses.
Answer : It checks whether each value is in a given list, e.g. df[df["Course"].isin(["MCA", "BBA"])] keeps rows whose course is MCA or BBA.
Answer : Assign to a new column name: df["Percent"] = df["Marks"] / 500 * 100. The values are calculated for every row at once.
Answer : df.drop(columns=["Percent"]) removes a column and df.drop(index=0) removes the row with label 0. Both return a new DataFrame, so assign the result.
Answer : These methods return a new DataFrame and do not modify the original. You must assign the result, e.g. df = df.drop(columns=["x"]).
Answer : df.sort_values("Marks", ascending=False).
Answer : It splits the rows into groups by course, takes the Marks column in each group and returns the average marks for every course.
MCQ Test : Check Your Understanding (5 questions)
Select an option and press Check Answer, or answer all questions and press Submit Test to see your score. Each answer comes with an explanation.
Bibliography/References
The notes, questions and MCQs on this page are original explanations written for this course. The works listed below were consulted and are cited for attribution and further reading. Product names, logos and trademarks belong to their respective owners. Short quotations, where used, are marked and attributed.
- pandas development team. (n.d.). Intro to data structures. pandas User Guide. https://pandas.pydata.org/docs/user_guide/dsintro.html
- pandas development team. (n.d.). Indexing and selecting data. pandas User Guide. https://pandas.pydata.org/docs/user_guide/indexing.html
- pandas development team. (n.d.). 10 minutes to pandas. pandas User Guide. https://pandas.pydata.org/docs/user_guide/10min.html
- pandas development team. (n.d.). Getting started tutorials. pandas Documentation. https://pandas.pydata.org/docs/getting_started/intro_tutorials/index.html
- McKinney, W. (2022). Python for Data Analysis (3rd ed.). O’Reilly Media. (Open-access online edition.) https://wesmckinney.com/book/