These are some python code snippets that I use very often.
Topics include: list manipulation, json manipulation, data frame and etc.
Contents
Python List Manipulation
Concatenate two python lists
|
1 2 3 4 5 6 |
listone = [1,2,3] listtwo = [4,5,6] joinedlist = listone + listtwo |
Convert a python string to a list of characters
|
1 2 3 4 |
word = 'abc' the_list = list(word) |
JSON Manipulation
Convert a dictionary to a json string
|
1 2 3 4 5 6 |
import json r = {'expert': 'True', 'rating': 1.5} r = json.dumps(r) |
Convert a json string back to a python dictionary
|
1 2 3 4 5 |
import json my_dict = json.loads(json_string) |
Load a json file into a pandas data frame
|
1 2 3 4 5 6 |
import pandas as pd #this assumes one json item per line in json file df=pd.read_json("path_to_json_file", lines=True) |
Pandas DataFrame Manipulation
Group by a column and keep the column afterwards
|
1 2 3 |
df.groupby(['column_name']).aggregate_function().reset_index() |
Convert a data frame to a list of dictionary values
Let’s say you want a list of dictionaries from a pandas data frame as follows:
From this:
To this:
|
1 2 3 |
[{"name":"rita","age":23}, {"name":"gita","age":45}] |
This is the code you would use:
|
1 2 3 |
dict_vals=df.to_dict(orient='records') |
Convert a dictionary to a pandas data frame
Let’s say you have a dict as follows:
my_dict={'mrr':0.4,'map':0.3,'precision':0.6}.
To convert this to a pandas Data Frame, you can do the following:
|
1 2 3 4 5 6 |
import pandas as pd my_dict={'mrr':0.4,'map':0.3,'precision':0.6} pd.DataFrame(list(my_dict.items()),columns=['metric','value']) |
You will see the following output:
|
1 2 3 4 5 6 |
metric value 0 mrr 0.4 1 map 0.3 2 precision 0.6 |
Select rows matching a specific column criteria
Let’s say you want to find rows where the column value matches a specific constraint. You could use the following:
|
1 2 3 4 |
import pandas as pd df=df[(df['column1']<=2) & df['column2']<==3) ] |
Create a new data frame column with specific values
Let’s say you want to add an additional column to a data frame with values generated via some external processing. You can transform the external values into a list and do the following:
|
1 2 3 4 5 |
vals=[1,2,3,4] df['vals']=vals |
Sort data frame by value
|
1 2 3 4 |
# Sort in descending order df.sort_values(by=["column_name1","column_name2"],ascending=False) |
Get unique values from a data frame column
|
1 2 3 4 |
# Get unique list of values in the df['column_name'] column uniq_vals=list(df.column_name.unique()) |
Create a new derived column in your data frame
The goal here is to create a new column with values populated based on the values of an old column. Let’s say you want a new column that adds 1 to a value from an old column.
|
1 2 3 4 5 6 7 8 |
# Generate a new value from `an_existing_column` # generate_a_value(x) is a python function that generates a value # based on the column value from `an_existing_column` df['my_new_column'] = df['an_existing_column'].apply(lambda x: generate_a_value(x)) |
Select /display specific columns from a data frame
|
1 2 3 4 |
# select specific columns in a data frame df= df[['my_col1','my_col2']] |
System Commands
Run a system command from within Python code
|
1 2 3 4 5 |
import os os.system("your_system_command") |
File / Directory Operations
Safely create nested directories in Python
|
1 2 3 4 5 6 7 |
import os #check if path exists, if not create the directory if not os.path.exists(directory): os.makedirs(directory) |
Evaluation
Compute per-class precision, recall, f1 scores
The goal here is to compute per-class precision, recall and f1 scores and display the results using a data frame.
The first step is to collect your labels as two separate lists. (1) the predicted labels and (2) the corresponding true labels. For example:
|
1 2 3 4 |
predicted_labels=["positive","positive","negative"] true_labels=["positive","other","negative"] |
Once you have the true and predicted labels in a list, you can use sklearn’s precision_recall_fscore_support module to compute all the scores for you. Here’s how you do it:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# the possible labels labels = ['positive', 'negative', 'other'] # setting average to None, returns precision, recall and f1 scores for individual labels per_class_prf=precision_recall_fscore_support(true_labels,predicted_labels,average=None,labels=labels) # Collect all the p/r/f labels for the 3 classes precisions = per_class_prf[0] recalls = per_class_prf[1] fscores = per_class_prf[2] supports = per_class_prf[3] # Zip the values to make rows for each label table_data = zip(labels, precisions, recalls, fscores, supports) # Place the zipped values in a dataframe df = pd.DataFrame(list(table_data),columns=['labels', 'precision', 'recall', 'f-score', 'num_of_examples']) print(df.sort_values(by=['num_of_examples'], ascending=False)) |
Example output:
|
1 2 3 4 5 6 |
labels precision recall f-score num_of_examples 1 negative 0.875000 0.933333 0.903226 15 0 positive 0.636364 0.777778 0.700000 9 2 other 0.500000 0.200000 0.285714 5 |