python - Measuring average cosine similarity between the groups

Question

I have the following data frame:

Group        Vector
1            [1 1 0 1 0 0]
1            [1 0 0 1 0 0]
1            [1 0 0 1 1 1]
1            [0 0 0 1 0 1]
2            [0 0 0 1 0 1]
2            [0 0 0 1 0 1]
2            [0 1 1 1 0 1]
2            [1 1 0 0 0 1]

How could I calculate the average cosine similarity within the groups? This is the expected outcome (Note I make up to numbers for the calculation)

Group        Vector            Average_Similarity
1            [1 1 0 1 0 0]      0.34
1            [1 0 0 1 0 0]      0.34
1            [1 0 0 1 1 1]      0.34
1            [0 0 0 1 0 1]      0.34
2            [0 0 0 1 0 1]      0.48
2            [0 0 0 1 0 1]      0.48
2            [0 1 1 1 0 1]      0.48
2            [1 1 0 0 0 1]      0.48

question from:https://stackoverflow.com/questions/65908200/measuring-average-cosine-similarity-between-the-groups

Answer 1

Suppose we read data from your example like:

from ast import literal_eval
df = pd.read_clipboard(sep="|", converters = {"Vector":literal_eval})
df
   Group              Vector
0      1  [1, 1, 0, 1, 0, 0]
1      1  [1, 0, 0, 1, 0, 0]
2      1  [1, 0, 0, 1, 1, 1]
3      1  [0, 0, 0, 1, 0, 1]
4      2  [0, 0, 0, 1, 0, 1]
5      2  [0, 0, 0, 1, 0, 1]
6      2  [0, 1, 1, 1, 0, 1]
7      2  [1, 1, 0, 0, 0, 1]

Then try:

from scipy.spatial.distance import pdist

df["Average_Similarity"] = df.groupby("Group")["Vector"].transform(
    lambda group: pdist(group.to_list(), metric="cosine").mean()
)
df

---

   Group              Vector  Average_Similarity
0      1  [1, 1, 0, 1, 0, 0]            0.380615
1      1  [1, 0, 0, 1, 0, 0]            0.380615
2      1  [1, 0, 0, 1, 1, 1]            0.380615
3      1  [0, 0, 0, 1, 0, 1]            0.380615
4      2  [0, 0, 0, 1, 0, 1]            0.365323
5      2  [0, 0, 0, 1, 0, 1]            0.365323
6      2  [0, 1, 1, 1, 0, 1]            0.365323
7      2  [1, 1, 0, 0, 0, 1]            0.365323