Consensus

The consensus tool allows you to compare the instances of an item to all the corresponding instances of that item. It scores each instance and you can filter out the ones that have low scores. SuperAnnotate currently supports computing the consensus of the following instance types in Vector Projects: Bounding Box, Polygon, and Point.

13531353

How does it work?

In a project, there are several folders, all containing the same images. Each folder is assigned to an annotator. When the annotation of all the images is complete, you can run the consensus for the entire project.

Calculate the consensus of Bounding Boxes and Polygons

To calculate the consensus of Bounding Boxes and Polygons:

  1. Match all the corresponding instances across all images and folders.
  2. Compute the intersection over union (IoU) score between the current instance and each of the matched instances.
  3. Average those computed scores.

Calculate the consensus of Points

To calculate the consensus of Points:

  1. Match all the corresponding instances across all images and folders.
  2. Check if the class of the instance is the same as each of the matched instances.
  3. The score is the number of matching instances that have the same class as the current instance divided by the number of matching instances to the current instance.

SDK functions

To compute the consensus scores among different folders:

consensus_df = sa.consensus(
  project = "Project Name"
  folder_names = ["Team 1", "Team 2", "Team 3"],
  annot_type = 'bbox')

The following function call exports the listed folders and computes the consensus scores between matching images. If the data is already exported, provide export_root with:

consensus_df = sa.consensus(
  project = "Project Name"
  folder_names =  ["Team 1", "Team 2", "Team 3"],
  export_root = "./exports",
  annot_type = "bbox")

Note that consensus scores are computed for all matching images between specific folders.
If the computation is relevant for a specific list of images, provide image_list with:

consensus_df = sa.consensus(
  project = "Project Name"
  folder_names =  ["Team 1", "Team 2", "Team 3"],
  export_root = "./exports"
  image_list = ["image1.png", "image10.png", "image100.png"],
  annot_type = "bbox")

The returned DataFrame lists all the annotated instances with their corresponding metadata and consensus scores:

instanceId

className

folderName

score

creatorEmail

imageName

0

car

Team 1

0.815302

[email protected]

berlin1.png

0

car

Team 2

0.822203

[email protected]

berlin1.png

0

car

Team 3

0.773299

[email protected]

berlin1.png

The show_plots parameter returns the following analytics plots:

consensus_df = sa.consensus(
  project = "Project Name"
  folder_names =  ["Team 1", "Team 2", "Team 3"],
  export_root = "./exports"
  image_list = ["image1.png", "image10.png", "image100.png"],
  annot_type = "bbox", 
  show_plots = True)

This is the box plot of consensus scores of each annotator:

700700

The scatter plot of consensus score vs. the instance area of each folder:

15201520

Did this page help you?