Skip to content

Commit

Permalink
Merge branch 'main' into provide_err_msg_empty_file_location
Browse files Browse the repository at this point in the history
  • Loading branch information
iulusoy authored Jun 27, 2023
2 parents 3afdf25 + 187d380 commit 9f8d3b7
Show file tree
Hide file tree
Showing 25 changed files with 1,334 additions and 251 deletions.
38 changes: 32 additions & 6 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,16 +27,42 @@ jobs:
run: |
python -m pip install --upgrade pip
python -m pip install -e .
- name: Run pytest linux (linux-only)
if: matrix.os == 'ubuntu-22.04'
- name: Run pytest test_colors
run: |
cd ammico
python -m pytest -m "not gcv and not long" -svv --cov=. --cov-report=xml
- name: Run pytest windows(windows-only)
if: matrix.os == 'windows-latest'
python -m pytest test/test_colors.py -svv --cov=. --cov-report=xml --cov-append
- name: Run pytest test_cropposts
run: |
cd ammico
python -m pytest -m "not gcv and not long and not win_skip" -svv --cov=. --cov-report=xml
python -m pytest test/test_cropposts.py -svv --cov=. --cov-report=xml --cov-append
- name: Run pytest test_display
run: |
cd ammico
python -m pytest test/test_display.py -svv --cov=. --cov-report=xml --cov-append
- name: Run pytest test_faces
run: |
cd ammico
python -m pytest test/test_faces.py -svv --cov=. --cov-report=xml --cov-append
- name: Run pytest test_multimodal_search
run: |
cd ammico
python -m pytest test/test_multimodal_search.py -m "not long" -svv --cov=. --cov-report=xml --cov-append
- name: Run pytest test_objects
run: |
cd ammico
python -m pytest test/test_objects.py -svv --cov=. --cov-report=xml --cov-append
- name: Run pytest test_summary
run: |
cd ammico
python -m pytest test/test_summary.py -m "not long" -svv --cov=. --cov-report=xml --cov-append
- name: Run pytest test_text
run: |
cd ammico
python -m pytest test/test_text.py -m "not gcv" -svv --cov=. --cov-report=xml --cov-append
- name: Run pytest test_utils
run: |
cd ammico
python -m pytest test/test_utils.py -svv --cov=. --cov-report=xml --cov-append
- name: Upload coverage
if: matrix.os == 'ubuntu-22.04' && matrix.python-version == '3.9'
uses: codecov/codecov-action@v3
Expand Down
22 changes: 14 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ Use pre-processed image files such as social media posts with comments and proce

The `AMMICO` package can be installed using pip:
```
pip install git+https://github.com/ssciwr/ammico.git
pip install ammico
```
This will install the package and its dependencies locally.

Expand All @@ -45,19 +45,21 @@ This will install the package and its dependencies locally.

There are sample notebooks in the `notebooks` folder for you to explore the package:
1. Text extraction: Use the notebook `get-text-from-image.ipynb` to extract any text from the images. The text is directly translated into English. If the text should be further analysed, set the keyword `analyse_text` to `True` as demonstrated in the notebook.\
**You can run this notebook on google colab: [Here](https://colab.research.google.com/github/ssciwr/ammico/blob/main/notebooks/get-text-from-image.ipynb)**
**You can run this notebook on google colab: [Here](https://colab.research.google.com/github/ssciwr/ammico/blob/main/ammico/notebooks/get-text-from-image.ipynb)**
Place the data files and google cloud vision API key in your google drive to access the data.
1. Emotion recognition: Use the notebook `facial_expressions.ipynb` to identify if there are faces on the image, if they are wearing masks, and if they are not wearing masks also the race, gender and dominant emotion.
**You can run this notebook on google colab: [Here](https://colab.research.google.com/github/ssciwr/ammico/blob/main/notebooks/facial_expressions.ipynb)**
**You can run this notebook on google colab: [Here](https://colab.research.google.com/github/ssciwr/ammico/blob/main/ammico/notebooks/facial_expressions.ipynb)**
Place the data files in your google drive to access the data.
1. Content extraction: Use the notebook `image_summary.ipynb` to create captions for the images and ask questions about the image content.
**You can run this notebook on google colab: [Here](https://colab.research.google.com/github/ssciwr/ammico/blob/main/notebooks/image_summary.ipynb)**
**You can run this notebook on google colab: [Here](https://colab.research.google.com/github/ssciwr/ammico/blob/main/ammico/notebooks/image_summary.ipynb)**
1. Multimodal content: Use the notebook `multimodal_search.ipynb` to find the best fitting images to an image or text query.
**You can run this notebook on google colab: [Here](https://colab.research.google.com/github/ssciwr/ammico/blob/main/notebooks/multimodal_search.ipynb)**
**You can run this notebook on google colab: [Here](https://colab.research.google.com/github/ssciwr/ammico/blob/main/ammico/notebooks/multimodal_search.ipynb)**
1. Color analysis: Use the notebook `color_analysis.ipynb` to identify colors the image. The colors are then classified into the main named colors in the English language.
**You can run this notebook on google colab: [Here](https://colab.research.google.com/github/ssciwr/ammico/blob/main/ammico/notebooks/colors_analysis.ipynb)**
1. Object analysis: Use the notebook `ojects_expression.ipynb` to identify certain objects in the image. Currently, the following objects are being identified: person, bicycle, car, motorcycle, airplane, bus, train, truck, boat, traffic light, cell phone.
**You can run this notebook on google colab: [Here](https://colab.research.google.com/github/ssciwr/ammico/blob/main/notebooks/objects_expression.ipynb)**

There are further notebooks that are currently of exploratory nature (`colors_expression.ipynb` to identify certain colors on the image). To crop social media posts use the `cropposts.ipynb` notebook.
**You can run this notebook on google colab: [Here](https://colab.research.google.com/github/ssciwr/ammico/blob/main/ammico/notebooks/objects_expression.ipynb)**
1. To crop social media posts use the `cropposts.ipynb` notebook.
**You can run this notebook on google colab: [Here](https://colab.research.google.com/github/ssciwr/ammico/blob/main/ammico/notebooks/cropposts.ipynb)**

## Features
### Text extraction
Expand Down Expand Up @@ -86,6 +88,10 @@ Emotion recognition is carried out using the [deepface](https://github.com/seren

Object detection is carried out using [cvlib](https://github.com/arunponnusamy/cvlib) and the [YOLOv4](https://github.com/AlexeyAB/darknet) model. This library detects faces, people, and several inanimate objects; we currently have restricted the output to person, bicycle, car, motorcycle, airplane, bus, train, truck, boat, traffic light, cell phone.

### Color/hue detection

Color detection is carried out using [colorgram.py](https://github.com/obskyr/colorgram.py) and [colour](https://github.com/vaab/colour) for the distance metric. The colors can be classified into the main named colors/hues in the English language, that are red, green, blue, yellow, cyan, orange, purple, pink, brown, grey, white, black.

### Cropping of posts

Social media posts can automatically be cropped to remove further comments on the page and restrict the textual content to the first comment only.
142 changes: 142 additions & 0 deletions ammico/colors.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
import numpy as np
import webcolors
import pandas as pd
from collections import defaultdict
import colorgram
import colour
from ammico.utils import get_color_table, AnalysisMethod

COLOR_SCHEMES = [
"CIE 1976",
"CIE 1994",
"CIE 2000",
"CMC",
"ITP",
"CAM02-LCD",
"CAM02-SCD",
"CAM02-UCS",
"CAM16-LCD",
"CAM16-SCD",
"CAM16-UCS",
"DIN99",
]


class ColorDetector(AnalysisMethod):
def __init__(
self,
subdict: dict,
delta_e_method: str = "CIE 1976",
) -> None:
"""Color Analysis class, analyse hue and identify named colors.
Args:
subdict (dict): The dictionary containing the image path.
delta_e_method (str): The calculation method used for assigning the
closest color name, defaults to "CIE 1976".
The available options are: 'CIE 1976', 'CIE 1994', 'CIE 2000',
'CMC', 'ITP', 'CAM02-LCD', 'CAM02-SCD', 'CAM02-UCS', 'CAM16-LCD',
'CAM16-SCD', 'CAM16-UCS', 'DIN99'
"""
super().__init__(subdict)
self.subdict.update(self.set_keys())
self.merge_color = True
self.n_colors = 100
if delta_e_method not in COLOR_SCHEMES:
raise ValueError(
"Invalid selection for assigning the color name. Please select one of {}".format(
COLOR_SCHEMES
)
)
self.delta_e_method = delta_e_method

def set_keys(self) -> dict:
colors = {
"red": 0,
"green": 0,
"blue": 0,
"yellow": 0,
"cyan": 0,
"orange": 0,
"purple": 0,
"pink": 0,
"brown": 0,
"grey": 0,
"white": 0,
"black": 0,
}
return colors

def analyse_image(self):
"""
Uses the colorgram library to extract the n most common colors from the images.
One problem is, that the most common colors are taken before beeing categorized,
so for small values it might occur that the ten most common colors are shades of grey,
while other colors are present but will be ignored. Because of this n_colors=100 was chosen as default.
The colors are then matched to the closest color in the CSS3 color list using the delta-e metric.
They are then merged into one data frame.
The colors can be reduced to a smaller list of colors using the get_color_table function.
These colors are: "red", "green", "blue", "yellow","cyan", "orange", "purple", "pink", "brown", "grey", "white", "black".
Returns:
dict: Dictionary with color names as keys and percentage of color in image as values.
"""
filename = self.subdict["filename"]

colors = colorgram.extract(filename, self.n_colors)
for color in colors:
rgb_name = self.rgb2name(
color.rgb,
merge_color=self.merge_color,
delta_e_method=self.delta_e_method,
)
self.subdict[rgb_name] += round(color.proportion, 2)

return self.subdict

def rgb2name(
self, c, merge_color: bool = True, delta_e_method: str = "CIE 1976"
) -> str:
"""Take an rgb color as input and return the closest color name from the CSS3 color list.
Args:
c (Union[List,tuple]): RGB value.
merge_color (bool, Optional): Whether color name should be reduced, defaults to True.
Returns:
str: Closest matching color name.
"""
if len(c) != 3:
raise ValueError("Input color must be a list or tuple of length 3 (RGB).")

h_color = "#{:02x}{:02x}{:02x}".format(int(c[0]), int(c[1]), int(c[2]))
try:
output_color = webcolors.hex_to_name(h_color, spec="css3")
output_color = output_color.lower().replace("grey", "gray")
except ValueError:
delta_e_lst = []
filtered_colors = webcolors.CSS3_NAMES_TO_HEX

for _, img_hex in filtered_colors.items():
cur_clr = webcolors.hex_to_rgb(img_hex)
# calculate color Delta-E
delta_e = colour.delta_E(c, cur_clr, method=delta_e_method)
delta_e_lst.append(delta_e)
# find lowest delta-e
min_diff = np.argsort(delta_e_lst)[0]
output_color = (
str(list(filtered_colors.items())[min_diff][0])
.lower()
.replace("grey", "gray")
)

# match color to reduced list:
if merge_color:
for reduced_key, reduced_color_sub_list in get_color_table().items():
if str(output_color).lower() in [
str(color_name).lower()
for color_name in reduced_color_sub_list["ColorName"]
]:
output_color = reduced_key.lower()
break
return output_color
24 changes: 24 additions & 0 deletions ammico/data/Color_tables.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
Pink;Pink;purple;purple;red;red;orange;orange;yellow;yellow;green;green;cyan;cyan;blue;blue;brown;brown;white;white;grey;grey;black;black
ColorName;HEX;ColorName;HEX;ColorName;HEX;ColorName;HEX;ColorName;HEX;ColorName;HEX;ColorName;HEX;ColorName;HEX;ColorName;HEX;ColorName;HEX;ColorName;HEX;ColorName;HEX
Pink;#FFC0CB;Lavender;#E6E6FA;LightSalmon;#FFA07A;Orange;#FFA500;Gold;#FFD700;GreenYellow;#ADFF2F;Aqua;#00FFFF;CadetBlue;#5F9EA0;Cornsilk;#FFF8DC;White;#FFFFFF;Gainsboro;#DCDCDC;Black;#000000
LightPink;#FFB6C1;Thistle;#D8BFD8;Salmon;#FA8072;DarkOrange;#FF8C00;Yellow;#FFFF00;Chartreuse;#7FFF00;Cyan;#00FFFF;SteelBlue;#4682B4;BlanchedAlmond;#FFEBCD;Snow;#FFFAFA;LightGray;#D3D3D3;;
HotPink;#FF69B4;Plum;#DDA0DD;DarkSalmon;#E9967A;Coral;#FF7F50;LightYellow;#FFFFE0;LawnGreen;#7CFC00;LightCyan;#E0FFFF;LightSteelBlue;#B0C4DE;Bisque;#FFE4C4;HoneyDew;#F0FFF0;Silver;#C0C0C0;;
DeepPink;#FF1493;Orchid;#DA70D6;LightCoral;#F08080;Tomato;#FF6347;LemonChiffon;#FFFACD;Lime;#00FF00;PaleTurquoise;#AFEEEE;LightBlue;#ADD8E6;NavajoWhite;#FFDEAD;MintCream;#F5FFFA;DarkGray;#A9A9A9;;
PaleVioletRed;#DB7093;Violet;#EE82EE;IndianRed;#CD5C5C;OrangeRed;#FF4500;LightGoldenRodYellow;#FAFAD2;LimeGreen;#32CD32;Aquamarine;#7FFFD4;PowderBlue;#B0E0E6;Wheat;#F5DEB3;Azure;#F0FFFF;DimGray;#696969;;
MediumVioletRed;#C71585;Fuchsia;#FF00FF;Crimson;#DC143C;;;PapayaWhip;#FFEFD5;PaleGreen;#98FB98;Turquoise;#40E0D0;LightSkyBlue;#87CEFA;BurlyWood;#DEB887;AliceBlue;#F0F8FF;Gray;#808080;;
;;Magenta;#FF00FF;Red;#FF0000;;;Moccasin;#FFE4B5;LightGreen;#90EE90;MediumTurquoise;#48D1CC;SkyBlue;#87CEEB;Tan;#D2B48C;GhostWhite;#F8F8FF;LightSlateGray;#778899;;
;;MediumOrchid;#BA55D3;FireBrick;#B22222;;;PeachPuff;#FFDAB9;MediumSpringGreen;#00FA9A;DarkTurquoise;#00CED1;CornflowerBlue;#6495ED;RosyBrown;#BC8F8F;WhiteSmoke;#F5F5F5;SlateGray;#708090;;
;;DarkOrchid;#9932CC;DarkRed;#8B0000;;;PaleGoldenRod;#EEE8AA;SpringGreen;#00FF7F;;;DeepSkyBlue;#00BFFF;SandyBrown;#F4A460;SeaShell;#FFF5EE;DarkSlateGray;#2F4F4F;;
;;DarkViolet;#9400D3;;;;;Khaki;#F0E68C;MediumSeaGreen;#3CB371;;;DodgerBlue;#1E90FF;GoldenRod;#DAA520;Beige;#F5F5DC;;;;
;;BlueViolet;#8A2BE2;;;;;DarkKhaki;#BDB76B;SeaGreen;#2E8B57;;;RoyalBlue;#4169E1;DarkGoldenRod;#B8860B;OldLace;#FDF5E6;;;;
;;DarkMagenta;#8B008B;;;;;;;ForestGreen;#228B22;;;Blue;#0000FF;Peru;#CD853F;FloralWhite;#FFFAF0;;;;
;;Purple;#800080;;;;;;;Green;#008000;;;MediumBlue;#0000CD;Chocolate;#D2691E;Ivory;#FFFFF0;;;;
;;MediumPurple;#9370DB;;;;;;;DarkGreen;#006400;;;DarkBlue;#00008B;Olive;#808000;AntiqueWhite;#FAEBD7;;;;
;;MediumSlateBlue;#7B68EE;;;;;;;YellowGreen;#9ACD32;;;Navy;#000080;SaddleBrown;#8B4513;Linen;#FAF0E6;;;;
;;SlateBlue;#6A5ACD;;;;;;;OliveDrab;#6B8E23;;;MidnightBlue;#191970;Sienna;#A0522D;LavenderBlush;#FFF0F5;;;;
;;DarkSlateBlue;#483D8B;;;;;;;DarkOliveGreen;#556B2F;;;;;Brown;#A52A2A;MistyRose;#FFE4E1;;;;
;;RebeccaPurple;#663399;;;;;;;MediumAquaMarine;#66CDAA;;;;;Maroon;#800000;;;;;;
;;Indigo;#4B0082;;;;;;;DarkSeaGreen;#8FBC8F;;;;;;;;;;;;
;;;;;;;;;;LightSeaGreen;#20B2AA;;;;;;;;;;;;
;;;;;;;;;;DarkCyan;#008B8B;;;;;;;;;;;;
;;;;;;;;;;Teal;#008080;;;;;;;;;;;;
Loading

0 comments on commit 9f8d3b7

Please sign in to comment.