Merge pull request #18 from tendai-zw/master

maltego-trx 1.3.8: Added overlay and genealogy functionality, automatic translation of legacy property names

README.md

@@ -18,6 +18,9 @@ maltego-trx start new_project
 
 This will create a folder new_project with the recommend project structure.
 
+Alternatively, you can copy either the `gunicorn` or `apache` example projects from the `demo` directory. 
+These also include Dockerfile and corresponding docker-compose configuration files for production deployment.
+
 **Adding a Transform:**
 
 Add a new transform by creating a new python file in the "transforms" folder of your directory.
@@ -201,6 +204,25 @@ The following constants can be imported from `maltego_trx.maltego`.
 - `LINK_STYLE_DOTTED`
 - `LINK_STYLE_DASHDOT`
 
+### Enums
+
+**Overlays:**
+
+Overlays Enums are imported from `maltego_trx.overlays`
+
+*Overlay OverlayPosition:*
+- `NORTH = "N"`
+- `SOUTH = "S"`
+- `WEST = "W"`
+- `NORTH_WEST = "NW"`
+- `SOUTH_WEST = "SW"`
+- `CENTER = "C"`
+
+*Overlay Type*
+- `IMAGE = "image"`
+- `COLOUR = "colour"`
+- `TEXT = "text"`
+
 ### Request/MaltegoMsg
 
 The request/maltego msg object given to the transform contains the information about the input entity.
@@ -213,18 +235,28 @@ The request/maltego msg object given to the transform contains the information a
 - `Type: str`: The input entity type
 - `Properties: dict(str: str)`: A key-value dictionary of the input entity properties
 - `TransformSettings: dict(str: str)`: A key-value dictionary of the transform settings
+- `Genealogy: list(dict(str: str))`: A key-value dictionary of the Entity genealogy, 
+    this is only applicable for extended entities e.g. Website Entity
 
 **Methods:**
 
-- `getProperty(name: str)`: get a property value of the input entity
-- `getTransformSetting(name: str)`: get a transform setting value
+- `getProperty(name: str)`: Get a property value of the input entity
+- `getTransformSetting(name: str)`: Get a transform setting value
+- `clearLegacyProperties()`: Delete (duplicate) legacy properties from the input entity. This will not result in 
+property information being lost, it will simply clear out some properties that the TRX library duplicates on all 
+incoming Transform requests. In older versions of TRX, these Entity properties would have a different internal ID when 
+sent the server than what the Maltego client would advertise in the Entity Manager UI. For a list of Entities with such 
+properties and their corresponding legacy and actual IDs, see `entity_property_map` in `maltego_trx/entities.py`. For 
+the majority of projects this distinction can be safely ignored.  
 
 ### Response/MaltegoTransform
 
 **Methods:**
 
-- `addEntity(type: str, value: str) -> Entity`: Add an entity to the transform response. Returns an Entity object created by the method.
-- `addUIMessage(message: str, messageType='Inform')`: Return a UI message to the user. For message type, use a message type constant.
+- `addEntity(type: str, value: str) -> Entity`: Add an entity to the transform response. Returns an Entity object 
+created by the method.
+- `addUIMessage(message: str, messageType='Inform')`: Return a UI message to the user. For message type, use a message 
+type constant.
 
 ### Entity
 
@@ -234,10 +266,51 @@ The request/maltego msg object given to the transform contains the information a
 - `setValue(value: str)`: Set the entity value
 - `setWeight(weight: int)`: Set the entity weight
 - `addDisplayInformation(content: str, title: str)`: Add display information for the entity.
-- `addProperty(fieldName: str, displayName: str, matchingRule: str, value: str)`: Add a property to the entity. Matching rule can be `strict` or `loose`.
+- `addProperty(fieldName: str, displayName: str, matchingRule: str, value: str)`: Add a property to the entity. 
+Matching rule can be `strict` or `loose`.
+- `addOverlay(propertyName: str, position: OverlayPosition, overlay_type: OverlayType)`: Add an overlay to the entity. 
+`OverlayPosition` and `OverlayType` are defined in the `maltego_tx.overlays`
+
+Overlay can be added as Text, Image or Color
+
+```python 
+
+        person_name = request.Value
+        entity = response.addEntity(Phrase, "Hi %s, nice to meet you!" % person_name)
+
+        # Normally, when we create an overlay, we would reference a property name so that Maltego can then use the
+        # value of that property to create the overlay. Sometimes that means creating a dynamic property, but usually
+        # it's better to either use an existing property, or, if you created the Entity yourself, and only need the
+        # property for the overlay, to use a hidden property. Here's an example of using a dynamic property:
+        entity.addProperty(
+            'dynamic_overlay_icon_name', 
+            displayName="Name for overlay image", 
+            value="Champion"  # references an icon in the Maltego client
+        )
+        entity.addOverlay('dynamic_overlay_icon_name', OverlayPosition.WEST, OverlayType.IMAGE)
+
+        # DISCOURAGED:
+        # You *can* also directly supply the string value of the property, however this is not recommended. Why? If
+        # the entity already has a property of the same ID (in this case, "DE"), then you would in fact be assigning the
+        # value of that property, not the string "DE", which is not the intention. Nevertheless, here's an example:
+        entity.addOverlay(
+            'DE', # name of an icon, however, could also accidentally be a property name
+            OverlayPosition.SOUTH_WEST, 
+            OverlayType.IMAGE
+        )
+
+        # Overlays can also be used to display extra text on an entity:
+        entity.addProperty("exampleDynamicPropertyName", "Example Dynamic Property", "loose", "Maltego Overlay Testing")
+        entity.addOverlay('exampleDynamicPropertyName', OverlayPosition.NORTH, OverlayType.TEXT)
+
+        # Or a small color indicator:
+        entity.addOverlay('#45e06f', OverlayPosition.NORTH_WEST, OverlayType.COLOUR)
+```
+
 - `setIconURL(url: str)`: Set the entity icon URL
 - `setBookmark(bookmark: int)`: Set bookmark color index (e.g. -1 for BOOKMARK_COLOR_NONE, 3 for BOOKMARK_COLOR_PURPLE)
 - `setNote(note: str)`: Set note content
+- `setGenealogy(genealogy: dict)`: Set genealogy
 
 **Link Methods:**
 

demo/apache/transforms/OverlayExample.py

@@ -0,0 +1,36 @@
+from maltego_trx.entities import Phrase
+from maltego_trx.overlays import OverlayPosition, OverlayType
+
+from maltego_trx.transform import DiscoverableTransform
+
+
+class OverlayExample(DiscoverableTransform):
+    """
+    Returns a phrase with overlays on the graph.
+    """
+
+    @classmethod
+    def create_entities(cls, request, response):
+        person_name = request.Value
+        entity = response.addEntity(Phrase, "Hi %s, nice to meet you!" % person_name)
+
+        # Normally, when we create an overlay, we would reference a property name so that Maltego can then use the
+        # value of that property to create the overlay. Sometimes that means creating a dynamic property, but usually
+        # it's better to either use an existing property, or, if you created the Entity yourself, and only need the
+        # property for the overlay, to use a hidden property. Here's an example of using a dynamic property:
+        entity.addProperty('dynamic_overlay_icon_name', displayName="Name for overlay image", value="Champion")
+        entity.addOverlay('dynamic_overlay_icon_name', OverlayPosition.WEST, OverlayType.IMAGE)
+
+        # DISCOURAGED:
+        # You *can* also directly supply the string value of the property, however this is not recommended. Why? If
+        # the entity already has a property of the same ID (in this case, "DE"), then you would in fact be assigning the
+        # value of that property, not the string "DE", which is not the intention. Nevertheless, here's an example:
+        entity.addOverlay('DE', OverlayPosition.SOUTH_WEST, OverlayType.IMAGE)
+
+        # Overlays can also be used to display extra text on an entity:
+        entity.addProperty("exampleDynamicPropertyName", "Example Dynamic Property", "loose", "Maltego Overlay Testing")
+        entity.addOverlay('exampleDynamicPropertyName', OverlayPosition.NORTH, OverlayType.TEXT)
+
+        # Or a small color indicator:
+        entity.addOverlay('#45e06f', OverlayPosition.NORTH_WEST, OverlayType.COLOUR)
+

demo/gunicorn/Dockerfile

@@ -5,7 +5,7 @@ RUN mkdir /var/www/TRX/
 WORKDIR /var/www/TRX/
 
 # System dependencies
-RUN apt-get update
+RUN apt-get update -y
 RUN apt-get install python3-pip -y
 
 COPY requirements.txt requirements.txt
@@ -18,4 +18,5 @@ COPY . /var/www/TRX/
 
 RUN chown -R www-data:www-data /var/www/TRX/
 
+# for running a production server, use docker-compose with prod.yml or prod-ssl.yml
 CMD ["python3", "project.py", "runserver"]

demo/gunicorn/prod-ssl.yml

@@ -0,0 +1,7 @@
+version: '3'
+services:
+  python:
+    build: .
+    command: "gunicorn --certfile=server.crt --keyfile=server.key --bind=0.0.0.0:8443 --threads=25 --workers=2 project:app"
+    ports:
+       - "8443:8443"

demo/gunicorn/prod.yml

@@ -1,7 +1,7 @@
 version: '3'
 services:
   python:
-    build: ..
+    build: .
     command: "gunicorn --bind=0.0.0.0:8080 --threads=25 --workers=2 project:app"
     ports:
        - "8080:8080"

demo/gunicorn/requirements.txt

@@ -1 +1 @@
-maltego-trx
+maltego-trx>=1.3.8

demo/gunicorn/transforms/OverlayExample.py

@@ -0,0 +1,36 @@
+from maltego_trx.entities import Phrase
+from maltego_trx.overlays import OverlayPosition, OverlayType
+
+from maltego_trx.transform import DiscoverableTransform
+
+
+class OverlayExample(DiscoverableTransform):
+    """
+    Returns a phrase with overlays on the graph.
+    """
+
+    @classmethod
+    def create_entities(cls, request, response):
+        person_name = request.Value
+        entity = response.addEntity(Phrase, "Hi %s, nice to meet you!" % person_name)
+
+        # Normally, when we create an overlay, we would reference a property name so that Maltego can then use the
+        # value of that property to create the overlay. Sometimes that means creating a dynamic property, but usually
+        # it's better to either use an existing property, or, if you created the Entity yourself, and only need the
+        # property for the overlay, to use a hidden property. Here's an example of using a dynamic property:
+        entity.addProperty('dynamic_overlay_icon_name', displayName="Name for overlay image", value="Champion")
+        entity.addOverlay('dynamic_overlay_icon_name', OverlayPosition.WEST, OverlayType.IMAGE)
+
+        # DISCOURAGED:
+        # You *can* also directly supply the string value of the property, however this is not recommended. Why? If
+        # the entity already has a property of the same ID (in this case, "DE"), then you would in fact be assigning the
+        # value of that property, not the string "DE", which is not the intention. Nevertheless, here's an example:
+        entity.addOverlay('DE', OverlayPosition.SOUTH_WEST, OverlayType.IMAGE)
+
+        # Overlays can also be used to display extra text on an entity:
+        entity.addProperty("exampleDynamicPropertyName", "Example Dynamic Property", "loose", "Maltego Overlay Testing")
+        entity.addOverlay('exampleDynamicPropertyName', OverlayPosition.NORTH, OverlayType.TEXT)
+
+        # Or a small color indicator:
+        entity.addOverlay('#45e06f', OverlayPosition.NORTH_WEST, OverlayType.COLOUR)
+

maltego_trx/__init__.py

@@ -1 +1 @@
-VERSION = "1.3.7"
+VERSION = "1.3.8"

maltego_trx/entities.py

@@ -35,3 +35,38 @@
 URL = "maltego.URL"
 Website = "maltego.Website"
 WebTitle = "maltego.WebTitle"
+
+# {entityName: {version2PropertyName: version3PropertyName,...}}
+entity_property_map = {
+    "maltego.Person": {"firstname": "person.firstnames", "lastname": "person.lastname"},
+    "maltego.Domain": {"whois": "whois-info"},
+    "maltego.IPv4Address": {"whois": "whois-info"},
+    "maltego.URL": {"maltego.v2.value.property": "short-title", "theurl": "url", "fulltitle": "title"},
+    "maltego.Document": {"maltego.v2.value.property": "title", "link": "url", "metainfo": "document.meta-data"},
+    "maltego.Location": {"area": "location.area", "countrysc": "url", "long": "longitude", "lat": "latitude"},
+    "maltego.PhoneNumber": {
+        "countrycode": "phonenumber.countrycode", "citycode": "phonenumber.citycode",
+        "areacode": "phonenumber.areacode", "lastnumbers": "phonenumber.lastnumbers"
+    },
+    "maltego.affiliation.Spock": {
+        "network": "affiliation.network", "uid": "affiliation.uid", "profile_url": "affiliation.profile-url",
+        "spock_websites": "spock.websites"
+    },
+    "maltego.affiliation": {
+        "network": "affiliation.network", "uid": "affiliation.uid", "profile_url": "affiliation.profile-url"
+    },
+    "maltego.Service": {"banner": "banner.text", "port": "port.number"},
+    "maltego.Alias": {"properties.alias": "alias"},
+    "maltego.Device": {"properties.device": "device"},
+    "maltego.GPS": {"properties.gps": "gps.coordinate"},
+    "maltego.CircularArea": {"area": "radius"},
+    "maltego.Image": {"properties.image": "description", "fullImage": "url"},
+    "maltego.NominatimLocation": {"properties.nominatimlocation": "nominatimlocation"},
+    "maltego.BuiltWithTechnology": {"properties.builtwithtechnology": "builtwith.technology"},
+    "maltego.FacebookObject": {"properties.facebookobject": "facebook.object"}
+}
+
+
+def translate_legacy_property_name(entity_type, v2_property):
+    """Function maps a legacy version 2 entity property name to version 3 entity property name"""
+    return entity_property_map .get(entity_type, {}).get(v2_property)

maltego_trx/maltego.py

@@ -1,7 +1,9 @@
 import uuid;
+
 from xml.dom import minidom
 
-from .entities import Phrase
+from .entities import Phrase, translate_legacy_property_name, entity_property_map
+from .overlays import OverlayPosition, OverlayType
 from .utils import remove_invalid_xml_chars
 
 BOOKMARK_COLOR_NONE = "-1"
@@ -44,6 +46,7 @@
 ADD_FIELD_TEMPLATE = "<Field MatchingRule=\"%(matching)s\" Name=\"%(name)s\" DisplayName=\"%(display)s\"><![CDATA[%(value)s]]></Field>"
 DISP_INFO_TEMPLATE = "<Label Name=\"%(name)s\" Type=\"text/html\"><![CDATA[' %(content)s ']]></Label>"
 UIM_TEMPLATE = "<UIMessage MessageType=\"%(type)s\">%(text)s</UIMessage>"
+OVERLAY_TEMPLATE = "<Overlay position=\"%(position)s\" propertyName=\"%(property_name)s\" type=\"%(type)s\"/>"
 
 
 class MaltegoEntity(object):
@@ -55,6 +58,7 @@ def __init__(self, type=None, value=None):
         self.additionalFields = []
         self.displayInformation = []
         self.iconURL = ""
+        self.overlays = []
 
     def setType(self, type=None):
         if type:
@@ -93,7 +97,7 @@ def setLinkLabel(self, label):
 
     def reverseLink(self):
         self.addProperty('link#maltego.link.direction', 'link#maltego.link.direction', 'loose', 'output-to-input')
-        
+
     def addCustomLinkProperty(self, fieldName=None, displayName=None, value=None):
         self.addProperty('link#' + fieldName, displayName, '', value)
 
@@ -103,6 +107,11 @@ def setBookmark(self, bookmark):
     def setNote(self, note):
         self.addProperty('notes#', 'Notes', '', note)
 
+    def addOverlay(
+            self, propertyName, position: OverlayPosition, overlayType: OverlayType
+    ):
+        self.overlays.append([propertyName, position.value, overlayType.value])
+
     def add_field_to_xml(self, additional_field):
         name, display, matching, value = additional_field
         matching = "strict" if matching.lower().strip() == "strict" else "loose"
@@ -139,6 +148,16 @@ def returnEntity(self):
                 lines.append(self.add_field_to_xml(additional_field))
             lines.append("</AdditionalFields>")
 
+        if self.overlays:
+            lines.append("<Overlays>")
+            for overlay in self.overlays:
+                overlay_tag = OVERLAY_TEMPLATE % {
+                    "property_name": overlay[0],
+                    "position": overlay[1],
+                    "type": overlay[2],
+                }
+                lines.append(overlay_tag)
+            lines.append("</Overlays>")
         if self.iconURL:
             lines.append("<IconURL>%s</IconURL>" % self.iconURL)
 
@@ -226,6 +245,15 @@ def __init__(self, MaltegoXML="", LocalArgs=[]):
 
             self.Weight = self._get_int(entity, "Weight")
             self.Slider = self._get_int(maltego_msg, "Limits", attr_name="SoftLimit")
+            self.Genealogy = []
+            genealogy_tag = maltego_msg.getElementsByTagName("Genealogy")
+            genealogy_types = genealogy_tag[0].getElementsByTagName("Type") if genealogy_tag else []
+            for genealogy_type_tag in genealogy_types:
+                entity_type_name = genealogy_type_tag.getAttribute("Name")
+                entity_type_old_name = genealogy_type_tag.getAttribute("OldName")
+                entity_type = {"Name": entity_type_name,
+                               "OldName": entity_type_old_name if entity_type_old_name else None}
+                self.Genealogy.append(entity_type)
 
             # Additional Fields
             self.Properties = {}
@@ -235,6 +263,10 @@ def __init__(self, MaltegoXML="", LocalArgs=[]):
                 name = field.getAttribute("Name")
                 value = self._get_text(field)
                 self.Properties[name] = value
+                for entity_type in self.Genealogy:
+                    v3_property_name = translate_legacy_property_name(entity_type["Name"], name)
+                    if v3_property_name is not None:
+                        self.Properties[v3_property_name] = value
 
             # Transform Settings
             self.TransformSettings = {}
@@ -247,6 +279,7 @@ def __init__(self, MaltegoXML="", LocalArgs=[]):
         elif LocalArgs:
             self.Value = LocalArgs[0]
             self.Type = "local.Unknown"
+            self.Genealogy = None
 
             self.Weight = 100
             self.Slider = 100
@@ -263,6 +296,16 @@ def __init__(self, MaltegoXML="", LocalArgs=[]):
                 self.buildProperties(text.split("#"), hash_rnd, equals_rnd, bslash_rnd)
                 self.TransformSettings = {}
 
+    def clearLegacyProperties(self):
+        to_clear = set()
+        for entity_type in self.Genealogy or []:
+            for prop_name in entity_property_map.get(entity_type["Name"], []):
+                to_clear.add(prop_name)
+
+        for field_name in to_clear:
+            if field_name in self.Properties:
+                del self.Properties[field_name]
+
     def buildProperties(self, key_value_array, hash_rnd, equals_rnd, bslash_rnd):
         self.Properties = {}
         for property_section in key_value_array: