Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: metadata api #44

Closed
wants to merge 12 commits into from
4 changes: 2 additions & 2 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ jobs:
REDIS_HOST: ${{ secrets.REDIS_HOST }}
REDIS_PORT: 6379
REDIS_DB: 0
TOKEN_METADATA_BUCKET: token-metadata-testnet
METADATA_BUCKET: metadata-testnet
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should add this in the TODO as well.

Create new buckets, migrate token info from old bucket to the new one and delete old bucket when everything is working fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's still missing some
image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it covers all, but i'll be more specific

CORS_ALLOWED_ORIGIN: https://explorer.testnet.hathor.network
- name: Deploy Lambdas Mainnet
if: ${{ needs.init.outputs.environment == 'mainnet' }}
Expand All @@ -94,7 +94,7 @@ jobs:
REDIS_HOST: ${{ secrets.REDIS_HOST }}
REDIS_PORT: 6379
REDIS_DB: 0
TOKEN_METADATA_BUCKET: token-metadata-mainnet
METADATA_BUCKET: metadata-mainnet
CORS_ALLOWED_ORIGIN: https://explorer.hathor.network
- name: Deploy Daemons Testnet
if: ${{ needs.init.outputs.environment == 'testnet' }}
Expand Down
2 changes: 1 addition & 1 deletion common/configuration.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,6 @@ def default(cls) -> 'Environment':
REDIS_PORT = config('REDIS_PORT', default=None)
REDIS_DB = config('REDIS_DB', default='0', cast=int)

TOKEN_METADATA_BUCKET = config('TOKEN_METADATA_BUCKET', default=None)
METADATA_BUCKET = config('METADATA_BUCKET', default=None)

CORS_ALLOWED_ORIGIN = config('CORS_ALLOWED_ORIGIN', default=None)
Empty file added domain/metadata/__init__.py
Empty file.
32 changes: 32 additions & 0 deletions domain/metadata/metadata.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
from dataclasses import asdict, dataclass
from enum import Enum


class MetadataType(str, Enum):
TOKEN = 'TOKEN'
TRANSACTION = 'TRANSACTION'


@dataclass
class Metadata:
id: str

@property
def type(self) -> MetadataType:
raise NotImplementedError

@property
def data(self):
raise NotImplementedError

@classmethod
def from_dict(cls, dikt: dict) -> 'Metadata':
raise NotImplementedError

def to_dict(self) -> dict:
""" Convert a Metadata instance into dict

:return: Dict representations of Metadata
:rtype: dict
"""
return asdict(self)
92 changes: 92 additions & 0 deletions domain/metadata/token_metadata.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
from dataclasses import dataclass
from enum import Enum
from typing import Optional

from domain.metadata.metadata import Metadata, MetadataType


class TokenNFTMediaType(str, Enum):
VIDEO = 'VIDEO'
IMAGE = 'IMAGE'
AUDIO = 'AUDIO'


@dataclass
class TokenNFTMedia:
"""Data of token NFT

:param type: NFT Type. image or video.
:type type: :py:class:`domain.metadata.token_metdata.TokenNFTMediaType`

:param file: Media file of the NFT
:type file: str

:param loop: If media will play in loop or not. Works for audio and video as well
:type loop: Optional[bool]

:param autoplay: If media will play automatically or not. Works for audio and video as well
:type autoplay: Optional[bool]
"""
type: TokenNFTMediaType
file: str
loop: Optional[bool] = False
autoplay: Optional[bool] = False


@dataclass
class MetaToken:
"""Metadata of a token

:param id: Token unique id
:type id: str

:param verified: If token is verified or not. None and False are equivalent
:type verified: Optional[bool]

:param banned: If token is banned or not. None and False are equivalent
:type banned: Optional[bool]

:param reason: Bannishment reason
:type reason: Optional[str]

:param nft: If token is a nft or not. None and False are equivalent
:type nft: Optional[bool]

:param nft_media: NFT media data, if any.
:type nft_media: Optional[:py:class:`domain.metadata.token_metdata.TokenNFTMedia`]
"""
id: str
verified: Optional[bool] = False
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why Optional? What does None mean in this case? Add the description in the docstring, please.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to let the json have only the fields that matter. Docstring updated.

banned: Optional[bool] = False
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why Optional? What does None mean in this case? Add the description in the docstring, please.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to let the json have only the fields that matter. Docstring updated.

reason: Optional[str] = ''
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why Optional? What does None mean in this case? Add the description in the docstring, please.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This property only make sense for banned tokens

nft: Optional[bool] = False
nft_media: Optional[TokenNFTMedia] = None


@dataclass
class TokenMetadata(Metadata):
data: Optional[MetaToken] = None
type: MetadataType = MetadataType.TOKEN

@classmethod
def from_dict(cls, dikt: dict) -> 'TokenMetadata':
""" Creates a new TokenMetadata instance from a given dict (inverse operation of `to_dict`)

:param dikt: Dict with TokenMetadata structure and data
:type dikt: dict

:return: The new instance
:rtype: :py:class:`domain.metadata.token_metdata.TokenMetadata`
"""
data = dikt.get('data', {})
if data.get('nft_media'):
data['nft_media'] = TokenNFTMedia(
TokenNFTMediaType(data['nft_media']['type'].upper()),
data['nft_media']['file'],
data['nft_media'].get('loop', None)
)

return cls(
id=dikt['id'],
data=MetaToken(**data)
)
33 changes: 33 additions & 0 deletions domain/metadata/transaction_metadata.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
from dataclasses import dataclass
from typing import Optional

from domain.metadata.metadata import Metadata, MetadataType


@dataclass
class MetaTransaction:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we add any new property in transaction metadata we need to update the explorer service? Imagine we want to mark a tx as banned or as important (don't know why but just imagining possible situations). How do we handle this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, and i think that is the right thing to do. Otherwise we could broke things in explorer or have other unpredictable problems.

id: str
context: Optional[str]
genesis: Optional[bool]


@dataclass
class TransactionMetadata(Metadata):
data: Optional[MetaTransaction] = None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we using two classes? I feel it would be simper to use one class with all fields there. Well, we can group in subclasses for specific cases if there are too many fields.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have only specific cases by now. We can use Union[Meta1, Meta2] in places that will handle metadata but i think that is not scalable.

type: MetadataType = MetadataType.TRANSACTION

@classmethod
def from_dict(cls, dikt: dict) -> 'TransactionMetadata':
""" Creates a new TransactionMetadata instance from a given dict (inverse operation of `to_dict`)

:param dikt: Dict with TransactionMetadata structure and data
:type dikt: dict

:return: The new instance
:rtype: :py:class:`domain.metadata.token_metdata.TransactionMetadata`
"""

return cls(
id=dikt['id'],
data=MetaTransaction(**dikt['data'])
)
71 changes: 0 additions & 71 deletions domain/tx/token.py
Original file line number Diff line number Diff line change
@@ -1,79 +1,8 @@
from dataclasses import asdict, dataclass
from enum import Enum
from typing import Optional

from dacite import from_dict


class TokenNFTType(str, Enum):
VIDEO = 'VIDEO'
IMAGE = 'IMAGE'
AUDIO = 'AUDIO'


@dataclass
class TokenNFT:
"""Data of token NFT

:param type: NFT Type. image or video.
:type type: :py:class:`domain.tx.token.TokenNFTType`

:param file: Media file of the NFT
:type file: str
"""
type: TokenNFTType
file: str


@dataclass
class TokenMetadata:
"""Metadata of token

:param id: Token unique id
:type id: str

:param verified: If token is verified or not
:type verified: bool

:param banned: If token is banned or not
:type banned: bool

:param reason: Bannishment reason
:type reason: str

:param nft: NFT data, if any.
:type nft: :py:class:`domain.tx.token.TokenNFT`
"""
id: str
verified: bool
banned: bool
reason: Optional[str]
nft: Optional[TokenNFT]

def to_dict(self) -> dict:
""" Convert a TokenMetadata instance into dict

:return: Dict representations of TokenMetadata
:rtype: dict
"""
return asdict(self)

@classmethod
def from_dict(cls, dikt: dict) -> 'TokenMetadata':
""" Creates a new TokenMetadata instance from a given dict (inverse operation of `to_dict`)

:param dikt: Dict with TokenMetadata structure and data
:type dikt: dict

:return: The new instance
:rtype: :py:class:`domain.tx.token.TokenMetadata`
"""
if dikt.get('nft'):
dikt['nft'] = TokenNFT(TokenNFTType(dikt['nft']['type'].upper()), dikt['nft']['file'])

return TokenMetadata(**dikt)


@dataclass
class Token:
"""Hathor Token
Expand Down
80 changes: 80 additions & 0 deletions gateways/metadata_gateway.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
import json
from typing import Optional

from common.configuration import METADATA_BUCKET
from domain.metadata.metadata import MetadataType
from domain.metadata.token_metadata import TokenMetadata
from domain.metadata.transaction_metadata import TransactionMetadata
from gateways.clients.s3_client import S3Client

TYPE_TO_FOLDER = {
MetadataType.TRANSACTION: 'transaction',
MetadataType.TOKEN: 'token'
}


class MetadataGateway:

def __init__(self, s3_client: Optional[S3Client] = None) -> None:
self.s3_client = s3_client or S3Client()

def get_transaction_metadata(self, id: str) -> Optional[TransactionMetadata]:
"""Retrieve transaction metadata from a json file stored in s3

:param id: transaction id
:type id: str
:raises Exception: The name of the bucket used to store the jsons must be on config
:return: TransactionMetadata object or None if not found
:rtype: TokenMetadata | None
"""
folder = TYPE_TO_FOLDER[MetadataType.TRANSACTION]
transaction_metadata = self._get_metadata(f"{folder}/{id}.json")

if transaction_metadata is None:
return None

return TransactionMetadata.from_dict(transaction_metadata)

def get_token_metadata(self, id: str) -> Optional[TokenMetadata]:
"""Retrieve token metadata from a json file stored in s3

:param id: token id
:type id: str

:return: TokenMetadata object or None if not found
:rtype: TokenMetadata | None
"""
folder = TYPE_TO_FOLDER[MetadataType.TOKEN]
token_metadata = self._get_metadata(f"{folder}/{id}.json")

if token_metadata is None:
return None

return TokenMetadata.from_dict(token_metadata)

def _get_metadata(self, s3_object_name: str) -> Optional[dict]:
"""Retrive metadata from file in s3

:param s3_object_name: name of object
:type s3_object_name: str

:raises Exception: The name of the bucket used to store the jsons must be on config

:return: file content as dict
:rtype: Optional[dict]
"""
metadata_bucket = self._metadata_bucket()

raw_metadata = self.s3_client.load_file(metadata_bucket, s3_object_name)
if raw_metadata is None:
return None

return json.loads(raw_metadata)

def _metadata_bucket(self) -> str:
metadata_bucket = METADATA_BUCKET

if metadata_bucket is None:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't it better to validate all configuration in the constructor? So we can assume everything is right when we get to this point?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so as we can have methods that don't need this constant.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All methods need this constant for now, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pedroferreira1 yes, but i still feel it's not good to check in constructor. We don't have this var in test env, for instance and i don't think we must to. Of course for some tests we have to mock it, but some tests just don't care about it and i think it doesn't make sense to mock a var only for the class to not break.

I know it's just test but i think it's a good indication why maybe it's not so good to check this on constructor.

raise Exception('No bucket name in config')

return metadata_bucket
37 changes: 4 additions & 33 deletions gateways/token_gateway.py
Original file line number Diff line number Diff line change
@@ -1,48 +1,19 @@
import json
from typing import Union

from common.configuration import TOKEN_METADATA_BUCKET
from domain.tx.token import Token, TokenMetadata
from domain.tx.token import Token
from gateways.clients.hathor_core_client import TOKEN_ENDPOINT, HathorCoreClient
from gateways.clients.responses.hathor_core.hathor_core_token_response import HathorCoreTokenResponse
from gateways.clients.s3_client import S3Client


class TokenGateway:
"""Gateway for Token

:param s3_client: Client for s3 manipulation, default to domain S3Client
:type s3_client:
:param hathor_core_client: Client for make hathor-core requests, default to domain HathorCoreClient
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HathorCoreClient?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. It is the client to make requests for a full-node. get_token method uses it therefore will be changed soon.

:type hathor_core_client:
"""
def __init__(
self,
s3_client: Union[S3Client, None] = None,
hathor_core_client: Union[HathorCoreClient, None] = None
) -> None:
self.s3_client = s3_client or S3Client()
def __init__(self, hathor_core_client: Union[HathorCoreClient, None] = None) -> None:
self.hathor_core_client = hathor_core_client or HathorCoreClient()

def get_token_metadata_from_s3(self, file: str) -> Union[TokenMetadata, None]:
"""Retrieve token metadata from a json file stored in s3

:param file: file name
:type file: str
:raises Exception: The name of the bucket used to store the jsons must be on config
:return: TokenMetadata object
:rtype: TokenMetadata
"""
token_metadata_bucket = TOKEN_METADATA_BUCKET

if token_metadata_bucket is None:
raise Exception('No bucket name in config')

token_raw_metadata = self.s3_client.load_file(token_metadata_bucket, file)
if token_raw_metadata is None:
return None

token_metadata = json.loads(token_raw_metadata)
return TokenMetadata.from_dict(token_metadata)

def get_token(self, id: str) -> Union[Token, None]:
response = self.hathor_core_client.get(TOKEN_ENDPOINT, {'id': id})

Expand Down
Loading