Improve include filtering #40

keiranjprice101 · 2019-08-15T13:54:41Z

closes #32 closes #24

I made an error when staging these changes so the commits got mixed up, but it's small enough to just look at files changed instead.

Include filtering now allows any arbitrary combination of includes. However still only one filter per request (#39)

Examples:
http://localhost:5000/datafiles?where={"ID":3}&include={"DATASET":"INVESTIGATION"}
http://localhost:5000/datafiles?where={"ID":3}&include=["DATAFILEFORMAT", {"DATASET":{"INVESTIGATION":["INVESTIGATIONTYPE","FACILITY"]}}]
http://localhost:5000/datafiles?where={"ID":3}&include=["DATASET","DATAFILEFORMAT"]
http://localhost:5000/datafiles?where={"ID":3}&include=["DATASET"]

e.g

[
    {
        "ID": "3",
        "CHECKSUM": "None",
        "CREATE_ID": "simple/root",
        "CREATE_TIME": "2019-05-21 11:24:47",
        "DATAFILECREATETIME": "0000-00-00 00:00:00",
        "DATAFILEMODTIME": "0000-00-00 00:00:00",
        "DESCRIPTION": "ut fugit est",
        "DOI": "None",
        "FILESIZE": "846400",
        "LOCATION": "aliquid/aspernatur/eum",
        "MOD_ID": "simple/root",
        "MOD_TIME": "2019-05-21 11:24:47",
        "NAME": "Datafile fdfdo",
        "DATAFILEFORMAT_ID": "1",
        "DATASET_ID": "1",
        "DATAFILEFORMAT": {
            "ID": "1",
            "CREATE_ID": "simple/root",
            "CREATE_TIME": "2019-05-21 11:24:44",
            "DESCRIPTION": "Uploads by the Topcat's users",
            "MOD_ID": "simple/root",
            "MOD_TIME": "2019-05-21 11:24:44",
            "NAME": "upload",
            "TYPE": "misc",
            "VERSION": "1",
            "FACILITY_ID": "1"
        },
        "DATASET": {
            "ID": "1",
            "COMPLETE": "0",
            "CREATE_ID": "simple/root",
            "CREATE_TIME": "2019-05-21 11:24:46",
            "DESCRIPTION": "Test",
            "DOI": "None",
            "END_DATE": "None",
            "LOCATION": "None",
            "MOD_ID": "simple/root",
            "MOD_TIME": "2019-05-21 11:24:46",
            "NAME": "Dataset 1",
            "STARTDATE": "None",
            "INVESTIGATION_ID": "1",
            "SAMPLE_ID": "None",
            "TYPE_ID": "20",
            "INVESTIGATION": {
                "ID": "1",
                "CREATE_ID": "simple/root",
                "CREATE_TIME": "2019-05-21 11:24:46",
                "DOI": "None",
                "ENDDATE": "2017-07-21 12:01:54",
                "MOD_ID": "simple/root",
                "MOD_TIME": "2019-05-21 11:24:46",
                "NAME": "Proposal 1",
                "RELEASEDATE": "None",
                "STARTDATE": "2017-07-13 12:01:54",
                "SUMMARY": "None",
                "TITLE": "quas accusantium omnis",
                "VISIT_ID": "Proposal 1 - 1",
                "FACILITY_ID": "1",
                "TYPE_ID": "16",
                "INVESTIGATIONTYPE": {
                    "ID": "16",
                    "CREATE_ID": "simple/root",
                    "CREATE_TIME": "2019-05-21 11:24:43",
                    "DESCRIPTION": "incidunt enim impedit",
                    "MOD_ID": "simple/root",
                    "MOD_TIME": "2019-05-21 11:24:43",
                    "NAME": "InvestigationType 5",
                    "FACILITY_ID": "1"
                },
                "FACILITY": {
                    "ID": "1",
                    "CREATE_ID": "simple/root",
                    "CREATE_TIME": "2019-05-21 11:24:41",
                    "DAYSUNTILRELEASE": "None",
                    "DESCRIPTION": "None",
                    "FULLNAME": "Lorum Ipsum Light Source",
                    "MOD_ID": "simple/root",
                    "MOD_TIME": "2019-05-21 11:24:41",
                    "NAME": "LILS",
                    "URL": "None"
                }
            }
        }
    }
]

All of these return properly and this PR will make #38 easier once merged in

louise-davies · 2019-08-16T12:06:48Z

Trying to perform http://127.0.0.1:5000/investigations?include={"INVESTIGATIONINSTRUMENT":"INSTRUMENT"}, I get the following error:

Traceback (most recent call last):
  File "C:\Users\mnf98541\anaconda3\lib\site-packages\flask\app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\Users\mnf98541\anaconda3\lib\site-packages\flask\app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "C:\Users\mnf98541\anaconda3\lib\site-packages\flask_restful\__init__.py", line 269, in error_router
    return original_handler(e)
  File "C:\Users\mnf98541\anaconda3\lib\site-packages\flask\app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Users\mnf98541\anaconda3\lib\site-packages\flask\_compat.py", line 34, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\mnf98541\anaconda3\lib\site-packages\flask\app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\Users\mnf98541\anaconda3\lib\site-packages\flask\app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "C:\Users\mnf98541\anaconda3\lib\site-packages\flask_restful\__init__.py", line 269, in error_router
    return original_handler(e)
  File "C:\Users\mnf98541\anaconda3\lib\site-packages\flask\app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Users\mnf98541\anaconda3\lib\site-packages\flask\_compat.py", line 34, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\mnf98541\anaconda3\lib\site-packages\flask\app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\Users\mnf98541\anaconda3\lib\site-packages\flask\app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "C:\Users\mnf98541\anaconda3\lib\site-packages\flask_restful\__init__.py", line 458, in wrapper
    resp = resource(*args, **kwargs)
  File "C:\Users\mnf98541\anaconda3\lib\site-packages\flask\views.py", line 88, in view
    return self.dispatch_request(*args, **kwargs)
  File "C:\Users\mnf98541\anaconda3\lib\site-packages\flask_restful\__init__.py", line 573, in dispatch_request
    resp = meth(*args, **kwargs)
  File "C:\code\datagateway-api\common\helpers.py", line 36, in wrapper_requires_session
    return method(*args, **kwargs)
  File "C:\code\datagateway-api\common\helpers.py", line 57, in wrapper_gets_records
    return method(*args, **kwargs)
  File "C:\code\datagateway-api\src\resources\entities\investigations_endpoints.py", line 16, in get
    return get_rows_by_filter(INVESTIGATION, get_filters_from_query_string()), 200
  File "C:\code\datagateway-api\common\database_helpers.py", line 311, in get_rows_by_filter
    return list(map(lambda x: x.to_nested_dict(query_filter["include"]), results))
  File "C:\code\datagateway-api\common\database_helpers.py", line 311, in <lambda>
    return list(map(lambda x: x.to_nested_dict(query_filter["include"]), results))
  File "C:\code\datagateway-api\common\models\db_models.py", line 40, in to_nested_dict
    dictionary[related_entity.__tablename__] = related_entity.to_nested_dict(include[list(include)[0]])
AttributeError: 'InstrumentedList' object has no attribute 'to_nested_dict'

I realise that this is probably due to this going through a many-to-many relationship rather than a one-to-one or one-to-many/many-to-one, but this is an example of a query that needs to be able to be done.

keiranjprice101 · 2019-08-16T12:33:26Z

@louise-davies Is that request meant to return the same as /investigations?include=["INVESTIGATIONINSTRUMENT", "INSTRUMENT"]
e.g

[
    {
        "ID": "1",
        "CREATE_ID": "simple/root",
        "CREATE_TIME": "2019-05-21 11:24:46",
        "MOD_ID": "simple/root",
        "MOD_TIME": "2019-05-21 11:24:46",
        "INSTRUMENT_ID": "11",
        "INVESTIGATION_ID": "1",
        "INVESTIGATION": {
            "ID": "1",
            "CREATE_ID": "simple/root",
            "CREATE_TIME": "2019-05-21 11:24:46",
            "DOI": "None",
            "ENDDATE": "2017-07-21 12:01:54",
            "MOD_ID": "simple/root",
            "MOD_TIME": "2019-05-21 11:24:46",
            "NAME": "Proposal 1",
            "RELEASEDATE": "None",
            "STARTDATE": "2017-07-13 12:01:54",
            "SUMMARY": "None",
            "TITLE": "quas accusantium omnis",
            "VISIT_ID": "Proposal 1 - 1",
            "FACILITY_ID": "1",
            "TYPE_ID": "16"
        },
        "INSTRUMENT": {
            "ID": "11",
            "CREATE_ID": "simple/root",
            "CREATE_TIME": "2019-05-21 11:24:41",
            "DESCRIPTION": "None",
            "FULLNAME": "Instrument 19",
            "MOD_ID": "simple/root",
            "MOD_TIME": "2019-05-21 11:24:41",
            "NAME": "I19",
            "TYPE": "None",
            "URL": "None",
            "FACILITY_ID": "1"
        }
    },

Did that request work previously on master?

louise-davies · 2019-08-16T12:48:15Z

I haven't played about with the include filters that much so I don't know if it worked on master - my thoughts are probably not, and /investigations?include=["INVESTIGATIONINSTRUMENT", "INSTRUMENT"] still produces the error for me

If you mean /investigationinstruments?include=["INVESTIGATION", "INSTRUMENT"], this does return the information, but that means I am applying my filters to the InvestigationInstrument object rather than the Investigation object which I am actually querying for

keiranjprice101 · 2019-08-16T12:53:21Z

@louise-davies Ok, I will have a look. Ah I posted the wrong endpoint I meant /investigationinstruments?include=["INVESTIGATION", "INSTRUMENT"]

That is what gives the result I posted previously. Which nests investigation and instrument into InvestigationInstrument

louise-davies · 2019-08-16T12:57:02Z

@keiranjprice101 yeah that does work, but that means that I either don't get the Investigation information in a way I can filter it, or I have to fetch Investigations and then send more individual requests to find out the instruments.

Another problem is that users can currently sort/filter by Instrument in both Diamond and ISIS - which means we need to be able to sort/filter on an included entity...

keiranjprice101 · 2019-08-16T13:08:31Z

@louise-davies Ok. I think this should be ok but, filtering on includes is very different to normal filtering due to the way the Filters are defined in their classes. I've had to extend filtering to get the first endpoint in #34 working and I believe it would be a similar process. I will open a separate issue for this.

louise-davies · 2019-08-16T13:41:18Z

@keiranjprice101 yeah, feel free to split them up into separate issues. Are you going to try and get many-to-many includes working in this PR or in a separate issue? aka should I approve this as is or should I wait?

keiranjprice101 · 2019-08-16T13:46:29Z

@louise-davies If everything else is fine with this, then you could approve it. It seems to me that, #42, many to many includes, #38, #39 and possibly #41 are all very very similar in what changes/code they will need. So I think I will have to put a lot of thought into changes that will play nicely between these issues. So I think this PR would start as a good base at least. Unless of course I am missing something obvious and these issues aren't as complicated as I believe.

keiranjprice101 added 2 commits August 15, 2019 14:48

#32: Add get_related_entity

6e8c8c1

#32: Change to_nested_dict

cdd1133

keiranjprice101 requested a review from louise-davies August 15, 2019 13:54

#32: User ternary

b60e2d1

louise-davies approved these changes Aug 16, 2019

View reviewed changes

keiranjprice101 merged commit 60c668d into master Aug 16, 2019

keiranjprice101 deleted the 32_improve_include_filtering branch August 16, 2019 13:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve include filtering #40

Improve include filtering #40

keiranjprice101 commented Aug 15, 2019 •

edited

Loading

louise-davies commented Aug 16, 2019

keiranjprice101 commented Aug 16, 2019 •

edited

Loading

louise-davies commented Aug 16, 2019 •

edited

Loading

keiranjprice101 commented Aug 16, 2019 •

edited

Loading

louise-davies commented Aug 16, 2019

keiranjprice101 commented Aug 16, 2019

louise-davies commented Aug 16, 2019

keiranjprice101 commented Aug 16, 2019 •

edited

Loading

Improve include filtering #40

Improve include filtering #40

Conversation

keiranjprice101 commented Aug 15, 2019 • edited Loading

louise-davies commented Aug 16, 2019

keiranjprice101 commented Aug 16, 2019 • edited Loading

louise-davies commented Aug 16, 2019 • edited Loading

keiranjprice101 commented Aug 16, 2019 • edited Loading

louise-davies commented Aug 16, 2019

keiranjprice101 commented Aug 16, 2019

louise-davies commented Aug 16, 2019

keiranjprice101 commented Aug 16, 2019 • edited Loading

keiranjprice101 commented Aug 15, 2019 •

edited

Loading

keiranjprice101 commented Aug 16, 2019 •

edited

Loading

louise-davies commented Aug 16, 2019 •

edited

Loading

keiranjprice101 commented Aug 16, 2019 •

edited

Loading

keiranjprice101 commented Aug 16, 2019 •

edited

Loading