MDS Agency and Provider Unification

[schnuerle]

Discussed in #644

^{Originally posted by schnuerle May 7, 2021}
During this week's MDS working group meeting, the idea of merging Agency and Provider together came up. The idea is that maybe it's time in the next major release (2.0) to 'merge' them somehow, but it's not clear how that would work, if it's a good idea, and what it would impact.

We decided to create this discussion area to allow people to share their thoughts, pros and cons on the idea.

Some points to consider:

1. API naming reconciliation work was done for MDS 1.0.0
2. There are more top level APIs that connect or impact both Agency and Provider
3. There is a proposal to make a version of trips in Agency
4. The General Information (common area) of MDS is growing
5. A difference is Agency is push and Provider is pull
6. A difference is who is hosting Agency feeds (agencies) and Provider feeds (providers)
7. Authentication and feed validity becomes simpler for cities with Agency since providers are responsible for sending data
8. Agency was meant for more real time, and Provider was meant for historic, though Provider does have RT endpoints now like events and vehicles. Does this distinction need to be made clear or is it less important now?

We welcome your thoughts on what is sure to be a long, detailed, and complex discussion!

See Unification Task Force for more details and how to help.

[schnuerle]

^{Originally posted by quicklywilliam Dec 2, 2021}
I brought this topic up again today in the context of 2.0 and New Modes in particular. The workgroup seemed to agree that the divergence between Agency and Provider is more a historical artifact than a design choice, but there is understandable hesitation to take on such a big task in the 2.0 timeframe. My concern is that when we are adding new modes, we'll be bringing (hopefully!) a lot of new implementations of MDS on both the city and operator side. And from our experience with MDS 1.0, it's going to be very hard to do this convergence work once those implementations are in production.

What I'd like to explore is whether there is an approach that allows us to have a unified framework for all of the different MDS endpoints without requiring new or existing implementations to adopt them all (ie forcing anyone to adopt agency or provider). I think we already have laid the groundwork for this with the MDS Requirements endpoint. Requirements gives cities the ability to say which parts of MDS they need. The only remaining work is to clarify and distinguish what the different parts of MDS are designed to do.

To kick off the discussion, here's a quick sketch of how we could begin to clarify this:

• The status endpoints are designed for report-driven regulation. Cities should use this when they need to periodically assess compliances, fees and performance of their program.

• The events (aka Agency) endpoints are designed for realtime-regulation (“air traffic control”). Cities should use this when they need to control the behavior of vehicles in realtime. Note that cities adopting events may also choose to use status as a useful fallback or validation mechanism. Cities might also choose to start more modestly with periodic regulation using status and aspire to realtime using events later. Hence, the two should be designed to work together.

• The vehicles endpoints are designed to give a quick, easy realtime view of the world. Cities should use this when they need a simple way to see what is going on right now. For example, vehicles can help cities respond to a complaint about an individual vehicle or diagnose a system-level problem. Cities may also use vehicles as a useful fallback or validation mechanism for status and/or events (ie to address the "drifting world view" problem).

• The trips endpoint isn't for regulation at all, but rather for the planning & analysis of route data. Here again, cities may also use trips as a useful fallback or validation mechanism for status and/or events.

@quicklywilliam

[schnuerle]

@marie-x who is the steering committee member leading the Unification Task Force has come up with a document proposing 4 different solutions to this.

Agency / Provider Unification in MDS 2.0 document

We will bring this to the public Working Group meeting tomorrow for discussion, so please comment here or attend and bring your thoughts.

[schnuerle]

For this to move forward we need some more feedback and direction, specifically from cities, providers, and software companies who are using both Provider and Agency now. New modes in MDS like taxis, ride hail, delivery robots, and car share may have more need for one of the options and should be considered and evaluated too.

[edecorbiere]

Hi @marie-x,
First of all thanks for all the work and propositions! Here is some feedback about the different options you drafted in the document. We can discuss it here or later on in group meetings.

Option 1: Leave Provider as-is, Add /trips to Agency. Convert non-state-change, outside of trip telemetry to be wrapped in events for both Agency and Provider.
Having non-state-change telemetries to be wrapped in events as a "heartbeat" would be a great idea. It would reduce the amount of telemetries ingested and deal with the look-back period issues. Moreover, adding /Trips to Agency would be a step to Unification, but we believe that we can have a smoother progression by going further and also adapt Provider.

Option 2: Add /telemetry to Provider and drop route data from /trips. Add /trips to Agency.
The advantage of the route data is that we are sure that it contains the exact data of a trip and allows not to have to recompute the trips based on the telemetries, which can sometimes lead to errors. We think that dropping routes would be a step back in the MDS.

Option 3: Abandon API symmetry, ignore Provider, add /trip_metadata to Agency
The value of the Provider API relies on the fact that we can use it to repoll data in case of issues (drop in the data, errors that need to be changed, etc.) in the past. Ignoring Provider would mean to loose the historical data.

Option 4: Merge Provider and Agency into a new endpoint
It is technically impossible to have one endpoint which can be pushed on and pulled from. What could be done would be to keep the different endpoints with a an exact symmetry of all the data, allowing to choose whether to use the push or pull method.

[NEW] Option 5 : Add /telemetries to Provider and /trips to Agency. Convert non-state-change, outside of trip telemetry to be wrapped in events for both Agency and Provider.
What we propose here is to keep all the great ideas exposed in the four options above. By adding /telemetries to Provider and /trips to Agency we would ensure the perfect symmetry of the APIs (with the unification of all the fields names). That way, it is possible to receive in "real-time" all the information with Agency and then poll it an hour later with Provider and make sure that no data was missed. That combined with the addition of the heartbeat would be a great way to reduce drastically the amount of telemetries.

Let us know what you think of our last option, or comment back on the first fours in case we missed anything.

[schnuerle]

We will be discussing this at tomorrow's working group meeting.

[marie-x]

Thanks for the additional feedback, @edecorbiere.

I think Option 5 is a non-starter, because it requires both Agency and Provider at all times, both maintaining server endpoints 24/7. This is unacceptable to some Agencies and also some Providers. Also it means two sets of authentication keys, for each direction. If we were going to follow this path, we would probably want to add long-polling to both APIs so that either side can initiate requests, rather than insisting on true symmetry. (This is how I would address your point in Option 4.)

Option 3 doesn't necessarily sacrifice historical data, although it does prevent rewriting history (an explicit design goal of Agency). LADOT has been on Agency for three years with no known data loss. Downtime gaps have been filled by the Providers sending retries.

[schnuerle]

Hi @edecorbiere could you specify which organization you represent? It would be good to know and you can follow our community guidelines here for your GitHub profile.

[edecorbiere]

Hi @marie-x & @schnuerle ,

I'm working with Blue Systems on the Mobility Manager.
I think I may have wrongly expressed what we meant in option 5 : of course, we're not suggesting that both Agency & Provider APIs are up 24/7 for everybody. The idea is much to have a perfect symmetry between the two, that way providers can use the one they prefer and agencies will either way have the exact same information.

[alexdemisch]

I like Option 2 here because it puts all the GPS breadcrumb/telemetry data all in one /telemetry endpoint. When I'm thinking about the passenger services work, this is a good thing since it makes spatial analyses of telemetry data regardless of state (e.g., calculating travel time/speed, looking at which streets have been used) easier – it would be a pain to have to piece together all spatial data associated with a device by pulling it from multiple endpoints. As noted in the Google doc, passenger services (and other modes) can have substantial non-trip activity.

Removing trip telemetry data from /trips isn't necessarily a bad thing – if agencies don’t want spatial data from trips (or at all), then they just don’t require /telemetry rather than specifically disallowing route data from /trips.

[schnuerle]

We will be discussing a detailed proposal about a solution for this at today's working group meeting.

[bhargav-lime]

@schnuerle I am working on implementing changes for /trips endpoint and noticed that 1.0 had information around trip route however, that is not present as part of MDS 2.0? Can you share more details around the reasoning please? Feel free to tag someone who maybe able to help here.

[jiffyclub]

@bhargav-lime For trip routes in MDS 2.0 the idea is that we'll use the telemetry endpoint. Telemetry objects have a trip_ids field for correlating telemetry pings with trips.