[WIP] feat($httpUrlParams): introduce new service abstracting params serialization

[pkozlowski-opensource]

@petebacondarwin @caitp here is another stab at having configurable strategies for $http request params serialization. This is still WIP (missing tests and doc updates) but wanted to check this one with you before putting more work into this approach.

Here is the basic idea:

• we introduce a new service (and the associated provider), $httpUrlParams, that is responsible for serializing $http's request params
• this new service has essentially one method: serialize = function(params, mode) where mode is a flag / key that people can use to trigger different serialization strategies (ex. per domain)
• the new provider allows you to configure a default mode (ships with the one named traditional) and register mode handlers (a function)
• people, while doing $http calls can trigger different modes by passing paramsMode in the config object, ex.:

$http.get(myUrl, {
    params: {...},
    paramsMode: 'jquery'
}).then(...);

It would be totally awesome if we could ship it with 1.4, so any input would be much appreciated.

[Narretz]

Should we throw if a mode is not available? I think we should.

[gkalpak]

If we decide to throw, it would be better to throw a more meaningful error (instead of undefined is not a function).
I would also consider logging the incident and falling back to the default mode.

BTW, you are using lowercase here, but not when registering. They should be consistent (and probably documented).

[petebacondarwin]

We should ditch the lowercase altogether. People should be able to specify the correct capitalization.

[petebacondarwin]

This is a great concept to be able to specify this mode in the $http requests.

My first thought is does this really need to be a separate service? It seems like it would quite easily be added to the $httpProvider/$http service. What is the benefit of a new service?

[pkozlowski-opensource]

@petebacondarwin you need a separate service due to #3311

[gkalpak]

While you are touching this, === would be nice :)

[pkozlowski-opensource]

@petebacondarwin @Narretz @gkalpak thnx for the detailed review of the code - all the comments are very valid and for sure I'm going to incorporate those into the final version.

But what I need mostly from you now is a OK / KO when it comes to concepts and APIs exposed to the users. Are there any other concerns apart from @petebacondarwin question about having this logic in a separate service? Once again, the need for a separate service was driven by those 2 issues:

• Custom URI encoding makes it hard to mock $http requests #3311
• feat(http): Please make buildUrl public #7429

If you think that the current approach / APIs are reasonable I would add tests and address all the comments so we can get it into 1.4. But if you've got any concerns regarding the approach it is time to speak up now :-)

[petebacondarwin]

@pkozlowski-opensource - OK so here is my take on it. This should by no means be taken as the right way, it is just an opinion...

I would steer away from using strings and lookups like this. It adds an additional opportunity for bugs to be disguised. Instead I would follow a similar design pattern as Http Interceptors, where a paramSerializer is just a service that implements an interface. Anyone is free to create their own version of this that will do the serialization in their own way:

mod.factory('myParamSerializer', function() {
  return function myParamSerializer(params) {
    ...
  };
});

Out of the box we would provide two standard versions: $traditionalParamSerializer and $jqueryParamSerializer.

Then you just pass the version you want to $http. You could do this explicitly in a request:

myMod.controller('MyCtrl', function($scope, $http) {
  $http.get(myUrl, {
      params: {...},
      paramSerializer: '$jqueryParamSerializer'
  }).then(...);
});

or you could set a default in the $httpProvider:

myMod.config(function($httpProvider) {
  $httpProvider.defaultParamSerializer('$jqueryParamSerializer');
});

Following the Http Interceptor design, you could also define the serializer "inline" instead of as a service.

[pkozlowski-opensource]

@petebacondarwin I like what you are saying here and my initial proposal was kind of going in this direction. But I've changed this, mostly based on @caitp input, as I believe that:

• injecting serializers is cleaner but more troublesome for users (this might be just my non-objective perception)
• setting defaults on the $httpProvider kills all the testing-related benefits I wanted to have (see Custom URI encoding makes it hard to mock $http requests #3311 and feat(http): Please make buildUrl public #7429)

@petebacondarwin @caitp or anyone interested - could we try to have a hangout on this topic somewhere early next week to clarify / agree upon the design? I would really love to ship this thing with 1.4!

[caitp]

i can do a hangout, but my 2c is still that it should be kept as simple as possible. $http(url, { params: ..., legacyParams: true }) or something

[petebacondarwin]

OK just to clarify...

There could be a getter on the $http service that returns you the current default serialised for use in tests.

My idea would allow the name of the serializer to be passed instead of an instance avoiding the need to inject it

[jmendiara]

I hope is not too late for debate...
@pkozlowski-opensource and mates, here you have #11386 another approach that solves the same issues and addresses @petebacondarwin concerns, giving more control to userland and keeping core simpler, IMHO.

[pkozlowski-opensource]

OK, so we start to have different proposals from @petebacondarwin, @caitp and @jmendiara now and if we want to land something in 1.4 we need to chose now :-) To facilitate the decision process here is once again a list of problems we are trying to tackle with this change:

1. support most common serilaization schema (the current one and jQuery's one) - Query params not serializes as expected #3740
2. ability to specify a default serialization schema (per app)
3. ability to specify a per-request serialization schema
4. allow people to have completely custom serialization schemas - feat(http): Please make buildUrl public #7429 and Passing a string that contains a semi-colon doesn't properly get encoded #9224
5. use serialization functionality in unit tests - Custom URI encoding makes it hard to mock $http requests #3311

Now, with my proposal here is how one would tackle each use-case.

For (1) and (2):

angular.module('app', [], function($HttpUrlParamsProvider) {
    $HttpUrlParamsProvider.defaultMode = 'jquery';
}).controller('MyCtrl', function($http) {
   $http.get('http://host.com', {params: {foo: ['bar', 'baz']}})
});

would result in http://host.com?foo[]=bar&foo[]=baz for each and every request.

For (3) people could do:

angular.module('app', []).controller('MyCtrl', function($http) {
   $http.get('http://host.com', {params: {foo: ['bar', 'baz']}, paramsMode: 'jquery'})
});

To tackle (4) one would do:

angular.module('app', [], function($HttpUrlParamsProvider) {
    $HttpUrlParamsProvider.registerSerializer('mycustom', function(params) {
        return //whatever crazy serialization people want to have
    });
    $HttpUrlParamsProvider.defaultMode = 'mycustom';
}).controller('MyCtrl', function($http) {
   $http.get('http://host.com', {params: {foo: ['bar', 'baz']}})
});

and finally, one could inject $httpUrlParams into test to express $httpBackend expectations both for the default serialization mode:

$httpBackend.expectGET('http://host.com?' + $httpUrlParams.serialize( {foo: ['bar', 'baz']}));

as well as for the per-request basis:

$httpBackend.expectGET('http://host.com?' + $httpUrlParams.serialize( {foo: ['bar', 'baz']}, 'mycustom'));

Now, what I really dislike here is relaying on string-based names that are really easy to get wrong... At the same time I would appreciate solutions where (4) and (5) are properly handled.

From what I understand @caitp is proposing:

angular.module('app', []).controller('MyCtrl', function($http) {
   $http.get('http://host.com', {params: {foo: ['bar', 'baz']}, serialiseAsJQuery=true})
});

which is easy to use and tackles (1) as well as (2) but doesn't help with the rest, as far as I can see.

@petebacondarwin @jmendiara how the code would look like with your respective proposals?

[petebacondarwin]

My suggestion is not that much different except it removes the need for this registration service. We would provide two basic serializers (as angular services) out of the box:

• $legacyParamSerializer (default)
• $jqueryParamSerializer

The $httpProvider provider would expose a getter/setter $httpProvider.defaultParamSerializer(value)
and the $http service would expose only a getter $http.defaultParamSerializer().

Custom serializers would simply be angular services, see customParamSerializer below.

Below are the demonstrations of how this would work for each use case.

1/2) Specify the serializer at the application level:

angular.module('app', [])

.config(function($httpProvider) {
  $httpProvider.defaultParamSerializer('$jqueryParamSerializer');
})

.controller('MyCtrl', function($http) {
   $http.get('http://host.com', {params: {foo: ['bar', 'baz']}});
});

3) Specify the serializer at the request level:

angular.module('app', [])

.controller('MyCtrl', function($http) {
   $http.get('http://host.com', {params: {foo: ['bar', 'baz']}, paramSerializer: '$jqueryParamSerializer'});
});

or more explicitly

.controller('MyCtrl', function($http, $jqueryParamSerializer) {
   $http.get('http://host.com', {params: {foo: ['bar', 'baz']}, paramSerializer: $jqueryParamSerializer});
});

4) Provide a completely custom serializer:

angular.module('app', [])

.factory('myCustomParamSerializer', function() {
  return function myCustomParamSerializer(params) {
    return // Whatever crazy serialization people want to have
  };
})

.config(function($httpProvider) {
  $httpProvider.defaultParamSerializer('myCustomParamSerializer');
})

.controller('MyCtrl', function($http) {
   $http.get('http://host.com', {params: {foo: ['bar', 'baz']}})
});

5) Enable serialization during testing:

it('should respond to a request with default serialization', inject(function($httpBackend, $http) {
  $httpBackend.expectGET('http://host.com?' + $http.defaultParamSerializer({foo: ['bar', 'baz']}));
}));

it('should respond to a request with a specific serialization', inject(function($httpBackend, myCustomParamSerializer) {
  $httpBackend.expectGET('http://host.com?' + myCustomParamSerializer({foo: ['bar', 'baz']}));
}));

[pkozlowski-opensource]

@petebacondarwin oh, I see, you would expose a custom serializer on the $http itself - OK - I think I like it.

@jmendiara how is your proposal different from what we are discussing here?

[jmendiara]

@pkozlowski-opensource It's more like @petebacondarwin approach, but clearly separates param serialization from URL Building (which includes requestParameters concatenation with ; or & #9224 and encoding #1388).

My WIP #11386 didn't ship jquery support in order to keep core small and simple (should core ship other fws functionalities? read @caitp comments ) and let users to implement this on their own apps (as an external dependency) or maybe provide an ngJquerySerialize optional module.

My approach registers an optional core provider that helps on urlBuilding, but is not needed at all, just sugar to developers to hack easily into the functionality. Although not in the WIP the jquery support can also be implemented

AssumingTaking @petebacondarwin snippets as starting point...

1/2) Specify the serializer at the application level:

angular.module('app', [])

.config(function($httpProvider) {
  $httpProvider.defaultURLBuilder('$jqueryURLBuilder');
 //OR   $httpProvider.defaults.buildUrl = '$jqueryURLBuilder';
// the last one is just like specifying the default Cache for request, keeping the same paradigm 
})

.controller('MyCtrl', function($http) {
   $http.get('http://host.com', {params: {foo: ['bar', 'baz']}});
});

3) Specify the serializer at the request level:

angular.module('app', [])

.controller('MyCtrl', function($http) {
   $http.get('http://host.com', {params: {foo: ['bar', 'baz']}, buildUrl: '$jqueryURLBuilder'});
});

or more explicitly

.controller('MyCtrl', function($http, $jqueryURLBuilder) {
   $http.get('http://host.com', {params: {foo: ['bar', 'baz']}, buildUrl: $jqueryURLBuilder});
});

4) Provide a completely custom serializer:
This is where the approached differ most:

//petebacondarwin use case is supported
angular.module('app', [])
.factory('myCustomUrlBuilder', function() {
  return function myCustomUrlBuilder(url, params) {
    return // Whatever crazy serialization people want to have
  };
})
.config(function($httpProvider) {
  $httpProvider.defaultURLBuilder('myCustomUrlBuilder');
})
.controller('MyCtrl', function($http) {
   $http.get('http://host.com', {params: {foo: ['bar', 'baz']}})
});

or using my optional-to-implement-in-the-core $httpUrlBuilderFactory core helper provider to implement url builders. The URL Builder factory creates encoded urls concatenated with & (the current angular default behavior)

angular.module('app')
//Create a function that says how an object should be converted to a `List of {key, value}` strings
.factory('arraySerializer', function() {
  return function arraySerializer(params, addKeyValue) {
    angular.forEach(params, function(value, key) {
      if(angular.isArray(value)) {
        angular.forEach(value, function(arrValue) {
          addKeyValue(key+ '[]', String(arrValue));
        });  
      } else {
         addKeyValue(key, String(value));
      }
    });
  }
})

//We rehuse the common way of building URLs
.factory('arrayStandardBuildUrl', function($httpUrlBuilderFactory, arraySerializer) {
  return $httpUrlBuilderFactory(arraySerializer);
})

//we rehuse the arraySerialization, but making other kind URLs
// semicolonURLs returns a function(url, params) that returns urls concatenated with ;
// see https://github.com/jmendiara/angular.js/blob/http_params_serialization/src/ng/http.js#L1230
.factory('arrayCustomBuildUrl', function(arraySerializer, semicolonURLs) {
  return semicolonURLs(arraySerializer);
})

.controller('MyCtrl', function($http) {
  $http.get('http://host.com', {params: {id: 5, foo: ['bar', 'baz']},
    buildUrl: 'arrayStandardBuildUrl'
  }); // GET http://host.com?id=5&foo[]=bar&foo[]=baz

  $http.get('http://host.com', {params: {id: 5, foo: ['bar', 'baz']},
    buildUrl: 'arrayCustomBuildUrl'
  }); // GET http://host.com?id=5;foo[]=bar;foo[]=baz
})

5) Enable serialization during testing:

it('should respond to a request with default serialization', inject(function($httpBackend, $http) {
  var url = 'http://host.com', params = {foo: ['bar', 'baz']};

  $httpBackend.expectGET($http.defaultURLBuilder(url, params));
  $http.get(url, {params: params});
}));
it('should respond to a request with a specific serialization', inject(function($httpBackend, $http, myCustomUrlBuilder) {
  var url = 'http://host.com', params = {foo: ['bar', 'baz']};

  $httpBackend.expectGET(myCustomUrlBuilder(url, params));
  $http.get(url, {params: params, buildUrl: myCustomUrlBuilder });
}));

As you can see $httpUrlBuilderFactory is optional, but helps developers to implement serializers

Why complete URL building and not only param serialization??
My approach covers the following use cases:

• As a user, I want to delete $http cache entries: Currently, $http cache has the complete URL as a key. If the URL is generated in core clausures, like your approaches, I cannot be truly confident about the url it was used. Why should a user want to delete cache entries? Some grumpy backends return statusCode === 200 and an error payload {success: false, data: ...} This entry should not be in the cache, but angular core assumes a good response. Users can code a response interceptor, execute config.cache.remove(config.buildUrl(config.url, config.params)) to obtain exactly the cache key and remove from the cache
• As a user, I want to have a serialized version of my params as jQuery does, without encoding, in order to make some operations with them, like oauth1 signing: Separating URL building (encoding + concatenation) from serialization lead to reuse the serialization and implement the custom url building.

What i dislike most of my approach
The signature function for a serializer, using false callbacks function like addKeyValue

What i like most of my approach
Completely freedom and user power to hack into $http. The important thing is to allow users to buildUrl, not the sugar provided by $httpUrlBuilderFactory, which can be removed

[petebacondarwin]

I really like the idea of using the $httpProvider.defaults object instead of my idea of a getter/setter since this is much more in keeping with the current API design.

[petebacondarwin]

I can see the benefit in providing a way to plugin in to the whole HTTP url building process but I think that this buildURL API not the best way... I would prefer to see the two processes completely decoupled, rather than the url building "owning" the parameter serialization.

In the meantime, I wonder if we could solve the given use cases without resorting to completely opening up the url building:

• For the cache clearing, we could easily add the fully generated URL to the request/response objects that are available to the interceptors.
• For the oauth processing, could a custom serializer not do this as part of processing the params?

[pkozlowski-opensource]

I would very much like to avoid going into cache-related issues as part of what we are doing here... IMO the current cache implementation for $http leaves much to be desired, especially in terms of what people can control (ex.: when things are cached, under which key etc.).

In short: cache has its own set of issues that we should tackle separately, IMO.

[petebacondarwin]

I agree that it would be best if we could keep this simple.

I believe if we attach the generated URL, as returned from the private function buildUrl(), to the request config object, as config.fullUrl say, then we can use this property throughout the rest of the sendReq function in the $http service instead of the url variable.

This would allow a request interceptor to have access to this URL but also have the opportunity to completely rebuild this URL in any way they want, which is effectively the requirement of @jmendiara.

What do you think?

[pkozlowski-opensource]

I don't think I feel comfortable with using config object as a container for temporary variables...

[petebacondarwin]

That's fine. It doesn't relate directly to this PR anyway and could always be added later if it was wanted. I think the response headers would also already contain this info, right?

[pkozlowski-opensource]

Probably. In any case I don't want to go into cache-related problems just when we should be shipping this one :-)

For now I'm leaning towards @petebacondarwin proposal - if there are no objections I can re-submit my PR with this idea implemented.

[Narretz]

I think Pete's proposal is the best with regards to consistency with the rest of Angular.
One thing, will this new serializer also work with $resource?

[jmendiara]

@petebacondarwin

For the oauth processing, could a custom serializer not do this as part of processing the params?

oauth1 signature needs params + url + method

If we have transformRequest and transformResponse functions, introducing a function buildUrl(config) function/service is very well aligned with $http() and $httpProvider.defaults API

I'm talking now about introducing the ability to buildUrl per request and in defaults, not my proposed $httpUrlBuilderFactory, as it is only sugar

Small code change, no new services/providers in core, fully customization by user and more use cases covered.

BTW, @pkozlowski-opensource whichever decission you take, tell me if you need help with anything

[pkozlowski-opensource]

Suppressed by #11461

[pkozlowski-opensource]

@jmendiara if you got a minute to review #11461, this would be much appreciated!