-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Iterator Guidelines
#Introduction
Google Cloud Platform API clients for Go should be as consistent as possible. Since Go has no standard iterator pattern, this document establishes guidelines for iterators.
Most iterators will result from the standard Google APIs List method with the pagination pattern: each call to a List method for a resource returns a sequence (“page”) of resource items (e.g. Books) along with a “next-page token” that can be passed to the List method to retrieve the next page.
Each List method will result in an iterator (which we call a List iterator) with two methods, one for individual items and one for pages. Iterators may also arise from other sources, like streaming RPCs or Cloud PubSub message subscriptions. These may or may not support page-by-page iteration.
Here is what iterators written according to these guidelines will look like to users. Here is the List iterator for the Book resource in the library example:
it := client.Books(ctx, shelfName)
for {
book, err := it.Next()
if err == library.Done {
break
}
if err != nil {
return err
}
process(book)
}Here's the same code using a switch:
it := client.Books(ctx, shelfName)
loop: for {
book, err := it.Next()
switch err {
case nil: process(book)
case library.Done: break loop
default: return err
}Here is what iteration by pages looks like:
it := client.Books(ctx, shelfName)
for {
books, err := it.NextPage()
if err != nil && err != library.Done {
return err
}
for _, b := range books {
process(b)
}
if err == library.Done {
break
}
}Here we retrieve the first page of 25 (or fewer) books and display the page and the next-page token:
it := client.Books(ctx, shelfName)
it.SetPageSize(25)
books, err := it.NextPage()
if err == nil {
display(books, it.NextPageToken())
}When the next-page token is handed back to us later (possibly in another process), we can get the next page:
it := client.Books(ctx, shelfName)
it.SetPageSize(25)
it.SetPageToken(token)
books, err := it.NextPage()
if err == nil || err == library.Done {
display(books, it.NextPageToken())
}An iterator should be represented by a type whose name ends in Iterator. If the iterator is a List iterator, the type's name should be ResourceIterator, E.g. BookIterator. The type should have at least one method, called Next. List iterators will also have a NextPage method. Next and NextPage are described below.
Example:
type BookIterator struct { ... }Typically, the client will have a single method that returns an iterator of a particular type. We will call this the creating method.
The name of the creating method for a List iterator should be the plural of the resource, e.g. Books (not ListBooks, which is a bit verbose for Go). For other kinds of iterators, the name of the creating method should be a plural noun, but a different name can be used if it makes more sense.
The first argument to the creating method should be a context.Context. The iterator will use that context throughout the iteration. In the unlikely event that neither the creating method nor the iterator make any RPCs, the context can be omitted.
If iteration makes any RPCs, the iterator-creating method may accept call options in a final ... parameter, and store them for subsequent RPCs.
In most cases, the creating method will simply create an instance of the iterator type and return it, leaving the initial work, including any RPCs, to the first call to Next. In other cases, a creating method may return an error along with the iterator.
An iterator over values of type T will have a method called Next that returns (T, error). For example,
func (it *BookIterator) Next() (*Book, error) { ... }Next will typically have no arguments, but it may in some cases. (For example, the Datastore iterator's Next method takes an argument into which it copies the entity.) None of the arguments should be a context, because Next should use the context passed when the iterator was created.
Following standard Go convention, if Next’s second return value is non-nil, then the first must be the zero value for T.
A special error value returned by Next signals the successful end of the iteration. This sentinel value will be named Done (unless that results in a conflict) and will be declared as a variable in the same package as the iterator. After Next returns Done, all subsequent calls to it will return Done.
If feasible, the user should be able to continue calling Next even if it returns an error that is not Done. If that is not feasible, it should be so documented.
Iterators that support pagination, such as List iterators, should also have a NextPage method that returns ([]Resource, error). For example,
func (it *BookIterator) NextPage() ([]*Book, error) { ... }If NextPage's second return value is neither nil nor Done (in other words, an actual error), then its first return value must be nil. If the second return value is Done, the first may be nil or a zero-length slice, or it may contain a final page of items. Hence loops with NextPage should check first for a true error, then process the list of items, and finally check for Done. See the third example above.
Iterators that support pagination should have a SetPageSize method which sets the page size for all subsequent RPCs from the current iterator. It should take an int32 argument and have no return value:
func (it *BookIterator) SetPageSize(int32)Callers who use only the iterator's Next method should treat SetPageSize as a hint, much as a buffer size argument in an I/O method.
Page size semantics may differ across clients.
In those intended to reflect the underlying API closely (e.g. machine-generated clients), the page size provided by SetPageSize is a maximum; fewer items may be returned in subsequent NextPage calls. Indeed, it is valid for NextPage to return (r, nil) where len(r) == 0. In other words, there can be empty pages in the middle of the iteration.
Other clients aspire to be a higher-level wrapper on top of the API, built to favor user convenience over faithfulness to the underlying RPCs. The NextPage methods in such clients should return exactly the page size (if that many items are available), to facilitate the common use case of displaying pages of items in a web or command-line interface.
Google API calls that support pagination typically return a next-page token, which can be passed to a subsequent RPC to obtain the next page of results. Iterators built on such RPCs should provide two methods to support this, NextPageToken and SetPageToken. Note that for a simple loop implementing page-by-page iteration as in the example above, explicit use of the page token is unnecessary; the iterator will perform the necessary manipulations (the equivalent of SetPageToken(NextPageToken())) itself. The page token methods are only necessary when an iteration needs to be resumed later, possibly in another process.
The NextPageToken method should take no arguments and return the next-page token, typically a string:
func (it *BookIterator) NextPageToken() stringThe SetPageToken method should accept a string and have no return value:
func (it *BookIterator) SetPageToken(string) { ... }It should set the page token for the next RPC.
An iterator may require clean-up work to happen after it completes, or if the user abandons it before reaching the end. The iterator type should have a method of no arguments that performs this clean-up.
If the clean-up work can result in an error that should be returned, the method should be named Close. If no error is possible (or if the error is to be exposed in another way), the method should be named Stop:
func (it *CleanupWithErrorIterator) Close() error { ... }
func (it *CleanupNoErrorIterator) Stop() { ... }If an iterator does not require clean-up, it should define neither Close nor Stop.
If Close or Stop does not support being called multiple times, that should be documented.
If an iterator supports both Next and NextPage, it is required to be “smart” about interleaved calls to those methods. That is, they need not support any particular semantics for a call to Next followed by a call to NextPage, or vice versa. For the life a single iterator, we expect users to call only Next, or only NextPage.
The Next and NextPage methods are not required to be thread-safe, and Close (or Stop), Next and NextPage are not required to be concurrently callable.
Iterators may have fields and methods other than those described here.