Skip to content

Enable use of page_size for clients - #408

Merged
kyleknap merged 2 commits into
boto:developfrom
kyleknap:client-page-size
Dec 10, 2014
Merged

kyleknap merged 2 commits into
boto:developfrom
kyleknap:client-page-size

Conversation

@kyleknap

@kyleknap kyleknap commented Dec 9, 2014

Copy link
Copy Markdown
Contributor

So the reason I made this pull request is because, you cannot paginate using a page size with clients. This is what the error you get before my PR:

>>> import botocore.session
>>> session = botocore.session.get_session()
>>> s3 = session.create_client('s3')
>>> paginator = s3.get_paginator('list_objects')
>>> for page in paginator.paginate(Bucket='mybucketfoo', page_size=1):
...     print page
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kyleknap/Documents/GitHub/botocore/botocore/paginate.py", line 67, in __iter__
    self._inject_starting_params(current_kwargs)
  File "/Users/kyleknap/Documents/GitHub/botocore/botocore/paginate.py", line 140, in _inject_starting_params
    page_size_name = self._operation.pagination['limit_key']
AttributeError: 'PageIterator' object has no attribute '_operation'

This also probably affects boto3 since it uses clients to paginate as well.

The fix was to retrieve the limit_key from the pagination config instead of relying on an Operation object for the limit_key, which the clients had no access to.

cc @jamesls @danielgtaylor

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the Paginator name is clobbered by the renaming of DeprecatedPaginator, I had to rename the paginator that we will eventually be using. I considered switching everything over to the paginator will be using in the future. However the CLI relies on the deprecated paginators, so I was hesitant to make the switch.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be nice to add this as a comment in the test so we don't forget why this was done.

@jamesls

jamesls commented Dec 9, 2014

Copy link
Copy Markdown
Member

:shipit: Looks good.

On a completely separate note, I'm finding it jarring that we use CamelCased and snake_cased params for pagination. I'm thinking for the client we should be consistent and use the CamelCased params for the pagination values as well. Thoughts? cc @danielgtaylor

@coveralls

Copy link
Copy Markdown

Coverage Status

Coverage decreased (-1.04%) when pulling 86ce5fd on kyleknap:client-page-size into 459da7f on boto:develop.

@danielgtaylor

Copy link
Copy Markdown
Contributor

LGTM 🚢-it.

CamelCased or snake_cased are both fine for me. It seems possible that some service in the future might not use CamelCasing for its args, and then we'll have the opposite problem we do today. If we want to go with CamelCasing for now then that's fine since (all?) services use CamelCased params.

@jamesls

jamesls commented Dec 9, 2014

Copy link
Copy Markdown
Member

I don't think a service will ever use snake_casing.

@kyleknap

kyleknap commented Dec 9, 2014

Copy link
Copy Markdown
Contributor Author

+1 CamelCase since we are moving away from service and operation objects, which accept snake_case.

@kyleknap

Copy link
Copy Markdown
Contributor Author

I added a todo comment to help us later on when we switched the test to use the non deprecated paginator. Merging.

kyleknap added a commit that referenced this pull request Dec 10, 2014
Enable use of page_size for clients
@kyleknap
kyleknap merged commit 7a2181f into boto:develop Dec 10, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants