RESTful API - limiting response length

[programster]

Hello all,
I am in the midst of developing a RESTful API, and I wish to implement a response length limit (similar to “Paging” practice often used in web frontends) e.g. the API will only return up to 5000 “resources” at a time and it is up to the requestor to ask for a different range if they need more results. I have experienced this with other APIs, such as Amazon S3, so I don’t believe it is an uncommon practice. This will help prevent the API, or whatever is using the API from running out of memory when they unexpectedly get a large request or response.

My question is whether there is a general “standard” (or at least popular implementation) for the “protocol” / behavior online that people are using and I can read. E.g. should my API return with a new header when the response has been limited, or the header always exists and is flipped to true when I reach my limit? Should the user send a page=2 variable or an offset=5000 in their requests when they want to get more results. Do I need to return the total number of possible results in my response or is that not required (likely to save me from running unnecessary full table scans).

Any tips or personal anecdotes in this area is appreciated.

[mihawk90]

Imo a REST API should return everything that matches the query.
Paging the results is not really the job of an API, it should just be serving the data that is requested.

The problem is that a REST API is supposed to always have the same records. If you were to fetch say 100 user records it may actually serve the first 100 records. But what happens if a user is deleted. Either the API returns a different result (which is technically not RESTful), or you fetch every user with an ID ranging from 1-100, in which case there wouldn’t be 100 results anymore.

When you return your result you’re supposed to send a header stating how big the response is anyway via Content-Length header. If the application then loads everything into RAM or not is not your concern to have, that’s the application’s job to manage.

Well, that’s just my opinion anyway

[programster]

Yes, that is a valid point. The possibility of inconsistencies creeping in due to a state change between subsequent requests also concerns me. If it was a matter of 100 records, I wouldn’t sweat it. However, in this case the API sometimes responds with over 100k records. I thought it would be prudent to put a large cap on there to catch extreme situations.

[mihawk90]

Hm… well… it also depends how the response is generated because the server-side language (PHP?) may reach its configured memory limit before that is a concern.

Kinda curious though, how is it responding with such an amount of records…

But, to the point, yes putting a cap on it might be a good idea. The question is if you want to respond with an error in that case, asking the application to make a more specific query (if that is possible) and therefore having consistency, or if you want to actually limit the number of results (which should be clearly noted in the docs) and return the maximum number of results.

[risk]

One alternative is to have an optional argument/field in your request that defaults to 5000 and just set a hard limit to something like 100k using a constant or a flag on the server, you could write a log when serving this

---

If you really want tokens for a frontend, e.g. to save on bandwidth by lazily loading a page and in order to be more browser friendly, typically you would do this via continuation tokens, for example, with no token, you get the first 5000 and a token, referencing this token later on gets you the next 5000 and another token, and so on.

Last batch has no token, if the server gets a request referencing a token it doesn’t know about, it should return an application level error that the client should decide how to handle.

[programster]

Kinda curious though, how is it responding with such an amount of records…

You’re right that the backend is PHP. It is able to serve up this many records by fetching them from the database in batches, and it constantly converts them to json string chunks before continuing. It’s able to be pretty memory efficient and I switched to Nginx with FPM to ensure that only x number of requests are being handled simultaneously but still be able to cope with being hit with more than that number of requests at once.

The trouble with relying on the user to make more specific queries is that this will be open to third parties and people do dumb stuff all the time.

[programster]

If you really want tokens for a frontend, e.g. to save on bandwidth by lazily loading a page and in order to be more browser friendly, typically you would do this via continuation tokens, for example, with no token, you get the first 5000 and a token, referencing this token later on gets you the next 5000 and another token, and so on.

Last batch has no token, if the server gets a request referencing a token it doesn’t know about, it should return an application level error that the client should decide how to handle.

I didn’t understand what you meant at first but realize now that is a clever solution. So if the server truncates the response, it gives the app the token that the external party can hit in order to get the next chunk, and so on until there are no more tokens. This would remove the consistency issue, but I will have to store the tokens and the results in the database for a short period of time. This shouldn’t be too hard and I quite like it. Can you link/reference anyone who is already doing this or did you just think of it?

[risk]

Correct, you need to store some data about the client temporarily, e.g. if you have rows from a database, and you figure out it’s too many, you stuff them into memcached, and when a request comes in with a token, you bypass parts of your reply flow and just lookup stuff already in memcached

I didn’t invent it, but can’t tell you OTOH what would be a good place to look.

[programster]

Sounds good, drawing up plans for storing the rest of the response temporarily and updating the relevant response headers to give the token etc to grab the temporary data.

I didn’t invent it, but can’t tell you OTOH what would be a good place to look.

Does OTOH stand for “off the top of my head”?

[programster]

When you return your result you’re supposed to send a header stating how big the response is anyway via Content-Length header.

I think the content-length header is a standard part of the HTTP protocol that was automatically filled out by Nginx/Apache with the size of your body in bytes for the receiver. Are you saying I should manually override it with an integer representing the number of records instead?

[jak_ub]

Imo a REST API should return everything that matches the query.

You forgot that the offset and size (one form of pagination) might be part of the “query”, and also simply included in the URL.

If the application then loads everything into RAM or not is not your concern to have, that’s the application’s job to manage.

I would also consider this very bad advice. Within a reason you should consider the client applications our their developers will simply choose better vendor of the services. Or simply company will hire a better designer.

But what happens if a user is deleted.

That depends.
Is the system you are doing and software that will be monitoring and managing critical health care equipment?
Then we mitigate such problems (example solution is mention in this topic).

Is it a cat food online store?.
We do nothing with this specific issue, and if it will happen that the client application will ask for last page than no longer exist we will return response informing that it went too far last page is there.

One of the solution between is to inform client application of a last modification date of the collection so the client could “react”. To be honest there are multiple ways to solve this issue depending on the use case, actual data, and how often it changes.

Content-Length

This specific header should be completely transparent to the REST API and be strictly only part of HTTP protocol. The only connection here is that is should be byte size of the entity in the response (if this size is known).

[jak_ub]

Use weapon of your choice:
(the language or framework do not matter)

http://www.django-rest-framework.org/api-guide/pagination/

https://blog.octo.com/en/design-a-rest-api/#pagination

API pagination best practices

https://developer.github.com/v3/guides/traversing-with-pagination/

https://developer.wordpress.org/rest-api/using-the-rest-api/pagination/

http://phlyrestfully.readthedocs.io/en/latest/halprimer.html#collections

[mihawk90]

this will be open to third parties and people do dumb stuff all the time.

While this is true, this is not really the job of the API to handle in my opinion.

Are you saying I should manually override it with an integer representing the number of records instead?

Not with the number of records, that would be a bad idea (it’s the number of bytes in the response by definition and getting it wrong can yield some very weird results). But you can override it if you need to or if it’s not correctly sent by the webserver. What I meant is that the API user should read out the header and throw away the response if it’s too big to handle.

[Argon]

Hello, sorry for the VERY LATE input, but having read through what you want/need, I think the best thing to do is to return let’s say 100 relevant items at a time. This way if you’ve got a relatively small server, no worries, I assure you that you’ll be fine, if anything I’ve recently been developing something very similar to what you’re talking about.

Here’s what I do, on the front end you have a ‘JSON template’ which mimics the query that you want to submit through to the back end, you update the properties of this object via inputs from the UI, you can do this however you like, personally I have a bunch of input fields and set a time delay so that when you’re typing, it’ll only perform a search after let’s say 1.75 seconds.

Then the back end, what I do is have a few endpoints in place, I have the generic ‘find’ endpoint which basically performs the query, and has a limit and an offset like you’re thinking about. So an example being:

Limit = 100
Offset = 0 (page 1)

The query may look a little like this:

SELECT ... FROM... JOIN..  OTHER STUFF... LIMIT = 100 OFFSET = 200

You then have your ‘size’ end point which is a simple get request which will ressentially execute:

WITH ( ... your query.... ) AS RESULT SELECT COUNT(*) FROM RESULT

This way you don’t need to have some sort of response to see when the limit has been reached, the logic can be done on the front end, sure you can have some validation on the back end, but a big part of REST is to keep the back end quite lightweight. Trust me, I’ve dived into transferring an admin system from JSF to JAX-RS, because the JSF implementation was eating up a tonne of memory!

I mean I can actually show you some demo source code of how I do it if you like? Keep in mind, due to the Java background, I use a tonne of OOP, like I’ve talked with PHP developers and they seem afraid of the idea of using abstract classes and methods and what not, which I find bizarre! … Each to their own I guess?

Back to the topic at hand, the way I’ve been doing this is INCREDIBLY lightweight on the back end and it’s super fast, instant, I mean I’m actually using front end template stuff that I’ve personally made myself, and that’s only down to animation and what not! … Plus the beauty of the approach I’ve implemented, it makes life super easy with regards to maintaining the code!

If you want support/help/advice at all, DM me!
Remember that simplicity is key!