[BREAKING] New Search API Result Limits

Have more questions? Submit a request

73 Comments

  • Anton Mintsev
    Comment actions Permalink

    Hi!, will updated search endpoint still return the total number of search results when it's more than 1000?

    0
  • Ali Ayaz
    Comment actions Permalink

    Same question as the one Anton asked. And related to it...

    Does this mean that there will be no way for us to grab results using the search api end point, when the results are greater than 1000?? No pagination, nothing?

    I hope this is reconsidered and/or a viable alternative is offered — I for one use this end point quite often (e.g. we have close to 300k users, and if we want to just grab a subset of those users, it’s easier to do so via the search api ep — however those users will be more than 10k at times and it’s almost impossible to further bifurcate them ... and using the users api ep in this case is extremely inefficient)

    5
  • Salvador Vazquez
    Comment actions Permalink

    @Anton: You will still be returned the count of all items matching the search with your request as you had before

    @Ali: With this change you will not be able to get any results beyond 1k. The idea of the search API is to get the top results of the search instead of all items. Can you give me a bit more information about your use case of getting that list of users and why the user API would be inefficient for you. 

    0
  • Daniel Ertman
    Comment actions Permalink

    @Salvador : Not to speak for Ali, but here's our situation : Sometimes, something in our system goes down. When that happens, Care can get flooded with Zendesk tickets - we've seen 10,000 - 20,000 tickets during some particularly bad outages. It would be impossible for humans to deal with those tickets one by one, so we just close all of the tickets and deal with the customers / orders en masse.

    To close all of the tickets, we do a search by time range, particular tags, and particular custom fields. There's nothing more specific we can do. We're going to get a result of 10,000 tickets, because there are 10,000 tickets we need to close.

    With this change, it seems like we will need to:

    1. Run the search, get the total number of results
    2. Guess at a time segmentation that will result in less than 1000 tickets
    3. Run many searches to re-accumulate all of those results
    4. If we were wrong and any search returns more than 1000 results, split that time window and try again.

    This isn't better than paging for us. If you don't want us to use the search for this use-case, that's fine, but we'd need something more than time range added to incremental export in order to only close tickets impacted by our outage.

    0
  • Ali Ayaz
    Comment actions Permalink

    @Salvador: grabbing 300k users, in total, iterating through them to only display those records in a g.sheet which meet a certain criterion (having a certain tag, etc) VS using the search api ep, which will only return those users that meet the required criterion and the script won’t be facing timeouts in the latter case, whereas those would be common in the former case. Furthermore, the former case means more api requests, while the later case is faster, and significantly less api requests....

    This is just ONE of the many use cases... believe you me, we have dozens more such examples.

    Additionally, one other point, which isn’t mentioned anywhere... this change will also impact the regular search in the Zendesk Support Product (for users, tickets, organizations, etc) — won’t it?
    I honestly believe that this will be a major pain point, if not for all, but at the very least for most of your customers, especially the ones who are utilising this particular api ep, for tickets, orgs, users, etc. We have several Zd instances in the organization and our main/largest instance has over a million tickets. The search api ep made it easier to export data out based on a certain criteria (a custom field value, tag, group, etc) and now we will have to face major overhaul of our approach towards all our Zd specific scripts and integrations, etc.

    3
  • Alex Pereira
    Comment actions Permalink

    Hi,

    Could someone please clarify whether or not this change will affect any of these end-points as well:

    /api/v2/users/search.json?query={query}

    https://developer.zendesk.com/rest_api/docs/support/users#search-users

    /api/v2/organizations/search.json?external_id={external_id}

    https://developer.zendesk.com/rest_api/docs/support/organizations#search-organizations-by-external-id

     

    Thanks!

    1
  • Ali Ayaz
    Comment actions Permalink

    @Alex — of course it will impact those endpoints. You will only get the first 1000 results returned in a single page, with absolutely no option for pagination. That’s the way this change will be rolled out, as per the standing currently. Here’s to hoping that this will revisited by Zendesk.

    1
  • Graeme Carmichael
    Comment actions Permalink

    Will there be anything in the web response to indicate that results are incomplete? (Other than hitting exactly 1000 items)-sorry. I see you have answered that. 

    0
  • Dustin King
    Comment actions Permalink

    Beyond all of the API uses mentioned above (we have some ourselves), assuming this affects Zendesk Support's own search functionality, this will be a major hindrance to our business. When an issue in one of our products arises, we sometimes receive well over 1K tickets about the issue, and need to be able to search and find them all for triage, bug identification/solving, and most importantly for reply and remediation for our customers. Depending on the issue, we don't always have a specific time frame to search within to try and narrow those results to do this in batches, and the incident might be over a larger time frame.

    Shouldn't there be at least a secondary endpoint that returns a larger number of results?

    1
  • Igal Dar
    Comment actions Permalink

    Lowering the limit from 1000 to 100 per page, and max results is 1000 results (10 pages) is disappointing.
    It means many of the real-time data is maxed at 1000. Explore does not process real time data, leaving search, views, etc.

    I was very excited to read the previous announcement of maximum results per page 1000. It allowed us to simplify a lot of our custom code, eliminating much unneeded pagination. For example - a ticket can have more than 100 comments, but unlikely to have more than 1000 comments. If it has, only the last 1000 are relevant. One query would have gotten all relevant comments. Now we're rethinking pagination.

    It would be great if this can be overturned. Or have a way that curtain customers, with curtain offering would benefit from 1000 results per page and 200K results.

    0
  • Jesse Rubenfeld
    Comment actions Permalink

    I'm confused. People are talking about no pagination, but the notice above references a max results per page of 100 (out of the 1,000 total maximum results). So it looks like we do still need to use pagination, and there's the total results limit. Maybe the change has changed? Can you please clear up the confusion? Did you previously advertise all 1,000 results in a single page and now have changed it to 10 pages of 1,000? (If so, why?) 

    1
  • Alex Pereira
    Comment actions Permalink

    @Jesse Rubenfeld, that seems to be exactly what happened. They have changed the narrative of the original notice. Originally, it was 1,000 results in a single page, no pagination...

    1
  • Jesse Rubenfeld
    Comment actions Permalink

    Will the tenth 100-result page of a search with 3,254 results have a null "next_page" value?

    0
  • Salvador Vazquez
    Comment actions Permalink

    Thank you all for the questions and feedback. I wanted to give 2 quick updates on this:

    1. There was a shift to go back to 100 per page to maintain consistency and stability of the API
    2. Those who have had API requests over 1k in the last month will be getting some further communications in the coming weeks. 

    As always please keep posting on here and if needed we can setup some time to talk through concerns and use cases. For those worried about going back down to 100 per page what is your use case for getting so many at one time?

     

    0
  • Jesse Rubenfeld
    Comment actions Permalink

    Salvador, will the tenth 100-result page of a search with 3,254 results have a null "next_page" value?

    1
  • Greg Sohl
    Comment actions Permalink

    Salvador, API design based only on recent usage will be insufficient from a planning perspective, having made that mistake a few times in my 35 year career.

    1
  • Salvador Vazquez
    Comment actions Permalink

    @Jesse Rubenfeld: if you have 100 per page and request the 100th page you will still get it but if you request the 101th page that has the 1001th result value will return an error letting you know you have hit the limit. Whichever page has the 1001th result will give you that error.

    @Greg Sohi it is a Search API and many people use it to pull top relevant result which is not the most recent usage. If your use case is different than getting the top relevant results I'd love to hear it.

    0
  • Jesse Rubenfeld
    Comment actions Permalink

    Thanks, Salvador, but that doesn't quite answer my question. First, I said the 10th page, not the 100th page. (There won't be a 100th page if I understand this change correctly.) Also, what I'm asking about is whether the 10th (and final) page will have a "next_page" value that is none. That should be what happens, rather than referring to an 11th page that we cannot access. Since we have to code our integrations to accommodate the change without any opportunity to test, it would be great if you could clearly articulate this aspect of the response.

    1
  • Dustin King
    Comment actions Permalink

    @Salvador,

    At my own workplace we use the Search API (often above 1k results) for the reasons previously stated above, and here, to name just some of our use cases:

    When an issue in one of our products arises, we sometimes receive well over 1K tickets about the issue, and need to be able to search and find them all for triage, bug identification/solving, and most importantly for reply and remediation for our customers. Depending on the issue, we don't always have a specific time frame to search within to try and narrow those results to do this in batches, and the incident might be over a larger time frame.

    Also, I don't believe it's yet been answered by Zendesk whether or not this will actually impact the native search within the Zendesk Support tool as well, or if it will only affect API calls made outside of that. If that could be cleared up, it would help a lot.

    1
  • Bryan Flynn
    Comment actions Permalink

    @Jesse -- to be specific, there will be an actual link to the 11th page, but going to that link returns a 422 error:

    {"error": "invalid","description": "Invalid search: Requested response size was greater than Search Response Limits"}

     

    0
  • Jesse Rubenfeld
    Comment actions Permalink

    OK; it'd be much nicer of ZD not to return a link we can't successfully follow, but I guess I can work around it by counting the number of links I've already followed and stopping after the 9th.

    1
  • Bryan Flynn
    Comment actions Permalink

    Thanks for the feedback Jesse. This is a new pattern of behavior, so I'm sure there will be interest in this input and any other points that come up around this change.

    0
  • Igal Dar
    Comment actions Permalink

    @Salvador Vazquez  I am one of those interested in 1000 results per page.

    I am wondering, why not let the users make the decision.Ask us in a Setting page to choose between two options:
    * Fast performance - 100 results per page
    * Slow performance - 1000 results page

    Thanks.

    0
  • Shane Mitchell
    Comment actions Permalink

    I also am having to do a lot of work-around to not be negatively effected by this change.

    We have to pull tickets based on a custom field value, which already isn't the most desirable option as we have more than one.  We have to use the search API to do this, and now the only way for us to pull tickets will be to create daily aggregations to pull ticket data to in-house transactional databases then pull tickets by ID, or just store the full ticket on our end which is a major reason for having Zendesk to handle tickets.

    1
  • Jake Carville
    Comment actions Permalink

    Is anyone aware if this will affect the Zendesk Power BI Integration? (https://powerbi.microsoft.com/en-us/integrations/zendesk/)

     

    3
  • Rhi Hunter
    Comment actions Permalink

    Will this have any impact on the number of results we can report on in Explore and/or Insights?

    0
  • Bryan Flynn
    Comment actions Permalink

    Hi Rhi -- no, this will not impact those reporting platforms. Those platforms collect information through the incremental export APIs, not the Search API. Thanks for asking!

    0
  • Matthew Heflin
    Comment actions Permalink

    I have a few concerns.

    1. Impact Blind: I received the notification that we have API calls in the last 30 days that went over 1000 results. Your current API reporting under Admin Settings only shows number of requests so I can't tell what this new limitation will impact. None of the API calls were constructed by our staff so we can't troubleshoot client side. It would be helpful to have some idea of what is happening before stuff just breaks on the 15th.
    2. MS Power BI integration is essential - will it be affected?
    3. Is there any relation between this and the Apps Framework v1 deprecation? We used Tymeshift, which uses the API and also has a V1 App I'm assuming they will need to change their use of the API but didn't know if the two issues are related?
    1
  • Matthew Heflin
    Comment actions Permalink

    Still waiting on a response.

    0
  • Bryan Flynn
    Comment actions Permalink

    Hi Matthew.

    #1 -- Agreed. The API dashboard in Zendesk Support is limited. I'll open a private ticket for you on this point, since account access will be needed for any detailed analysis.

    #2 -- not sure -- the MS Power BI integration is outside of Zendesk's control. I'll check with our tech alliance folks on this one to see if they know anything. Regardless of what comes back, it will be up to the MS Power BI group to manage changes.

    #3/Tymeshift -- according to this announcement of theirs, they are updating the app today (10/10). It looks like they are doing it "in place", so anyone with the v1 version will automatically get the v2 version after a browser refresh.

    2

Please sign in to leave a comment.

Powered by Zendesk