[BREAKING] New Search API Result Limits

Have more questions? Submit a request

76 Comments

  • tel-inform Entwicklung
    Comment actions Permalink

    A few days ago we requested an exception for this change, but until now we don't have any informations regarding our exception.

    @Brian Flynn:

    Could you please check this for us?

    0
  • Bryan - Community Manager
    Comment actions Permalink

    tel-inform Entwicklung -- I was informed that you should get a verification today.

    0
  • Gilad Shriki
    Comment actions Permalink

    When and where was this change announced? Assuming this post which is 4 days old is not the actual announcement, as this will never give time to developers to adjust their code...

     

    1
  • Chris Schaffer
    Comment actions Permalink

    As Gilad Shriki I would like to know if there was any other method of announcing this to customers.  It was only after our BI solution started showing major issues that we found this forum post and I have yet to be able to find this in release or update information.

    Given that there are replies to this thread from 5 months ago there was certainly time to inform in a more comprehensive manner (that is if it wasn't done, I am not an admin on our instance and so may not have visibility into other communications such as emails from your team should anything of that nature been sent out.)

    We are currently scrambling to get our datasets working again and it is frustrating to be unable to find clearer information about these changes.

    2
  • Greg Sohl
    Comment actions Permalink

    To get notified of breaking changes, you need to FOLLOW this section:

    https://develop.zendesk.com/hc/en-us/sections/360000007488-Breaking-changes

     

    Thankfully ZenDesk is (finally) understanding the full extent of what constitutes a Breaking Change, and is doing better at notifying of them and getting user comment within that section.

    0
  • Chris Schaffer
    Comment actions Permalink

    Greg Sohl - thank you for the reference!

    That seems .... non-optimal... but at least that is a place to track.  Again, thank you!

    0
  • Carlo Beckman
    Comment actions Permalink

     

    We had the same issue; we weren't aware of this change until it broke in our production environment. How difficult would it have been to do a query of customers who were going to become broken on October 15th and have sent proactive emails to them? Or, how about the account manager owning that communication for those of us that have them. Instead of creating customer delight, you are creating customer despair. On top of it all, I still don't know if our extension has been granted.

    1
  • Bryan - Community Manager
    Comment actions Permalink

    Just to close the loop on Carlo's points -- on top of confirming the granted extension, there has been further direct discussions with our support and product management groups. We do care about these points and know this has been a bumpy change.

    0
  • Dave Allison
    Comment actions Permalink

    Bumpy to say the least. Is it possible to get an exemption? This has, like it has for others, completely derailed our BI reporting. While other integrations were easier to get around, with out the ability to use Power Bi we might need to consider other options.

    1
  • JohnCollins
    Comment actions Permalink

    This is crazy, these limits are so low.  Why not emulate Salesforce, have a bulk method API that supports large pagesets at a slower rate? 

    I'm getting response limit errors on count queries, indicating that info about counts still returning was false btw.  We aren't particularly large, I can't imagine how larger clients are working around this. 

     

    0
  • Matt Savage
    Comment actions Permalink

    The scope reduction from 200k to 1k is enormous.  As others have mentioned, search API has legit uses beyond smaller / more recent data sets that might be more useful in the Agent UI. I would argue that the API & the Agent UI should have different rate limits/functions entirely, though I understand that's a hassle for Zendesk to engineer.

    I personally use it in lieu of a formal GDPR policy enforcement system for Zendesk.  Doing so requires that we search & return typically > 20k tickets each iteration of this script - those tickets are all 3+ years old, as we don't wish to retain tickets beyond their usefulness for privacy/data collection reasons.  As of this this change, I'll have to modify this to either:

    1. break this search into 20+ chunks to prevent returning > 1k results, then run it manually each time (ironically, also having to break those into 100-ticket sub-chunks for the bulk API endpoints)
    2. have my script figure out how to break these into < 1k chunks on the fly via some looping mechanism

    I'm not thrilled about either of those prospects, honestly.  My functions here are especially asynchronous, as the search & bulk deletions don't need to happen immediately - they just need to be identified & queued for batch processing.

    More generally, I can't believe there aren't common uses that would fall far beyond this threshold: listing all tickets handled by an agent in a year, all tickets around specific incidents/product lines/etc. for a month, etc.  This limit feels arbitrarily way too low.

    2
  • Bryan - Community Manager
    Comment actions Permalink

    Matt, John, and Dave -- I've passed along these recent comments to product management. Thank you. There is a process for getting an extension, but it starts through a conversation with you account representative. Please reach out to them.

    For others, too, please continue to pass along pain points and use cases -- they are being recorded and aggregated with others.

    0
  • Chris Plapp
    Comment actions Permalink

    Hi Bryan,

    I have to say that the process for getting an extension does not work.  We attempted to go through that process ourselves and were unsuccessful.  We had a conversation with someone in support and showed them how we were using the API and why we needed the extension.  It ended with being told that Zendesk could not make an exception.

     

    I echo the sentiment of everyone else here in sharing my extreme disappointment around this sudden change and the negative impact it has had on all of our business processes that relied on the ZD APIs.

    0
  • Bryan - Community Manager
    Comment actions Permalink

    Hi Chris,

    I notified your account's representative to help move an extension along. Because of the possible impact to allowing extensions, requests need to go through account rep's versus through regular support. Apologies for the hassle and pain around this.

    For others who are running into business process impact, please have a discussion with your account representative -- that will be the quickest route to an extension.

    0
  • Ian Goldberg
    Comment actions Permalink

    This is terrible. All of our zendesk reporting is built off of the search API, because it's the easiest for BI tools. We manage 60+ zendesks and have a massive contract with you guys and just started hitting the rate limits on some of our zendesks. We need an exception...

    1
  • Anthony Sessa
    Comment actions Permalink

    Ali Ayaz Alex Pereira -- https://develop.zendesk.com/hc/en-us/articles/360022563994/comments/360001980394

     

    Just to clarify - I do not think that this breaking change affects the `/users/search.json` endpoint.

    0
  • Kevin Goold
    Comment actions Permalink

    So it would appear that by doing this you have broken your Power BI connector too (the one that is shipped with Power BI).

    Could you give a timeline as for when you will fix this?

    5
  • Rob Tihanyi
    Comment actions Permalink

    200k search responses down to only 1k??? This is the Pharma Bro move of ticketing systems!!

    Come on Zendesk. You are better than this. I get that you had some limitations or constraints that you wanted to put in place, perhaps you had some resource issues to sort out.

    But to so massively reduce a key feature in one fell swoop is quite unbelievable.

    We are only a small company, and we run reports each month on our users to determine behaviours and measure performance. So we have a month where we add a couple of thousand users (yay!) but now we cant report on them? We have to guess at what points those users signed up in chunks of 1000 in order sort through them??

    And how is splitting a query into multiples any different from pagination anyway? Wont running 5 queries with 10 paginations end up having the same impact as running one query with 50 paginations? Confusing.

    2
  • Julio Garcés Teuber
    Comment actions Permalink

    Hi there

    I was checking the docs here and for large queries with more than 1,000 results the suggested Option 1 looks to work

    Let say I need some tickets based on a tag `some-tag`

    I will call api/v2/search.json?query=type:ticket tags:some-tag&order_by=created&sort=asc

    then, when we reach the 10th page and see still are new pages (or results count is more than 1000) I can start a new search reading latest ticket creation date

    eg: 2019-10-10T19:20:21Z 

    So my app can start a new query like this 

    api/v2/search.json?query=type:ticket tags:some-tag created>2019-10-10T19:20:21Z&order_by=created&sort=asc

    and when the 10th page is reached again we should repeat the logic above.

    What I found in some cases tickets or users can have the same creation date, so greater than or equal operator should be used

    api/v2/search.json?query=type:ticket tags:some-tag created>=2019-10-10T19:20:21Z&order_by=created&sort=asc

    and therefore a post-processing needed to cleanup duplicated results.

    So looks like with a few code changes API will continue working as usual and this makes me wonder: what is the intention here?

    I read in the original post "In order to align search usage with its intended scope"

    In Search API doc's say's:
    "The search API is a unified search API that returns tickets, users, and organizations. You can define filters to narrow your search results according to resource type, dates, and object properties, such as ticket requester or tag."
    Say's nothing about search small sets of results, so I am curious about what is the intended scope

    Thank you,
    Julio.

     

    3
  • Igal Dar
    Comment actions Permalink

    This has just hit us and it his us hard.

    I now assume we somehow got an extension for 3 weeks. Just a couple of hours ago, all searches got capped at 1000.

    We have multiple users, from technical to business users whose reporting is not working anymore.

    I wouldn't think it is a big issue if the search api would support granularity, but it's so basic that no matter what it has more than 1000 records to return.

    We're now in damage control mode.

    2
  • Jake Carville
    Comment actions Permalink

    Is anybody here using the Power BI connector and been able to get a solution for refreshing your reports you built?

    Mine have stopped refreshing since last week, tbh the refresh took ages! (even though in the grand scheme of things there isn't a huge amount of data), so I'd be keen to see if anyone has got a different solution that is a bit faster.

    2
  • Jorge Herrera
    Comment actions Permalink

    Well this was a terrible decision without providing a proper alternative, we are dealing with a major issue now as our automated process was feeding from the search API for QA purposes. Now we need to await for upper management to contact your Customer Relations Department and hope for a proper solution.

    Meanwhile we will have to resort to what Julio Garces has proposed or another workaround, but this goes totally against the new technology trends, furthermore this is not the only service we have with you guys that you are stripping down of previous working functionalities.

    If you may suggest an idea to get all solved tickets id within the day we will appreciate it

    Kind Rgds,

    1
  • Rob Tihanyi
    Comment actions Permalink

    Jorge Herrera If you are pulling tickets automatically at set intervals, and you just want to find those that have recently been solved, then perhaps you can take a look at using Incremental Exports instead?: https://developer.zendesk.com/rest_api/docs/support/incremental_export

    Unfortunately, this doesn't work for when you want to search large amounts of tickets or users for specific criteria. Like us.

    0
  • Lori Groger
    Comment actions Permalink

    Our Power BI refreshes started failing at 2pm on 11/11/19 with this error.  I've put in a ticket with Power BI but has anyone found a workaround?

    Web.Contents failed to get contents from 'https://****.zendesk.com/api/v2/search.json?page=11&query=type%3Aticket' (422): Unprocessable Entity Table: Tickets.

    1
  • Spencer Hutton
    Comment actions Permalink

    Our Power BI refreshes began failing around 10/15/19 with the same error:

    Web.Contents failed to get contents from 'https://*****.zendesk.com/api/v2/search.json?page=11&query=type%3Aticket' (422): Unprocessable Entity

     

    For almost a month, we have been unable to produce reports from PBI.  I have had to recreate metrics in excel and do a CSV export.  A HUGE headache.

    1
  • Dave Allison
    Comment actions Permalink

    Yes we have had to completely rebuild our Power Bi reporting using the CSV export. Incredibly frustrating. Would like to go back to the API connector if anyone knows how to fix it.

    1
  • Rob Tihanyi
    Comment actions Permalink

    What I would like to understand is how on earth this will benefit Zendesk in the short or long term? Its clear that it hasn't stopped anyone from running queries or trying to grab the data. Instead now people are resorting to writing essentially different forms of pagination (ie. write additional queries that run using the date of the final record the initial query returned) or dumping out data via CSV.

    If the aim is to reduce load on certain servers, it aint working.

    If the aim is to generate more revenue for Zendesk with a "new product" then can we at least get some clarity and transparency?

    And why go about it with such a sledgehammer approach anyway?

    2
  • Kevin Goold
    Comment actions Permalink

    I've reached out to Microsoft with respect to the Power BI Connector, and here's their reply:

     

    Hello Kevin,

     

    Thank you for raising a ticket on this issue.  The change you mention took the Power BI Team by surprise too.   However, the connector is just a passthrough for the API and therefore is constrained by the Zendesk API restriction.  We unfortunately have no way of resolving this at our end which I appreciate must be frustrating for you.  I would recommend you go back to Zendesk and ask for a clearer explanation as to why they think the connector needs changing when they have restricted their API.    Apologies we are unable to do more on this.

     

    Richard

     

     

    So are ZenDesk just going to say it's Microsoft's fault?

     

    0
  • Bryan - Community Manager
    Comment actions Permalink

    Hi Kevin Goold. This is no doubt frustrating.

    We're working on connecting with the right person at Microsoft. MS Power BI uses the /api/v2/search API to retrieve data. The Search API was not meant for bulk, repeated exporting of changed data -- the incremental APIs were designed for that. The Search breaking change unfortunately did indeed break this particular solution.

    As long as the MS Power BI integration uses the Search API, this breaking change will affect it. It needs to be modified to use the incremental APIs instead. Zendesk doesn't own or control the Power BI's connector code, however. We are working on partnering with Microsoft on getting this addressed and will post updates here.

    0
  • Salvador Vazquez
    Comment actions Permalink

    Hello,

    Although we do read and understand frustrations on this page we do urge everyone here to submit a ticket to Zendesk for their concerns so we can work with you individually and better track conversations and next steps.

    If your issue is with a connector like Microsoft or Domo I understand frustrations and although we're working with those partners and looking at options we recommend that you reach out to those third party companies to ask them to update their integrations for their products as well. 

    I would also like to clarify that these limits truly do make a difference in our performance and making more queries in smaller scales make a significant difference versus making calls into the deep pages of Search. In the end we just want to deliver a reliable and performant solution for all of you.

    0

Article is closed for comments.

Powered by Zendesk