CDN Timeouts at https://cazoo.cdn.prismic.io/api/v2/documents/search?

In the last few days we’ve experienced timeouts at this endpoint for a given document type. We set the timeout at 1.5s as we are querying the data for an isomorphic Next.JS application deployed via serverless. I don’t see why the requests should take more than 1.5s, I’m wondering why this happens.

Here are the time stamps:

More timeouts

Screenshot 2020-06-22 at 10.13.41

Hi David,

Thank you for reporting this to us, I will be happy to investigate this issue with you.

Can you tell us if the problem has been resolved after you did set the timeout to 1.5s?

Fares

Hey, So we previously had a bespoke timeout that fired after 1.5 seconds but we switched that using the timeout option provided by the prismic-javascript package.

We are still suffering timeouts because the response takes longer than 1.5 seconds. Is this normal behaviour?

@Fares Just to let you know we had over 100 requests between 3 and 4pm yesterday take over 1.5s to resolve. This led to a timeout (that we configured) and content not being shown on the page, affecting our SEO performance.

Hi David,

I’ve searched our logs and I found some timeouts, so I will create a ticket on our issue tracker and reach back to as soon as possible.

1 Like

Hey Fares,

Just a quick update to say we had another spate of timeouts yesterday between 14:00 and 17:00. Like the above, these are incidents where we have terminated the connection with Prismic because the request exceeded >1.5s.

Tom

Hi Tom,

I’ve updated the issue and we will reach to once we have more info about this.

Hi,

I have started experiencing timeouts again this afternoon. Nothing changed on our end to impact/cause timeouts. Please find the timestamps in the screenshot. Could you please help on the issue ?

1 Like

Hi @Fares,

We’re again seeing a large number of timeouts, starting ~13:45 up top now.

From ~15:15 the number of requests which were timing out significantly increased.

No changes our side - would you be able to let us know when the issue might be resolved?

Thanks,

Tom

1 Like

Thank you for letting us know, I will update the ticket with the new info.

Hey @Fares, it’s been quite some time, and numerous timeouts. You have updated the ticket, which we appreciate but I don’t think it should be taking this long to get some resolution to our degraded performance. You mentioned 2 weeks ago there were timeouts on your end. Is this something we should just expect and handle, or is there something (other than utilising the timeout) we can do to mitigate brownouts from the Prismic API.

I would appreciate more of a response than just updating the ticket each time.

Hi David,

Thank you for your patience, I have already brought this issue multiple times in our internal meetings, and we are still investigating this issue.

In fact I’m currently in contact with our devOps team and they are investigating this issue.

I will get back to you as soon as I get more info,
Fares

1 Like

Thanks @Fares, really appreciate it.

Hey David,

So the dev team is still investigating the issue.
We have noticed that only 0.3% of requests goes over 0.8s that still doesn’t meet our standards and we need to continue investigating the issue.
For that, we have already moved you to another cluster.