Webhooks Intermittently Don't Fire

Dashboard / Writing room
#1

Hey Prismic support,

Is there something going on webhooks? We've been tracking an issue with our publishing pipeline for the last week or so, and it looks like there may be some kind of intermittent failure with Prismic's webhooks. Its difficult to tell for certain because the webhook logs don't have a timestamp, and it seems to be happening randomly, but every day or two a webhook seems to be failing to fire when a post is published. After some testing, we're confident that our publishing pipeline is working properly, but we've had several publishing events not hit our backend through the webhook we setup. Is there something going on with webhooks on your end that could explain this?

To give you a rough idea of what our publishing pipeline looks like, we have Prismic hit a webhook whenever a publish or release happens. This triggers a background job that fetches any new documents from Prismic's API, and then compares the results to what we have in our database and performs an upsert.

We've confirmed that our background job workers, the webhook endpoint, and our DB upserts are all working properly. Which leads me to the conclusion that the webhooks must be failing to fire for some reason.

Any help or guidance would be much appreciated.

3 Likes
#3

I've been having the same problem! While doing some testing a few days ago to verify the webhook workflow actually works I noticed occasionaly the webhook endpoint wasn't being hit. It's very intermittent. I had actually set up 2 webhooks to fire, one going to a server and one going to my local machine, and sometimes only 1 of the 2 endpoints was being hit.

This is a big problem for scheduled releases because no one would be checking/monitoring it the same way an editor could check the page got updated after he just published it.

#5

Hi there.

Can you provide us with your repository names and a screenshot of the logs of the failed events triggered (in a private message if necessary) in order to check our logs for any errors?

Looking forward to your reply,
Fares

#6

Hey @Fares, thanks for the response.

In my case the repository name is Taste. As for logs of failed events, there are none, that's kind of the problem. Prismic doesn't indicate that they're was a failed webhook event, it just doesn't hit the webhook at all. Like @martin.kzh mentioned, I too have two separate webhooks that fire on publish, and often when there's such a failure to trigger it'll trigger for one or the other, but not always both.

#7

I have managed to document this. The repository I did it on is https://mynewsite.prismic.io/, but I think this happens on any repository (I've seen it on a different one). I've used 2 webhooks to show the difference, but trigger failure potentially happens in any case.

Screenshots and explanation here: Prismic webhook test

(I've blacked out my local server IP)

P.S. The webhook logs really need timestamps!

1 Like
#8

Hi Martin,

Thank you for your contribution, and documenting this issue.
So I have create an issue in our tracker with all the details mentioned and forward it to our dev team.

And I find adding timestamps is essential for debugging issues, the I have created a feature request and forwarded it to the @activations-team.

#10

Hi @martin.kzh @devon.noeldetilly Could you figure out the type of trigger event that is being dropped? is it only document publishing?

#11

In my case it's publishing. I haven't tested with other events.

1 Like
#12

Hi @martin.kzh can you confirm if the issue is still happening?

#13

Hi Marc,

I don't have a live website that I can monitor for this, I have to manually keep saving documents lots of times and then verify after each save that everything has fired. Do you guys have no way to automate this and confirm?

I might try again at some point when I have some free time, but could be a while.

#14

Well actually I'm having a problem recreating the issue,
we would like to check out dead letter queue (it's cleaned out frequently), and some additional logs we have've added

#15

This issue has been closed due to inactivity. Flag to reopen.

#16
#17

Hi @devon.noeldetilly @martin.kzh or anyone who has this issue, we have added some logs to our system to track those failures, but it seems we can't catch any errors, are you still having this issue?

#19

Hi @Fares and @marc.mcintosh,

Yes, still happening. Brand new repository: https://romanopoly.prismic.io/. Two webhooks are set up, you can clearly see in the logs (top row) the latest trigger only happened on 1 of them (log for a trigger attempt isn't there, so no failure), whereas the previous 4 triggers are in sync.

#20

Thanks for sharing this info, I have added that to the issue.

#21

I am also experiencing this issue with my repositories.

#22

Hi Augie,

Thanks for letting us know. Please share your repository name to update the issue name (you can share it in a private message if necessary).

Best,
Fares

#23

If needed, flag to reopen.

#24

Hi there, this issue should now be fixed. Let us know if you're still having trouble with it.