Thanks, it seem the access token did it, still have some caching issues but nothing mayor, thanks for the help.
For anyone else that stumbles on this issue add an access token in prismic settings
Add it to the environment variables in netlify site configuration:
And then modify prismicio.js to use the access token:
this solved the issue for me.