Yes, it is normal when using a Static site generator such as Next.js. This happens because the Prismic Toolbar script, which is the window that connects your project to Prismic's built in support for hot reloads, is always expecting to detect new queries, but it stops listening when all pages are built and it only refreshes the page when, as you said, a new preview session is opened from the writing room.
Now, if you've already done all the steps for configuring the preview in your project you may have noticed that Next.js has its own mechanism for storing the state of the preview ref (Preview mode) which is required to bypass Static Generation and successfully load the most recent changes from a new preview session.
This value is primarily used to query the API again using the preview ref but you can also use it to make the Toolbar subscribe to any ref changes and replay prismic queries on demand.
To do this you'll need to add react hooks to work with the Toolbar. Luckily, we're just about to release a project where you can see an example of how to do it.
Here you can see the project code, and here's the file that makes up the hot reload.
Here's a gist where you can see an example of the function being used in a page component
If you have any questions about this or anything else, please let us know
Thanks