Loading more than a 100 entries

[kristian (Kristian Johansen)]

I'm problems with loading more than a 100 entries.

I dispatching an action in nuxtInitServer, that calls a async method with a do while loop that increments the page number and return the data

 const getPages = async (context) => {
  const data = []
  let pageNum = 1
  let lastResult = []

  do {
    const resp = await context.$prismic.api.query(
      context.$prismic.predicates.at('document.type', 'artists'),
      {
        pageSize: 100,
        page: pageNum
      }
    )
    lastResult = resp

    data.push(resp.results)

    pageNum++
  } while (lastResult.next_page !== null)

  return data
}

But when I run "npm run generate" I get this error for some of my subpages.

FetchError: request to https://rtt2020.cdn.prismic.io/api/v2/documents/search?&page=1&pageSize=1&ref=X7KK4xIAACAAeYOk&q=%5B%5Bat(document.type%2C%20%22globals%22)%5D%5D failed, reason: read ECONNRESET
    at ClientRequest.<anonymous> (/Users/kristianjohansen/Sites/rtt2020.nuxt/node_modules/cross-fetch/node_modules/node-fetch/lib/index.js:1461:11)
    at ClientRequest.emit (events.js:314:20)
    at ClientRequest.EventEmitter.emit (domain.js:486:12)
    at TLSSocket.socketErrorListener (_http_client.js:469:9)
    at TLSSocket.emit (events.js:314:20)
    at TLSSocket.EventEmitter.emit (domain.js:486:12)
    at emitErrorNT (internal/streams/destroy.js:100:8)
    at emitErrorCloseNT (internal/streams/destroy.js:68:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21)

If I only load the first 100 it works. So guess there is something wrong with the function. Anyone knows how to load more than a 100 results?

Many thanks!

[samlittlefair]

Hey Kristian,

Thanks for posting this question! When I tested your function, I get an array of arrays. Here's the result of getPages() I get when I query all documents on a repo with 475 items:

However, it worked perfectly when I spread resp.results before pushing it to data:

lastResult = resp

data.push(...resp.results)

Let me know if that changes anything. If not, could you zip your project and send it to me, or share your GitHub repo, and also your Prismic repo name? (In a DM if you like).

Sam

[samlittlefair]

Thanks, @kristian! I got the DM with your info. I tried debugging this myself. I got the same errors, and couldn't figure them out. I'm not sure if it's a problem with your function. I was able to get all 185 documents using your function (with the spread operator) just fine. It seems to me like the error happens after the function runs — though I can't be sure.

I've escalated this issue to our dev team, and I'll let you know what I hear back.

Sam

[kristian (Kristian Johansen)]

Thanks very much Sam! I'm looking forward to hear back..

And thanks for an amazing product!

[kristian (Kristian Johansen)]

Hey Sam, I hope you had a good weekend.

Just want to check in if you have heard back from the dev team on this matter?

Thanks,
Kristian

[samlittlefair]

Hey Kristian,

No word yet, but I'll send a message to follow-up.

Sam

[samlittlefair]

Hey Kristian,

Thanks for your patience! My colleague took a look at your project, and identified a few ways to troubleshoot:

First of all, in ~/store/ui.js, you use a do/while loop to fetch your content. Do/while loops don’t run asynchronously, so that is likely causing some errors (more info here). You could replace it with a recursive function, like this:

const getPages = async (context) => {
  const data = []
  let pageNum = 1
  let lastResult = []
  const fetch = async () => {
    const resp = await context.$prismic.api.query(
      context.$prismic.predicates.at(‘document.type’, ‘artists’),
      {
        pageSize: 100,
        page: pageNum
      }
    )
    data.push(...resp.results)
    if (resp.next_page) {
      pageNum++
      await fetch()
    }
  }
  await fetch()
  return data
}

You’re querying your content in the store, which is not really what the store is designed for. Instead, I’d suggest that you refactor the queries out of the store and into your pages and components. You can use the asyncData() method on pages and async fetch() in components. That’s the recommended way of fetching data in Nuxt. That goes for content as well as all of the global settings that you query from Prismic — it’s more simple to query global settings on the page or component level. In general, I think using the store is likely overkill, and it seems like it could be causing some issues for you.

In particular, your nuxtServerInit() function gets called for every route that gets generated, which means that you’re generating your globals for every page, and so Nuxt makes the exact same call to the Prismic API 100+ times for your site settings. This could cause errors on your machine, which I think are causing the breaking issues that you’re asking about.

Finally, it looks like you’re using a slightly older version of Nuxt. The current version of Nuxt has great support for static site generation, and doesn’t require ~/modules/crawler/ or ~/modules/static/.

So, if you make the API calls on the page / component level, instead of in the store, I think you could simplify your app a lot. If you want to query more than 100 documents, I’d recommend a recursive function instead of a do/while loop. And it might also help to upgrade to the latest version of Nuxt.

Let me know if you have any questions about any of this, or if you need any resources to get it to work. Thanks again for posting your question and sharing your project!

Best,
Sam