# Pagination

> How list operations page their results. Covers page and limit, defaults and caps, the pagination object, ordering, and loops that safely fetch every page.

Every list operation returns its results one page at a time. You choose the page with `page` and its size with `limit`, and every list response tells you how many records and pages there are in total. This page explains the rules, shows loops that fetch everything, and covers what to watch for when the data changes while you page.

## Which operations are paginated

| Operation | Path | Order of results |
|---|---|---|
| [listCountries](https://hub.main-team.org/api/reference/list-countries) | `GET /v1/country` | No fixed order |
| [listGrades](https://hub.main-team.org/api/reference/list-grades) | `GET /v1/grade` | The platform's own grade order (oldest record first), not alphabetical |
| [listOrganizations](https://hub.main-team.org/api/reference/list-organizations) | `GET /v1/organization` | No fixed order |
| [listStudents](https://hub.main-team.org/api/reference/list-students) | `GET /v1/student` | No fixed order |
| [listOrgStudents](https://hub.main-team.org/api/reference/list-org-students) | `GET /v1/<organizationId>/student` | No fixed order |
| [listExamCategories](https://hub.main-team.org/api/reference/list-exam-categories) | `GET /v1/<organizationId>/exam-category` | No fixed order |
| [listExams](https://hub.main-team.org/api/reference/list-exams) | `GET /v1/<organizationId>/exam` | Soonest sitting first, then by `_id` |
| [listApplications](https://hub.main-team.org/api/reference/list-applications) | `GET /v1/<organizationId>/application` | No fixed order |
| [listExamApplications](https://hub.main-team.org/api/reference/list-exam-applications) | `GET /v1/<organizationId>/application/exam-applications/<examId>` | No fixed order |
| [listStudentApplications](https://hub.main-team.org/api/reference/list-student-applications) | `GET /v1/<organizationId>/application/student-applications/<studentId>` | No fixed order |
| [listStudentCertificates](https://hub.main-team.org/api/reference/list-student-certificates) | `GET /v1/<organizationId>/certificate/<userId>` | No fixed order |
| [listStudentReports](https://hub.main-team.org/api/reference/list-student-reports) | `GET /v1/<organizationId>/report/<userId>` | No fixed order |

"No fixed order" means exactly that: do not rely on a record's position, and do not assume the newest record comes first or last. Identify records by `_id`.

Not paginated:

- **The exam picker**, [listAvailableExams](https://hub.main-team.org/api/reference/list-available-exams). One student's options are a small tree (category, then sitting, then language), so it comes back whole in one response.
- **Single reads and downloads.** They return one record or one file.

## The two parameters

| Parameter | Default | Smallest | Largest |
|---|---|---|---|
| `page` | `1` | `1` | No upper limit. A page past the end is simply empty |
| `limit` | `20` | `1` | `100` |

```bash
curl -sS "https://api.main-team.org/v1/student?page=2&limit=100" \
  -H "Authorization: Bearer $TOKEN"
```

The API is forgiving with both parameters. It never answers `400` for them. It uses the nearest value it can:

| You send | The API uses | Why |
|---|---|---|
| Nothing | `page=1`, `limit=20` | Defaults |
| `limit=500` | `limit=100` | Capped at the largest |
| `limit=0` or `limit=-5` | `limit=1` | Raised to the smallest |
| `limit=abc` | `limit=20` | Not a number, so the default |
| `limit=10.9` | `limit=10` | Whole numbers only; the fraction is dropped |
| `limit=1e2` | `limit=1` | Only the leading digits are read, so write numbers out in full |
| `page=0` | `page=1` | Raised to the smallest |

Because a bad value is corrected silently, check `pagination.limit` in the response if you are not sure what you asked for. It always shows the values the API actually used.

## The pagination object

Every list response carries a `pagination` object next to `data`:

```bash
curl -sS "https://api.main-team.org/v1/student?page=2&limit=2" \
  -H "Authorization: Bearer $TOKEN"
```

This response shows each student with only a few of its fields:

```json
{
  "success": true,
  "message": "Students fetched successfully.",
  "data": [
    {
      "_id": "652f1c9b8e4b2a0012a3c4d5",
      "username": "XXK1042",
      "firstName": "Jane",
      "lastName": "Doe",
      "grade": { "_id": "5f1a2b3c4d5e6f7a8b9c0d1e", "name": "10" }
    },
    {
      "_id": "652f1c9b8e4b2a0012a3c4d6",
      "username": "XXB1043",
      "firstName": "Arben",
      "lastName": "Hoxha",
      "grade": { "_id": "5f1a2b3c4d5e6f7a8b9c0d1f", "name": "11" }
    }
  ],
  "pagination": { "page": 2, "limit": 2, "total": 7, "totalPages": 4 }
}
```

| Field | Meaning |
|---|---|
| `page` | The page you received |
| `limit` | The page size the API used, after any correction |
| `total` | How many records match in total, across all pages |
| `totalPages` | `total` divided by `limit`, rounded up |

Edge cases:

- **Nothing matches.** `data` is `[]`, `total` is `0` and `totalPages` is `0`.
- **The last page** holds whatever is left. With 7 records and `limit=2`, page 4 has one record.
- **A page past the end**, such as `page=9` above, returns `data: []` with the same `total` and `totalPages`. It is not an error.

## Fetching every page

Use `limit=100` whenever you want everything: it takes the fewest requests. Keep asking for the next page until you have received page `totalPages`, and stop early if a page comes back empty.

With curl and [jq](https://jqlang.github.io/jq/), writing one student per line:

```bash
page=1
while : ; do
  resp=$(curl -sS --fail-with-body \
    -H "Authorization: Bearer $TOKEN" \
    "https://api.main-team.org/v1/student?page=$page&limit=100") || { echo "$resp" >&2; exit 1; }

  echo "$resp" | jq -c '.data[]' >> students.ndjson

  total_pages=$(echo "$resp" | jq '.pagination.totalPages')
  count=$(echo "$resp" | jq '.data | length')
  if [ "$count" -eq 0 ] || [ "$page" -ge "$total_pages" ]; then break; fi
  page=$((page + 1))
done
```

Node.js (18 or later). An async generator that yields one record at a time and pauses when the operation's [rate limit](https://hub.main-team.org/api/rate-limits) budget runs out:

```js
const BASE = process.env.MTO_API_BASE ?? 'https://api.main-team.org/v1';
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

export async function* everyRecord(token, path) {
  for (let page = 1; ; page += 1) {
    const sep = path.includes('?') ? '&' : '?';
    const res = await fetch(`${BASE}${path}${sep}page=${page}&limit=100`, {
      headers: { Authorization: `Bearer ${token}` },
    });

    if (res.status === 429) {
      await sleep(Number(res.headers.get('retry-after') ?? 60) * 1000);
      page -= 1; // ask for the same page again
      continue;
    }
    const body = await res.json();
    if (!res.ok) {
      throw new Error(`${res.status} ${body.error?.code}: ${body.error?.message} (${body.error?.request_id})`);
    }

    yield* body.data;

    const { totalPages } = body.pagination;
    if (body.data.length === 0 || page >= totalPages) return;

    // Out of budget for this operation: wait for the window to reset.
    if (res.headers.get('x-ratelimit-remaining') === '0') {
      await sleep(Number(res.headers.get('x-ratelimit-reset') ?? 60) * 1000);
    }
  }
}

// Usage: every student, then every application on one organization.
for await (const student of everyRecord(token, '/student')) {
  console.log(student._id, student.username);
}
for await (const application of everyRecord(token, `/${organizationId}/application`)) {
  console.log(application._id, application.user?.mainId);
}
```

PHP (8.1 or later). A generator with the same behavior:

```php
<?php
function everyRecord(string $token, string $path): Generator
{
    $base = getenv('MTO_API_BASE') ?: 'https://api.main-team.org/v1';
    $page = 1;

    while (true) {
        $headers = [];
        $sep = str_contains($path, '?') ? '&' : '?';
        $ch = curl_init("{$base}{$path}{$sep}page={$page}&limit=100");
        curl_setopt_array($ch, [
            CURLOPT_HTTPHEADER => ["Authorization: Bearer {$token}"],
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_TIMEOUT => 30,
            CURLOPT_HEADERFUNCTION => function ($ch, $line) use (&$headers) {
                $parts = explode(':', $line, 2);
                if (count($parts) === 2) {
                    $headers[strtolower(trim($parts[0]))] = trim($parts[1]);
                }
                return strlen($line);
            },
        ]);
        $raw = curl_exec($ch);
        $status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
        curl_close($ch);

        if ($status === 429) {
            sleep((int) ($headers['retry-after'] ?? 60));
            continue; // same page again
        }
        $body = json_decode((string) $raw, true);
        if ($status >= 400) {
            $e = $body['error'] ?? [];
            throw new RuntimeException("{$status} " . ($e['code'] ?? '') . ': ' . ($e['message'] ?? '') . ' (' . ($e['request_id'] ?? '-') . ')');
        }

        yield from $body['data'];

        if (count($body['data']) === 0 || $page >= $body['pagination']['totalPages']) {
            return;
        }
        if (($headers['x-ratelimit-remaining'] ?? null) === '0') {
            sleep((int) ($headers['x-ratelimit-reset'] ?? 60));
        }
        $page++;
    }
}

// Usage
foreach (everyRecord($token, '/student') as $student) {
    echo $student['_id'], ' ', $student['username'], PHP_EOL;
}
```

## When the data changes while you page

Pages are counted from the start of the list each time you ask: page 3 with `limit=100` means "skip 200 records, then take 100". The API keeps no cursor between your requests. If records are added or removed while you are paging, the rest of the list shifts:

- **A record removed** from an earlier page moves every later record up by one. The first record of your next page was on the page you already read, so you **skip** one record.
- **A record added** to an earlier page moves every later record down by one. You **see one record twice**.

How to cope:

- **Deduplicate by `_id`.** Always, on every full walk.
- **Walk quickly.** The shorter the walk, the less can change during it. Fetch pages one after another, not spread over an hour.
- **Do not delete on absence.** If a record you hold did not appear in a walk, fetch it by id before you conclude it is gone. Paging may have skipped it.
- **Re-walk for completeness.** For a nightly sync, a second walk that finds nothing new is good evidence you have everything.
- **`total` is a snapshot.** It was counted when the page was served, and the next page can have a different total. Do not stop just because you have collected `total` records; stop on the page rules above.

The exam list changes by itself over time. [listExams](https://hub.main-team.org/api/reference/list-exams) returns only exams still open for application, so an exam drops out as its sitting approaches or when applications close. That is also why its order (soonest sitting first) can move records forward between two walks.

## Totals count only what you can see

`total` counts the records you are allowed to see, never the whole platform:

| List | Counts |
|---|---|
| [listStudents](https://hub.main-team.org/api/reference/list-students) | Your own students |
| [listOrgStudents](https://hub.main-team.org/api/reference/list-org-students) | Your own students who have access to that organization |
| [listApplications](https://hub.main-team.org/api/reference/list-applications), [listExamApplications](https://hub.main-team.org/api/reference/list-exam-applications) | Applications of your own students who have access to that organization and have signed in to it |
| [listStudentCertificates](https://hub.main-team.org/api/reference/list-student-certificates), [listStudentReports](https://hub.main-team.org/api/reference/list-student-reports) | That student's **released** documents only (see [Certificates and reports](https://hub.main-team.org/api/guides/certificates-and-reports)) |
| [listCountries](https://hub.main-team.org/api/reference/list-countries) | Countries that can be selected; some cannot, and are left out |
| [listExams](https://hub.main-team.org/api/reference/list-exams) | Exams open for application now |

## Budgeting pages against the rate limit

Each operation allows your account 100 requests per 60 seconds, counted on its own (see [Rate limits](https://hub.main-team.org/api/rate-limits)). At `limit=100`, one list operation can therefore deliver up to 10,000 records a minute. Some typical numbers:

| Task | Requests | Minimum time |
|---|---|---|
| All 2,350 of your students | 24 pages of `GET /v1/student` | Well under a minute |
| All applications on one organization, 8,000 rows | 80 pages | Under a minute |
| Certificates for 1,200 students on one organization | 1,200 calls to `listStudentCertificates` (one or more per student) | About 12 minutes |
| The same for two organizations | 2,400 calls to the same operation | About 24 minutes: one operation shares one budget across organizations |

Per-student lists cost one request per student, whatever `limit` is, so plan those jobs by the number of students, not by the number of records. A different operation, such as the certificate download, has its own budget and can run alongside. [Collect results](https://hub.main-team.org/api/tutorials/collect-results) builds a complete nightly job on these numbers.