Glossary
Pagination
Reading a list that is split across pages, by following the next link until there is none or a limit is reached.
A paginate step extracts the same schema from each page and joins the lists. It stops at max_pages, when a condition you set holds, or when the next button is gone, so a site that adds a thousand pages overnight cannot run up your bill.
example
{
"paginate": {
"next": {
"role": "link",
"name": "Next"
},
"max_pages": 20
},
"extract": {
"rows": {
"selector": "tbody tr",
"many": true,
"fields": {
"id": "td:first-child"
}
}
}
}