Retrieve huge amount of data from REST (get) call API

0
I am trying to retrieve a bout 50,000 lines of data, which is data for different cities. as data is too huge API gets error after 20 minutes. Is there any method by which i can call the REST Api and just get data for one city. i means limiting the retrieve rate. My purpose is to call API for each city one by one or when need. 
asked
2 answers
2

Assuming that the api supports pagination a solution couldbe like;

- create a process queue mechanism (see appstore) for the 50k of data, whereby each queued action retrieve 1k over the api or per city (or what your system could handle) and persist it to your db, whereafter you can do your own custom processing

 

 

 

answered
0

You could also try retrieving the cities data via OData. That technique can usually handle a larger dataload.

answered