Splitting microflow activities into parallel sequences
0
I'm doing a migration of database data and files from app A to B and during my loadtest of ~2000 files, I noticed it took quite some time to run (~50 files (e.g. 50 requests) per minute. In a batch job of 2000 files, it'll take (2000/50) minutes. In this case, it simply goes down a list of files and does a “GetById” rest call for each file, e.g. “Get file 1, Get file 2, Get file 3 (...)”. Once the whole list is done, it continues. It would be nice to seperate that batch. For say, it splits up the list per 100 items (thus 20 batches) and retrieves the files parallel simultaneously and once all files are retrieved, it then continues, as if all calls would be done sequentially. I understand Mendix microflows run like “Step 1, step 2, step 3, and so on”, but it would be nice to split the same step across multiple threads to speed up (large batches.
asked
Sander Schaeffer
2 answers
3
How about using the process queue? You could create jobs for the queue and tell your queue that it should run 4 or 5 in parallel.
Process queue can be found in the appstore. I use it a lot and it’s perfect for batch processing like this.
answered
Andreas Blaesius
1
As Andreas suggested, a queue is probably the best way to handle this.