OData List and use of Batches

0
Dear all, I know that for a better performance when retrieving a lot of data from database and processing it, it is better to use Batches (https://academy.mendix.com/link/module/21/lecture/85/3.3-Using-Batches) and retrieve for example 250 items at once, and than another 250 and so on. In my case I have an OData API and receiving about 2800 items. I am using the “$skip” and “$top” and “$orderby” parameters to retrieve 100 items at once and process it in an microflow (let’s called it “refresh list”). The microflow “refresh list” ends as soon as the OData API call sends an empty list. Refresh List microflow looks like this: Get 100 Items (produces memory cache) Process 100 Items (which for sure also creates extra memory cache) Commit 100 Items to the database (now I thought the memory should be empty) Continue with step 1 as long the list ist not empty, otherwise end microflow   What I also remember is that I might read in the community forum that the microflow has to end so that the memory cache gets emptied. How could I do that in my case? Would it be enough to call the relevant activities as a submicroflow?        
asked
2 answers
0

Dear Omer, 

Assuming you are calling an odata service somewhere external and not providing it..

Is your microflow completing? If it does then the garbage collection will do the clean-up later. 

 I see your cpu is > 100%, thats interesting. 

Could it be that you're retrieves are looping? 

Anyway maybe rephrase what problem you are facing and add a picture of your microflows...

Kind regards

Jacob

answered
0

What i normally do is add 'EndTransaction' and 'StartTransaction' (available in CommunityCommons) after each commit before returning to the retrieve action and process the next batch. This way the memory will be released after each commit and the MF will not slow down. You can test the behavior locally by downloading visualVM and connecting it to your SDK. Within the program you can check a graph which shows the used memory within a thread.

Another cleaner option would be to process the batches in a queue, this takes some more time to setup but you won't need the Java actions. GL! 

answered