System crash after high offset

0
Is there any reason why a retrieve with an offset higher than 300.000 would crash the system? I have a MF that iterates over 800.000 records in batches of 250. After every 100.000 I perform an EndTransaction. when I reach 300.000 (or about) the application shuts down without any error. Initially I first thought it was because of an error in the MF but when I start the MF with 300.000 offset it goes down immediately 
asked
3 answers
2

I think you’re doing a batch process for your iteration process? But this amount of objects could be too much for Mendix.

I think what you can do is to try to run the process in multiple steps by reducing the amount of handled objects. 

What you also could do is to use the Process Queue Module and then run a batch process to create a amount of queued actions and retrieve in your queued action the related objects. So finally you will reduce the amount of objects handled in a single, independent transaction.

Good luck!

answered
0

Hi Willem, 

I know this module is good performing large import, maybe you find a hint here…

https://appstore.home.mendix.com/link/app/432/

 

Cheers, Andries

answered
0
  • Is it failing with Out of memory exception or something else? Did you had a chance to look at what is the error it is failing with. 
  • I noticed you are already doing batches
    • But, you are ending the transaction after 100.000 records. It is hard to say whether it is acceptable limit or more without knowing the size of the container you are dealing with. 
    • Again, it is more or less trial and error to find the optimal numbers for the infra you are working with.
    • If this is being done in the peak hours, the users might be using the system and next to the size of infra this variance also gets added up. 
  • One thing that does not addup is, you are retrieving, do you need all the data to be retrieved. Are you really retrieving or committing to database? 
  • Is it failing due to database being overloaded or system memory overloaded? Please look in for answers from the logs and metrics of Mendix.
answered