Question

Bulk Data Migration from On premises to Mendix Database

0

Hello. We are developing a App that requires huge amount of data to be migrated from On prim database to Mendix database, We will need to do an Initial load and then do hourly sync up. Have 2 questions on this:- Initial Load from On premises to Mendix Database –> Based on discussion with Mendix team, they suggested to use excel import feature to load data but wanted to check if there is some other better approach. if you have experience of loading 1 million or more records , please provide some pointers. We are thinking of disabling indexes and loading data. Sync data every hour between On Prim and Mendix DB –> As per mendix team, they have recommended to use rest services to sync data. We might have a scenario where bulk change on Premises data and if we use rest approach, then we will have to call Rest services in batches . e.g. if 10000 records changes, then we will have to 100 rest service calls ( per call processes 100 records) . Is there a way to create bulk APIs in Mendix that can process more number of records. Can anyone suggest any best approach of if there is any mendix document that describes bulk API or SLAs. Thanks, Thanks.

asked 2019-09-19

rajesh khatri

3 answers

Andreas Blaesius · Answer 1 · 2019-09-19

Do you have the option to directly connect to the database? If so, you could use the Database Connector module from the app store.

If the amount of data is too big, rest is not an option. We had the same issue on a data migration we just did some time ago. Rest was too slow and needed too many ressources. Excel import is also not an option if your excel sheet has too many lines.

A database based migration can be the best approach on gughe data amounts.

rajesh khatri · Answer 2 · 2019-09-19

Thanks Andreas for the response.

Initial load using database connectors –> As per my understanding, database connectors can only be used within Microflow. So you suggest to retrieve data in a microflow and populate mendix tables.

Ongoing data sync – we cannot use database connectors due to security reasons and will have to go with rest services. With rest do you know if Mendix supports Bulk APIs.

Thanks

rajesh khatri · Answer 3 · 2019-09-19

Thanks. I will check on Process queue module.

Bulk API – > To add more information, as mentioned we have to sync data between On Premise ERP to Mexdix app. We are thinking of writing data from On Premises to Azure Cloud DB ( our Mendix runtime will be deployed on azure) , and then write a Logic app to invoke MEndix REst Service to write data to Mendix tables ( ERP DB→ Azure Cloud DB → Mendix DB).

I was going through mendix documentation ( Publish rest services) and I couldn’t see any option of specifying number of records a rest service can handle or there is no Queue in Mendix which we can use to write streams of data which we can be used to write data to mendix DB. Do you know if we can write a microflow to consume data from a Queue sitting on azure cloud.