Here are some steps to follow:

• (Assuming you have SQL Server installed and running locallay
• Create an empty database in SQL Server Management Studio
• In the modeler, in the Project Explorer expand the "Project" tree
• Double-click settings
• In the Configurations tab, click the New button
• In the Database tab:
  • Select Microsoft SQL Server under the Type dropdown
  • Enter the database name, URL (localhost:1433 usually for SQL Server)
  • Enter username and password or choose integrated security
  • Click OK
• Make your new configuration active by clicking the 'Make Active' button
• Start your app. Mendix will automatically create the data structures necessary to match your project's domain model.

This part of the how-to article on using Postgres locally may also be helpful.

Not sure if you want to connect to a SQL server and retrieve data or that you want your modeler to use a SQL DB as a default DB. for the last option,

You can configure this in your project settings, create a new configuration and choose the DB type you want.