Filemaker getting much slower when importing data into larger database

In my Xojo app emails can be written to Filemaker. I write an xml file which is then imported by Filemaker. The larger the Filemaker database is the slower the import becomes. Does anyone have an idea how to speed Filemaker up? Improving indexing?

Emails imported into a Filemaker database with 20k records:

Time started: 2020-10-01, 13:39
Time finished: 2020-10-01, 13:50
Time elapsed: 11 minutes 27 seconds.
No date range selected.
Archived 156 emails for the email client “Mail”.
Skipped 151 duplicate emails for the email client “Mail”.

Emails imported into an empty database:

Time started: 2020-10-01, 13:52
Time finished: 2020-10-01, 13:52
Time elapsed: 53 seconds.
No date range selected.
Archived 156 emails for the email client “Mail”.
Skipped 151 duplicate emails for the email client “Mail”.

The import is 10times slower with a larger database.

Use MBS filemaker plugins ?

Using MBS is not possible here, unfortunately. The licensing terms for Filemaker don’t fit. The Filemaker database is open-source.

can you import to a temp empty table, then append that table to the master??

1 Like

Very good idea, @DaveS, I’m going to test that.

For a while, importing LARGE files into FM slows down as time passes.

I don’t think Dave’s idea will solve the slowdown. It makes sense though!

One way to speed it up is to split the import file into x files and import each one sequentially. In the past, I’ve copy pasted the entire import file into an FM text field. Then wrote a script to process the import. One way would be to parse it manually, but even better would be to grab x lines from the FM text field, export that text, then import as records.

Yeah, it’s dumb, but if you need more speed…

The import is for one email already. The files I tested were typical emails. The html of the message body may be larger than typical because I add image previews to the message body.