.NET – How to quickly save a lot of XML data into my database?

By XML

I used .NET to parse an XML file of about 20 million rows (1.56 GB), build a LINQ object from the data, and then insert it into a SQL database. This took a long time Time.

In order to improve performance, I am thinking of requiring pipes to separate files. I also want to know if Perl will be faster. Does anyone have suggestions for speeding up this process?

This is a radical idea, honestly I don’t know if it will improve yours Performance, but it has a good chance of doing so. I bet you instantiate your context object once and then use it to insert all records, right? Doing this means that the context will track all these objects until it is disposed, which may explain the performance degradation over time.

Now, you can clear the context cache, but I have one A firmer idea. These context objects are designed to have minimal overhead when instantiated (in any case it is said in the documentation, I have not tested the assertion), so if the context is instantiated in each iteration, it may Would help. That is to recreate the context while creating the object. Or, it is a better idea to maintain an internal list of data objects and submit the list to a separate method for submission to the database every n iterations The method should be instantiated and deal with the context. Reasonable?

I use .NET to parse an XML file of about 20 million rows (1.56 GB), build a LINQ object from the data, and then insert it into a SQL database. This It took a long time.

In order to improve performance, I am thinking of requiring pipes to separate files. I also want to know if Perl will be faster. Does anyone have suggestions to speed up this process?

This is a radical idea, honestly I don’t know if it will improve your performance, but it has a good chance to do so. I bet you instantiate your context object once and then use it to insert all records, right? Doing this means that the context will track all these objects until it is disposed, which may explain the performance degradation over time.

Now, you can clear the context cache, but I have one A firmer idea. These context objects are designed to have minimal overhead when instantiated (in any case it is said in the documentation, I have not tested the assertion), so if the context is instantiated in each iteration, it may Would help. That is to recreate the context while creating the object. Or, it is a better idea to maintain an internal list of data objects and submit the list to a separate method for submission to the database every n iterations The method should be instantiated and deal with the context. Reasonable?

, , , , , ,

Leave a Comment

Your email address will not be published.