C# Discussion Boards - CodeProject

16-Jul-23 9:47

My program inserts thousands of records into a SQLite database. That works fine.

I do it by building an INSERT statement with 1000 VALUE groups, each group representing a different record to add. If I were to execute a separate INSERT statement for each record, then the insert would take hours, so I make the compound INSERT statement.

So the question is, how can I transform the INSERT query into a parameterized query, if every VALUE group has unique values?

There are four fields that need to be populated for each record, so that means I must create a SQLiteCommand with 4000 parameters?

Or is there a better way?

The difficult we do right away...
...the impossible takes slightly longer.

Dave Kreskowiak16-Jul-23 11:44

Dave Kreskowiak

16-Jul-23 11:44

I don't think you have much of a choice. Sqlite doesn't support any bulk insert so you're either going to reuse the same command over and over again or build a block of inserts.

You also have a command length limit of 1GB but, unless you go nuts building a huge block of insert statements, you'll not likely hit that.

The other problem is a limit on the number of parameters you can have. The default using numbered named parameters is 32766 for recent versions of Sqlite.

Quote:
Maximum Number Of Host Parameters In A Single SQL Statement

A host parameter is a place-holder in an SQL statement that is filled in using one of the sqlite3_bind_XXXX() interfaces. Many SQL programmers are familiar with using a question mark ("?") as a host parameter. SQLite also supports named host parameters prefaced by ":", "$", or "@" and numbered host parameters of the form "?123".

Each host parameter in an SQLite statement is assigned a number. The numbers normally begin with 1 and increase by one with each new parameter. However, when the "?123" form is used, the host parameter number is the number that follows the question mark.

SQLite allocates space to hold all host parameters between 1 and the largest host parameter number used. Hence, an SQL statement that contains a host parameter like ?1000000000 would require gigabytes of storage. This could easily overwhelm the resources of the host machine. To prevent excessive memory allocations, the maximum value of a host parameter number is SQLITE_MAX_VARIABLE_NUMBER, which defaults to 999 for SQLite versions prior to 3.32.0 (2020-05-22) or 32766 for SQLite versions after 3.32.0.

The maximum host parameter number can be lowered at run-time using the sqlite3_limit(db,SQLITE_LIMIT_VARIABLE_NUMBER,size) interface.

Asking questions is a skill
CodeProject Forum Guidelines
Google: C# How to debug code
Seriously, go read these articles.

Dave Kreskowiak

Richard Andrew x6416-Jul-23 12:39

16-Jul-23 12:39

Thanks, just wanted to be sure I wasn't overlooking something. Thumbs Up | :thumbsup:

The difficult we do right away...
...the impossible takes slightly longer.

OriginalGriff16-Jul-23 12:24

OriginalGriff

16-Jul-23 12:24

SQLite isn't open to SQL Injection, because it doesn't support multiple commands in the same command^*: it doesn't support bulk inserts, even via a DataAdapter. it is meant to be a "lite" version, after all!

So if you build your command with multiple VALUE fields, it is technically safe (but needs to be commented in case someone later converts to MSSql / MySql)
I'd recommend using a transaction though, just to speed the operation up if nothing else...
This may help: https://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite[^]

* Unless you use the CommandText property instead of a SqlLiteCommand - then you are wide open to SQL Injection.

"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
"Common sense is so rare these days, it should be classified as a super power" - Random T-shirt
AntiTwitter: @DalekDave is now a follower!

Richard Andrew x6416-Jul-23 12:42

16-Jul-23 12:42

OriginalGriff wrote:
* Unless you use the CommandText property instead of a SqlLiteCommand - then you are wide open to SQL Injection.

Uh oh, that's what I'm using. I'm setting the CommandText property of a SQLiteCommand to the long INSERT statement.

I wasn't aware that there's any different way. Is there?

EDIT: I misread your post. What is the other way of using the CommandText property instead of a SQLiteCommand? (Just curious)

The difficult we do right away...
...the impossible takes slightly longer.

OriginalGriff16-Jul-23 19:34

OriginalGriff

16-Jul-23 19:34

No, the CommandText is a property of the SqLiteCommand object: SqliteCommand.CommandText Property (Microsoft.Data.Sqlite) | Microsoft Learn[^]

SqLite is just that: a "lite" version of SQL - it isn't meant for big jobs.
I'd be tempted to say that you might be better off using SQL Server if you are working with large databases - but that's your decision and I don't know enough about your system(s) to reccomend it.

Richard Andrew x6417-Jul-23 6:35

17-Jul-23 6:35

Anyway, I appreciate you pointing out that it's not susceptible to injection attacks. That was the reason for my question.

Thanks!

The difficult we do right away...
...the impossible takes slightly longer.

Gerry Schmitz16-Jul-23 20:32

Gerry Schmitz

16-Jul-23 20:32

Did you actually time the separate inserts? "Thousands" isn't very much and the server is "local". The usual consideration is don't do "mass inserts" with an index ... drop and build the index afterwards.

I've experience seconds, but never "hours".

Richard Andrew x6417-Jul-23 6:33

Getting a VST3 effect name and vendor

17-Jul-23 6:33

I might have used hyperbole, but the database will contain upwards of 4 million rows.

The difficult we do right away...
...the impossible takes slightly longer.

spice3d8-Jul-23 10:19

spice3d

8-Jul-23 10:19

Using Marc Jacobi's VST.NET2, I'm able to get the effect name and vendor for a VST2 DLL file, but an error is thrown, if I try it on a VST3 file. There must be an easy way to do this. I had read somewhere that VST3 files are basically just DLLs, but it appears there's a diff for what i'm trying to do. I've been searching and testing for days. Not a whole lot of C# examples. Seems C++ is used nearly exclusively.

Re: Getting a VST3 effect name and vendor

Pete O'Hanlon8-Jul-23 11:09

Pete O'Hanlon

8-Jul-23 11:09

Why not ask in the forums on his articles.

Advanced TypeScript Programming Projects

Re: Getting a VST3 effect name and vendor

spice3d8-Jul-23 12:18

spice3d

8-Jul-23 12:18

I will do that.

Advices for my learning path

coco2437-Jul-23 22:10

7-Jul-23 22:10

Hi,

I want to become a freelancer programmer able to take jobs posted
on freelancing sites or be part of a team where I can do at the begining small tasks for gaining experience.

For that I have started to learn C++ and then WINAPI and when I was asking here questions about my WINAPI isues the experienced guys here told me that I am learnig an old tool and a very nice member here that helped me a lot recommended me .NET field with the book of Charles Petzold - .NET book zero, that was an excelent book because it dosen't entered in details.

Now I have finished this book to, what tool you recommend me to learn next, what steps should I take
that help me gaining experience and to be able to start working on real life
project? I think that now would help me books or tutorials that are treating topics on surface.

I want to mention that also I had installed, connected and tested a local database trough c++ and some years ago I installed an local apache server and played with html, php.

Guide me please on what tools I have to learn next to be able to enter on the market, now I am working on another technical field and I want to gradually sustain myself from programming.

Thank you în advance for your help.

OriginalGriff7-Jul-23 22:19

OriginalGriff

7-Jul-23 22:19

If you have finished the book - and done all the exercises - then you are ready for a bigger, more in-depth book: Petzold is excellent, but only an introduction! Addison Wesley, Wrox, and MS Press all do excellent books on C#, but they tend to average around the 1000 page mark rather than 250 odd of Petzold!

Experience is the key: practice, practice, and then practice some more - C# is a pretty small language (you can learn the whole of it in an afternoon if you have solid C++ OOPs experience) ... but the .NET framework is enormous, and comes in several flavours now, as do the presentation framework part of it (WinForms, WPF, Blazor, Xamarin all target differently and what works in one doesn't work in the other!)

So decide what framework you will specialise in to start with, and get a book on that - then practice, practice, practice! You can always add a second presentation framework later when you have solid experience under your belt.

Good luck!

Richard Andrew x648-Jul-23 4:26

8-Jul-23 4:26

To support what Griff has already said, the thing you need to do most in order to enter the market as a professional developer is to create working applications. Find projects that pique your interest. There are plenty of articles here on CP that represent working applications.

Look at the CP home page. Set your filter to C#, and then browse the articles that it shows. Good luck!

The difficult we do right away...
...the impossible takes slightly longer.

coco2439-Jul-23 10:07

9-Jul-23 10:07

Thank you for your responses.

Blazor seems intresting because it is used for frontend and for backend development, this thing looks atractive I will take a look on it.

Thank you for your time.

jschell10-Jul-23 4:50

jschell

10-Jul-23 4:50

coco243 wrote:
Blazor seems intresting because it is used for frontend

I suggest javascript rather than that. There is more market for javascript.

BillWoodruff9-Jul-23 16:06

BillWoodruff

9-Jul-23 16:06

imho, you need to make decisions about:

1) type of app(s): windows, mac, linux ... or, app that runs on all those, or, only runs in browsers.

2) if you choose the .NET framework, language choice gets easier.

3) front-end, and/or back-end ? MS facility or non-MS backend tools ?

i suggest you start by using WinForms ... get familiar with interacting with controls using events ... understand inheritance (classes, subclasses), and using interfaces. study use of abstract and virtual.

There excellent books on learning C# WinForms.

imho, what you learn will help you move to another .NET language and/or .NET framework.

«The mind is not a vessel to be filled but a fire to be kindled» Plutarch

coco24310-Jul-23 19:34

10-Jul-23 19:34

When I finished to read the C# introduction book thats were the questions în my head, to start with UI windows programming or web developing?
The ideea is that I've spended a lot of time with C++ and WINAPI without
succeding entering in the freelanging programming market and now I wish to get the way that leads me to be able to work and make an income while I am adding more and more knowledge, I want to learn tools that help me not those that pushes me away from my goal.
Thank you

BillWoodruff10-Jul-23 19:58

BillWoodruff

10-Jul-23 19:58

coco243 wrote:
I want to learn tools that help me not those that pushes me away from my goal.

i've given you my best advice ... how long are you going to sit by the river, thirsty, thinking about what's the best water ?

Make choices, and get to work. Research skills most in use now, that companies are hiring entry-level positions for.

«The mind is not a vessel to be filled but a fire to be kindled» Plutarch

Richard MacCutchan10-Jul-23 22:55

Richard MacCutchan

10-Jul-23 22:55

Forget about freelancing for the moment, you are focusing on the wrong issue. Decide whether you want to do desktop or web apps and get learning and practicing on the skills you need for whichever path you choose. Then find a job that will increase your learning and experience and give you a solid grounding in the commercial world. Only then will you have the necessary attributes to consider looking for freelance work.

Gerry Schmitz18-Jul-23 7:03

Gerry Schmitz

18-Jul-23 7:03

I "reimaged" myself along your line of thought but scanned the web sites first to determine where the demand lay. I went for (small) fixed bids, to open hourly (no web cams or keyboard loggers, thank you) ... then took that experience and parlayed that into independent long term contracts. All remote. I focused on .NET at the time which has carried me for the last 15+ years.

I'd become an "AI expert" today ... but I think half of it is a con. Ethics.

coco24319-Jul-23 19:36