|
Have you seen (or read) this book: Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems[^]
I read about half of it and it was so deeply technical I couldn't even believe it.
People all over the Internet _say_ they're reading it or have read it, but I couldn't complete it.
i'm not that smart though. I gave up because it explained Databases etc. in such a deep way it went flying over my head. But, that is probably easy. 
Check it out, because I think it may be what you're looking for.
|
|
|
|
|
|
I haven't heard of any, but you might find that different RDBMS vendors handle parallelization of large queries differently.
I know how to search millions of rows in parallel on SQL Server in T-SQL, including how to configure the database system and the hardware it runs on (important!).
I'd be hard pressed to tell you how using Oracle or Postgre. I don't even know if they have "distributed partitioned views" or what they would call them.
My point is you might need to look at books that target a particular flavor of database engine.
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
Any book I've read would be dated by 20 years. But, I can give some points from real world experience...
For a SQL/RDBMS
- Do not normalize past level 3 for most things. It's not worth the performance hit.
- Do not use old school Hungarian notation like
tblUsers for names. - Learn what an index is and how to properly use it (hint, they help with reads).
- Learn when to use a flattened/denormalized table (hint, when speed is a must or for a reporting db/table).
- If you're in MySQL, learn the different storage types.
- Learn how to use diagrams well.
- For all things pure, make sure you get a proper handle foreign key constraints to help with data integrity.
- Learn when or when not to use a compound key (hint, many to many).
- Speaking of, learn when to use a many to many relationship (hint, clickety).
- Learn the pros and cons of building a self-referential table (hint, when building a hierarchy of data).
- Lean when to use views (hint, permissions and data access restrictions, read only denormalization).
- Understand why concepts like code-first DB design is only purported by those who don't understand proper DB design.
 - Understand that by their design relational DBs are slower than non-relational ones.
- Learn to use triggers to automate some things that would otherwise be tedious.
- And for all things pure again, get a grasp on transactions and connection pools.
- Learn to use SQL profiling tools, these will be your BFF.
There may be a few more points I didn't think of of the top of my head, but mastering those will put you ahead of most peeps.
For a NoSQL
- Learn when to use one. If you need to do semi-realtime anything, NoSQL is awesome. So, stuff like logging is perfect for it.
- They make a great reporting DB if you want to use this rather than a flattened table in your relational DB above.
- They don't use schemas (for the most part) so the learning curve is a breeze compared to SQL.
- Do not use them for mission critical data where it's ok to run a bit slower when data integrity is paramount.
Don't follow the buzzword/hype train. NoSQL is awesome but there's a time and place for both relational and non-relational DBs. NoSQL is super fast and that's its main benefit, but that's not always the only concern.
Jeremy Falcon
modified 24hrs ago.
|
|
|
|
|
Side note on Triggers: Know WHEN to use triggers. They are a performance hit. It may be negligible, so always load test.
|
|
|
|
|
FWIW, since you mentioned UI/UX specifically and dealing with large data sets, reporting and datagrids will likely be things you will need. Pay the one-time fee for a suite of quality data-centric libraries that are built to handle large data...pagination/asynch ops/aggregates/etc.
The suite we use is fantastic and well worth the expense...libs work in windows and web development. I won't 'name names' unless requested, but I believe they are a sponsor here. Also, they have tons of tutorials and demos.
"Go forth into the source" - Neal Morse
"Hope is contagious"
|
|
|
|
|
for me it's like this, on a form that shows a list from a table, if this list can be longer than a certain amount of rows, I don't allow fetching the rows unless the user has entered one or more filters than I can use in the where clause, otherwise I just fetch them all
|
|
|
|
|
Maximilien wrote: books ... with large databases ?
I thought about that question maybe as long as 20 years ago. Even then it wasn't feasible. Much less so today.
Consider how does one start from scratch to build Netflix? Or Amazon? What about a log aggregator. Are the problems there really going to translate into general purpose enterprise solutions?
And is one book going to cover all of it?
Or even consider these days is a single persistent data store going to usable for the entire enterprise?
Can you cover 15 different data stores in one book?
And all of that doesn't cover that a person that does understand that must be able to write about it and be willing to spend the time writing a book on it. And perhaps be surprised at how little money they are going to get for writing that book even if it does become a best seller (as a technology book.)
Myself I have probably 100 books right now that cover various technologies. And I don't even do UIs. Further I have gotten rid of other books because they were just old.
I will say that as general enterprise book that I have read I found the following book informative. I would also say however that it tends towards the high level and as a survey of current (relatively speaking) technologies relevant to handling back end services. But it is intended to cover larger systems.
https://www.amazon.com/Building-Microservices-Sam-Newman-ebook/dp/B09B5L4NVT?ref_=ast_sto_dp[^]
I will also say that I believe the author makes it clear that attempting to design a large system from day one is unlikely to do anything except lead to larger maintenance costs. But maybe I misread what he said.
|
|
|
|
|
If I find myself repeating the same thing over and over it occurs to me that I need to optimize.
Doubly so when working with a computer - after all, the computer is there to do work so I don't have to.
There's something to be said for repetition of course. Practice helps, and there's something to be said about tried and true methods, especially in production code, but again, repetition raises the question of optimization.
This is why design patterns make me low key uncomfortable. I feel like they should be baked into the language, rather than having to repeat the same boilerplate over and over again. With C# they spend all this time adding fluff to the language, when they could be baking in design patterns. Seems a missed opportunity.
I'm a fan of the idea of Domain Specific Languages for this reason, even though I've never used one aside from Synthmaker/Flowstone which I'm not sure counts.
What I'd really like is a language that allows one to augment the grammar, sort of like C# source generator tech except it extends the keywords and syntax of the language and then uses the new grammar to generate code.
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
Maybe as a hobbyist my opinion does not count for much as I do not live this everyday. But I feel that the language should be as simple and basic as possible. Let libraries and frameworks do the real heavy lifting. I use C++ which has pointers, references, classes, inheritance, polymorphism, and templates built in. Using that people were able to build libraries like the STL and frameworks like MFC and QT among many others.
I feel if all that was built into the C++ language itself then the language becomes too overbearing and too hard to learn.
But then again, maybe I am totally missing your whole point.
Within you lies the power for good - Use it!
|
|
|
|
|
No, you make a good point, and C++ is a great example of what happens when a language tries to solve everything.
It's just that if libraries could fill the gap by themselves, I feel like they would have already?
It seems to me that libraries tend to fulfill one pattern while creating another (the one you have to use to interact with a library) and based on some DSL work by a former Microsoft mucky muck whose name escapes me at the moment but whose work I've looked over in the past, it seems like having language support for some of these patterns is really the way to solve that last mile integration issue.
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
Quote: It's just that if libraries could fill the gap by themselves, I feel like they would have already?
Stop inventing new problems and processes. Then the current libraries will be enough.
As long as there is a perceived need for something new, someone will write a library to fill that need. Take your pixel library for instance. I am sure there is already libraries out there that would do what yours does, but not the way you want it done. So you wrote another library for it.
Just more people reinventing the same wheel over and over again.
Within you lies the power for good - Use it!
|
|
|
|
|
Forgive me but I think you're missing my point.
You can't write a library to do this:
public component Foo {
public Bar { get; set; }
}
The issue there is at some point you need to insert those actual methods into your code, or inherit from something that already implements Bar which you can't do if Bar is your project specific property.
As such, you're left with one other option for automating this. Source Generators, which require some kind of input spec. Metadata attributes don't quite cut it here, because they aren't descriptive enough. Being able to extend the language via Source Generators solves this. That was the last mile integration I was talking about.
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
honey the codewitch wrote: What I'd really like is a language that allows one to augment the grammar,
They call it Forth. 
"In testa che avete, Signor di Ceprano?"
-- Rigoletto
|
|
|
|
|
I would argue that Forth doesn't have any grammar to speak of. It's just a string of words strewn together.
Would stream-of-consciousness language* be an appropriate name? 
*) Name calling made with love: in another life I wrote a Forth interpreter for a PDP-11
Mircea
|
|
|
|
|
|
honey the codewitch wrote: What I'd really like is a language that allows one to augment the grammar,
Extensible languages, bell-bottoms and polka dots were a thing in the 60-es. They didn't age too well
For your reading pleasure check this SPG language.
Mircea
|
|
|
|
|
There are all kinds of ways to skin that cat though, and the 1960s you had some severe limitations in terms of compilers due to the limited RAM and speed of computers in that era.
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
And yet even though there are probably 100 new languages a year that actually make it into some sort of larger visibility there are none like that.
Which suggests to me at least that they just do not work.
In the same way that I have seen, several different times, people attempt to 'improve' C/C++ by adding clever macros that serve as nothing but attempts to do custom enhancements to the language. For example adding 'begin' and 'end' as substitutions for open/closing brackets.
|
|
|
|
|
I say, stop bloating the languages. Do the basics well, and use libraries for everything else. If you keep doing the same thing repeatedly, libraries are well suited for that purpose - do it once, wrap it up, and re-use it.
Otherwise you're just extending the language to suit your own needs, and everybody's needs are different - and that's what leads to bloat. Not only that, but you end up spending more time tinkering with the language extensions than the actual code and business logic. And not everybody can work at that level.
|
|
|
|
|
Seems like C# got bloated without really doing any of this.
is it really such a bad idea for example to do
public component Foo { // implements IComponent, IPropertyNotify, IDisposable etc
public Bar { get; set; } // generates OnBarChanged/BarChanged, etc
};
and save all of that code? You can't really make that into a library. At best you're using a C# Source Generator and some sort of input document.
or even
public entity Contact { // implements all of the data bound stuff
};
And if that's too much, what about simply the ability to extend the grammar via Source Generators so you could create or install these (as nuget packages) to extend the language as the project requires?
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
What you describe is exactly why I think MSFT is on point with the "targetted models" approach to AI-assist in code.
|
|
|
|
|
Now that's interesting. It occurred to me that AI might be able to insert itself here with some benefit to the developer but I wouldn't know enough about it - having not even used CoPilot for example - to really say so this is news to me.
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
|
That is what IDEs should help with.
They will write all of that boiler plate for you.
The new AI generators might help, too. I have not used them directly, but I have watched team mates flounder trying to have AI generate simple commands that I was able to find in the man page.
|
|
|
|
|
I mean, I think it would be better if the boilerplate didn't have to exist, if possible, simply because even if generated, it still creates a maintenance point.
If some of the boilerplate were baked in as language constructs, or if the language was extensible, you could eliminate that boilerplate at the last mile .i commented to some others here giving an example that a library couldn't solve in C#, but the IDE approach, while it would solve it and does, even with like Winforms tech and InitializeComponent(), now you have InitializeComponent() to maintain. Not the greatest example because it round trips from the designer but most of the time with most tools that's not the case.
Check out my IoT graphics library here:
https://honeythecodewitch.com/gfx
And my IoT UI/User Experience library here:
https://honeythecodewitch.com/uix
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
