|
Why not make the application acquire an exclusive lock on the file whenever it's being edited?
That will definitely prevent corrupted changes.
EDIT: On second thought, even this wouldn't work because a person might still write an older version of the file when committing changes.
The difficult we do right away...
...the impossible takes slightly longer.
|
|
|
|
|
Richard Andrew x64 wrote: On second thought, even this wouldn't work because a person might still write an older version of the file when committing changes.
So, either lock the file, read what you need to make a decision, and write it back, in one go.
Or, if you require some time to ponder the changes: Read lock, take note of the last modification time, read, release. Do all your pondering, preparing your changes. When changes are ready to be written: Exclusive lock, check time of last modification. If later than the one you saw the first time, read the new data, release and repeat from the pondering step. If write timestamp is unchanged, write your new data and release.
This is a file system version of database 'optimistic locking'. With database transactions, you should always be prepared for a rollback if some other transaction modifies data you have based your calculations on. I am not saying that all database applications are prepared for rollback, only that they should be  . Databases with optimistic locking favors short transactions: The shorter the time span from data are read to the commit, the less risk for someone else making modifications inbetween. Your DIY optimistic file system locking is similar: Do all the preparations that you can do without reading the file, leaving the minimum work between reading and writing, to reduce the risk of a rollback. But you must be prepared for it, forcing you to read the updated data before attempting a redo transaction.
|
|
|
|
|
Scenario:
We have (as of today, will be more) 200 devices sending telemetry into an SQL server
with a frequency of 5 rows each pr. second.
Each row contains 6 decimal values to be processed into another dataset/table
containing min/max/avg of these values in 1 minute intervals.
I have never worked with DataLakes and similar tech, so I wonder:
Should i read up on storing the raw telemetry in a datalake, and set up post processing there,
or Just go for my existing SQL server and create a c# job post processing the data in the background myself?
TIA...
|
|
|
|
|
There are so many "it depends" that need to be looked at in here. We'll start with a simple question that should, hopefully, start us towards the answer.
Question: Do you need to do anything else with the data that you are inserting? Once you have created these six values, is the telemetry data ever used again?
|
|
|
|
|
The telemetry data are also needed later, yes.
|
|
|
|
|
Where are you hosting the data? Is it in a cloud provider such as Azure or AWS or are you managing it yourself?
|
|
|
|
|
The DB, and the code that receives tha data it hosted in Azure.
Data is recieved as MQTT messages by Azure IoTHub.
A C# worker subsribes to the incoming messages, and stores them normalized in an Azure SQLdb.
|
|
|
|
|
So, you have data coming in from 200 devices, 5 times a second. Let's think about the problem areas here:
- When you add data, at that volume, you're adding 1000 items a second. Not a problem in SQL Server.
- I'm assuming it's one aggregation per device every minute, so you're looking at a read of 60K rows every minute, with a write of 200 rows.
I would be tempted to invert the problem slightly. Rather than write to one source, write to two. So, write the incoming messages to SQL Server and, at the same time, write them to something like Redis Cache. Then, every minute, get your data from Redis and use that to calculate the values for the other table and write to that. This approach follows (somewhat), the CQRS so you're separating the reads and writes from each other.
The beauty is, this scales well.
|
|
|
|
|
That was actually a brilliant Idea, as I already have Redis running 
The number of devices will hopefully increase 10 to a 100-fold, but this will still scale.
So will it with additional transformations in the future.
Thanks!
The only drawback with this solution is that I have no excuse to sit down and play with Datalakes
and all its tools... 
|
|
|
|
|
What worries you most? Is it the data volume? A back of the napkin calculation shows something like 10-20k per sec. Almost any DB system could handle that.
Mircea
|
|
|
|
|
Volume, Yes.
As of today, the volume is not a problem at all.
But when we get thousands of devices online, I'm a little bit afraid that the
Job generating the aggregates will struggle/lead to locks.
Also, who knows what other Post-processing needs will occur in the future.
|
|
|
|
|
How many more devices you anticipate to connect: 20000, 200000? At 20000 level, I think you should still be safe. At 200000, not so sure.
Keep in mind however that inserting is the more probable bottleneck as it has to create all those indexes, allocate pages, etc. Querying should be much faster.
Mircea
|
|
|
|
|
Thanks!
|
|
|
|
|
In one of the sub-threads you said the following
"The number of devices will hopefully increase 10 to a 100-fold"
You should have a very hard idea of what this increase looks like before you start making decisions. You should know the following.
1. What is the timeline for the increase? Also will the increase be gradual or will it be abrupt?
2. What is the maximum? If you owned the market what size would you need to support?
Typically when I size I do the following
1. Document my assumptions.
2. Document my sources - where did I get the assumptions and how did I derive them.
3. I then increase the size from that by 3 to 10.
Don't attempt to create a system the will work for the next 100 years. One that works for 10 is fine.
After you have those numbers then you start looking for solutions that will handle those numbers. Keeping in mind of course that your architecture/design should support the larger values, but the implementation does not need to support that now. You just want to make sure the implementation does not preclude sizing it up.
|
|
|
|
|
Thanks!
Very good points.
The question was more about what would a datalake offer in terms of post processingand other benefits, since I am not familiar with the technology.
I have no problems implementing an efficient solution with Azure/sql/c# for the foreseeable future.
Just needed some insights in a tech I'm not familiar with.
|
|
|
|
|
i am building an application in which one of the part is calendar updates , push notification ,
i am wondering what is best way/architectural pattern/technologies are best suited for this scenario . I am a .net developer .if possible kindly guide me on same platform
|
|
|
|
|
|
erummirza wrote: push notification
Pushing requires that the destination is available and capable of accepting the request. That seems unlikely for "calendar" in general since it would suggest client machines (which can be off or have no network access.)
So you are going to need to refine your requirements more before you can do anything.
If the requirements are very specific, such as using a MS Exchange server, then there still is not enough detail for an "architecture". But in that case you would start with how you are going to get the updates in the first place.
|
|
|
|
|
I have a framework that I'vev been developing in .Net 4.5.2 over the years. Its all stored in individual repos in GitHub.
There are 25 individual projects. Now I want to increment the version to 4.7.2, but when I try to update the versions, it breaks some apps.
So, what's the right way to set this all up in GitHub? Would you create a branch for each version?
In theory, theory and practice are the same. But in practice, they never are.”
If it's not broken, fix it until it is.
Everything makes sense in someone's mind.
|
|
|
|
|
If you update the target framework for an assembly, then you need to update the target for anything that references that assembly. That should only be a problem if you're not updating the target for everything, which doesn't sound like the case here.
If you start from the bottom up, then you may get errors when you first open a project that targets 4.5.x and references an assembly you've just updated to 4.7.x; but those should go away once you change that project's target framework.
If that bothers you, then it may be easiest to work from the top down; start with the applications, then the assemblies they reference, then any transitive references, and so on. Once that's done, you can work your way back up, updating the references at each level.
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
The problem us that I still have apps that target my 4.5 assemblies, so I think I need to create copies (new versions) of my assemblies.
I was considering create a new branch under MyAssembly called 4.5, then another called called 4.7, etc.
In theory, theory and practice are the same. But in practice, they never are.”
If it's not broken, fix it until it is.
Everything makes sense in someone's mind.
|
|
|
|
|
If the apps target 4.5, then the assemblies they use need to target 4.5.
Creating separate branches might work; but there's a good chance the branches would drift further and further apart over time, making it harder to apply bug fixes to them all.
I'd be more inclined to use multi-targeting[^] - have one branch, with one project, building for multiple framework versions. Where needed, you can use conditional compilation to add features to the 4.7 version that won't be included in the 4.5 version.
How to multitarget | Develop libraries with the .NET CLI - .NET | Microsoft Learn[^]
If you're updating the apps to at least 4.7.2, then it would probably be a good idea to have the libraries target .NET Standard 2.0[^], the last "standard" version supported by .NET Framework.
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
I'm a part of a Members Only club where I'm working on an app for them. We have coded membership cards.
They have a keypad to open the door and they want to implement an electronic Card Reader Door Lock and we want to build it into the app so they can lock people out as needed, say when their membership expires.
This would be like a hotel door lock, except I only need one lock, and, most importantly, I need to be able to build it into my app, so I need a solution that comes with an API I can tap into.
Anyone know of anything like this?
In theory, theory and practice are the same. But in practice, they never are.”
If it's not broken, fix it until it is.
Everything makes sense in someone's mind.
|
|
|
|
|
Sounds like a job for NFC. Many phones these days do NFC, and for the members that don't have one, there's a "credit card" option.
Software rusts. Simon Stephenson, ca 1994. So does this signature. me, 2012
|
|
|
|
|
Can you elaborate on this a little more? I looked up 'NFC', but the results are a bit vauge
In theory, theory and practice are the same. But in practice, they never are.”
If it's not broken, fix it until it is.
Everything makes sense in someone's mind.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
