|
I have an third party application that will be processing CSV files and pushing data to a data store. However, due to limitations in the application, the CSV files need to be pre-processed to add some additional data, reformat some fields and split the original file into smaller files.
The question is: to write a script to process it daily or to write a compiled app...
Either way, the solution will be running as a scheduled task on a server.
Thoughts? Opinions?
|
|
|
|
|
Why not both? 
This space intentionally left blank.
|
|
|
|
|
Which one will you look at in six months after it's become a critical production process and be able to understand/adapt more easily?
|
|
|
|
|
I started with scripting for adaptability, but, now I'm leaning towards a console app for expanded control (garbage cleanup, better logging, debugging, etc).
It WILL be a mission critical app to the users once in full production.
Ugh...
|
|
|
|
|
I'm not sure what any of those things have to do with deciding between a compiled or scripted solution.
|
|
|
|
|
Started with the scripted solution because it's quick and should have worked.
However, despite destroying objects, there is still an issue with something not being released, but, of course, the scripting app won't tell me what it is.
If scripting worked, I could edit in place and move on.
Since it is causing issues, I will be moving to an app (.NET based) where I can control the environment better.
And, what does one have to do with the other and this response? Prefer the simple and elegant, but moving to the more involved, but probably better in the long run. And.. continued to try to get scripting working after original post.
|
|
|
|
|
Your main question is related to maintainability of the solution imo.
Who is going to maintain it?
What technology (script language) do you intend to use?
Who else can handle this technology in your team?
What is your rollout strategy like? Do you require formal testing and signoff for example...
Just to name a few...
For me; scripts are easier to roll out than executables. And .NET is not a data friendly solution if all you need is CSV formats to be handled.
Cigarettes are a lot like hamsters. Perfectly harmless, until you put it in your mouth and light it on fire.
|
|
|
|
|
I would have said .NET from the get-go; the difference between a script and source code is one, measly compile, but the .NET will give you so many options and so much reference in the literature.
Of course, sometimes you have to hack a prior script, in which case you just have to dive right in and stay with it. 
|
|
|
|
|
Tim Carmichael wrote: It WILL be a mission critical app to the users once in full production.
Scripting means someone else, who thinks they're smarter than you, can "improve" upon it.
|
|
|
|
|
u$ SQL's BCP utility and/or Integration Services for SQL may be worth a gander.
|
|
|
|
|
You say potata...[^]
My take on these sorts of issues is use whatever you are comfortable with.
The advantage of the script is that it won't need recompiling if you need to make a change the logic is easily viewed by opening the script in notepad.
The advantage of the compiled code app is that the debug facilities in the IDE may be better.
You say potata..
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens
|
|
|
|
|
I'd say write a script. It is what the scripting languages are there for. Size of your script (depending on the scripting language) will be much smaller (thus less effort) than the app. It should be much quicker to execute too (depending on the scripting language and your scripting skills). In the long run you can build yourself a scripting tool kit. Advantages of writing a script for the current project will possibly not show yet but will definitely show in the projects to come.
When you require the use of a scripting language in the future, you'll have scripts performing script related tasks and apps performing app related tasks and not a unmanageable toolkit consisting of both scripts and apps performing only script related tasks.
I agree with GuyThiebaut that you won't have advantages of debugging... thus would suggest sanity checks throughout your script (a good practice even if you're writing a compiled app).
"Program testing can be used to show the presence of bugs, but never to show their absence."
<< please vote!! >>
modified 11-Feb-14 8:56am.
|
|
|
|
|
R. Erasmus wrote: I agree with GuyThiebaut that you won't have advantages of debugging... thus would suggest sanity checks throughout your script (a good practice even if you're writing a compiled app).
Not ok... even scripts should get a complete test set. Especially as it is going to be important in production. Don't cut corners there, you'll regret it.
Cigarettes are a lot like hamsters. Perfectly harmless, until you put it in your mouth and light it on fire.
|
|
|
|
|
Agreed, if its going to be production code then it needs to be unit tested which makes the decision for you.
Scripts are good for adhoc operations tasks but not for a production system that's going to evolve and go through a number of versions
|
|
|
|
|
Have you thought of doing a best of both?
Personally I'd have an app that can read the input file and then use scripted functions for the reformatting.
speramus in juniperus
|
|
|
|
|
When I process CSV files this way I usually just do it all in SQL with temp tables. If you have to do a lot of string manipulation then procedural code might be better, but otherwise processing the CSV with a stored procedure is generally going to be fast and easy to maintain.
Otherwise I don't see much difference between a script and a compiled app, other than possibly speed. But if you can do it in set operations with SQL rather than with loops, that's going to be a huge speed gain right there.
|
|
|
|
|
For me, everything rounds to what would be easier, if the script language (or environment) have everything I need then I do a script, if the compiled app would be easier to create, then I go for the compiled app.
As a rule of thumb, if the script will end up with more than a hundred of lines, then a compiled app may be better.
|
|
|
|
|
Try this package, CsvHelper [^]
For my team this made handling csv files a snap in C#.
|
|
|
|
|
Try Power Shell script if like
|
|
|
|
|
I have several apps that do this and I have used scripts, console apps and now I pretty much create services.
1. Unless you control the box the script is on, some one is going to open up the script in a text editor just to look at it. Come you know you have. Ever think you can make it better with just a little tweak?
2. If people log into the box with different credentials and run your console app. Oops different results - must be a bug, not a security feature.
3. With a service I can control the user the app is executing under, and people are less likely to mess with it.
|
|
|
|
|
The issues that you expressed shouldn't have any thing to do with "script" or not unless the scripting language you have is severely limited in some way. It should still be able to log, recover from errors, etc.
|
|
|
|
|
I would prefer to do it in a compiled .NET application because I'm more familiar with C# than scripting languages and, as you pointed out in a later post, memory management is (IMHO) better in .NET.
However, whatever you choose to do, MAKE SURE THAT YOU WRITE TESTS FOR IT so that eight months down the line you can catch any bugs that the "tiny little small change" that you need to do might introduce.
|
|
|
|
|
Hi I've been searching awhile but haven't found what I was looking for. I am trying to upload a file to SQL database using HTML5 . I have the html5 side done but am looking for a resource or example of server side code that will upload the file to a SQL server. I know I'll using httpwebrequest. Do I use a webservice (asmx file)? a handler (axhx file)? or something else?
Thank you!
John
|
|
|
|
|
|
Thanks Richard, I'll give it a try, though I'm not using an aspx page.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
