|
Jeremy Falcon wrote: Ladies can't resist an async coder.
What for...certainly not her and her sister...
|
|
|
|
|
I can't help, but reading "parse" in the body of the message...
This is clearly a case for... HONEY THE @CODE-WITCH tatatataaaaaaa 
M.D.V. 
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
|
|
|
|
|
are you sure the bottleneck isnt disk i/o?
Real programmers use butterflies
|
|
|
|
|
Yes, just to make sure.
I've made test runs just reading an ID from every record which goes twice as fast, and that's on a slow HDD here at home.
And when I move this to a server the disks will be considerably faster.
|
|
|
|
|
40GB of your 80GB XML file are tags. So much for the overhead.
The suggestion is to worldwide drop all markup languages (XML, JSON and similar shiite)
|
|
|
|
|
But compact binary formats are almost impossible to patch up using vi. Linux guys will feel completely lost!
|
|
|
|
|
|
Lots of Linux guys working in Windows environments. Lots of them hate it, they do it just to earn money so they can pay for their home computers to contribute to Linux based open source projects in their spare time. And they spend a lot of energy on bithcing about things not being exactly as they are used to in the Linux world.
My comment was "based on a true story". I made one application storing a fairly complex persistent data structure in a binary format. This was met with heavy critisism: What if that data structure becomes inconsistent - how can we fix up the inconsistencies when it is not in a readable format? I guess I wasn't too polite when answering them that one major reason for not using a readable format was to prevent them from poking into the file with vi, introducing inconsistencies.
In this system I am working on now: It is a Windows desktop application, but there is a function for converting all file system paths to Unix style forward slash path separators, and a handful utility functions that fails if you submit a DOS/Windows style path with backwards slashes. Forward slashes is the only "correct" path format, they claim - DOS/Windows was simply wrong until they started accepting the correct format. So the (Windows) users of this program must simply accept that when using the conventions of their OS, they are simply wrong.
In an earlier project, the Linux mafia forced me to make special adaptations in my (very) Windows-specific utililty: They inisisted on running it, in their shell based batch jobs, from a Linux-adapted command shell that enforced case sensitive environment symbols. They make use of it, too: Their jobs started crashing, and it boiled down to my utility treating symbols differing only in case as synonyms, while they were distinct in their jobs.
In my current project, one of the first thing I did was to replace case sensitive file name comparisons with case insensitive ones. It was argued, "But cmake always uses CMakeLists.txt, with exactly that casing! There is no need to do a case insensitive comparison!" Well... Why did the program then barf? Someone wrote CmakeLists.txt, and the program just failed, because it didn't find the file.
I would have tolerated this a lot more if it wasn't for the constant bitching from the Linux mafia about Windows users refusing to learn anything new, but cling to Windows ways of doing things (when working under Windows) rather than learning the way these wonderful command-line utilities ported from the wonderful world of free and unsupported software expects you to put everything in a loooong command line. This bitching about unwillingness to learn is nothing new: I have heard it constantly repeated for at least 15-20 years, in numerous different environments.
I recently discovered that the Compare plugin to Notepad++ was incapable of comparing two generated build jobs: The command lines invoking gcc was in excess of 3800 characters. Looking up the documentation for the generator, I found explicit warnings about Windows incapable of handling command lines exceeding 8 Ki characters; this could cause problems (but Npp Compare obviously has a far lower limit). Hooray for command line interfaces, where every detail is available at your fingertips, not in a silly screen form!
Now, a lot of newer Linux born utilities do use binary formats - but the Linux mafia always have explanations for this very special case where it is justified. For us who have lived through several wars, it this interesting to see how a lot of arguments that were boasted as super-essential, a few years after the war was won is laid silently down and more or less replaced by what the loosing side was promoting, although usually with a twist, so it will not be recognized.
Let me give one example of this: Packet routing. One of the fundamental strengths of IP as compared to e.g. ATM/FR/OSI-NP is that if a link is broken or congested, packets can select a different route: Each packet contains the full address and is, in principle, routed independently, and can follow any route to the destination. Connection oriented protocols assume that all packets follow the same route; a link/physical failure requires a full connection reestablishment. ... Yeah, right. In today's Internet, every IP packet finds its own path. Right. You chop of an international trunk fiber, and no router anywhere in the world requires manual intervention for having its routing tables changed; that goes by itself, automatically. Believe the old myths, if you like.
There are several other examples, like Internet and US phone guys insisting on inband signaling (reducing a 64 kbps line to 56 kbps data capacity), while Europeans favor OOB singaling, both in phone systems and data networks. Then when the SIP protocol for establishing IP phone connections (or other kinds of connection) was defined, all that heavy critisism of OBB signaling was kept low - SIP is OOB signaling in a nutshell.
There is no way to completely escape poor solutions promoted by the Linux and Internet communities (which has a large degree of overlap); we have to live with it, in spite of extremely poor tools, poor user interface and high overhead (especially when it comes to space requirements). But when I make Windows specific tools, aimed at another target audience that Linux hackers, I prefer to do things in better ways.
|
|
|
|
|
This should be published to ever lasting memory. My congrats.
|
|
|
|
|
As I see it:
There must be a reason people are paying for not having to use a free OS.
|
|
|
|
|
Hahahaha "
I would have tolerated this a lot more if it wasn't for the constant bitching from the Linux mafia about Windows users refusing to learn anything new, but cling to Windows ways of doing things (when working under Windows) rather than learning the way these wonderful command-line utilities ported from the wonderful world of free and unsupported software expects you to put everything in a loooong command line. "
Where are the Async adventures.
|
|
|
|
|
At least 55GB actually.
The Database I'm copying the data to is only 25GB at the moment and that's also having quite some overhead.
|
|
|
|
|
There you go. Something went terribly wrong in the history of computing.
|
|
|
|
|
XML parsing is a forgotten art, but I have a story that might inspire your creativity. 😉
I once had to build an XML parser that could process a 1.8 gig file on demand, with the intent of generating C++ header files. The hard part was terrible formatting and not being able to pre-process the darn thing, which forced me to use a single pass multi-line regex implementation. After a couple of weeks struggling with it, my biggest time save was finally gained by switching to a stream reader.
When I read your story, my first idea was to use a non-locking stream reader and simply running the thing 4 times on 4 cores.
Personally, I don't see the need for parallelism in this instance and I think it's a red herring to be honest.
Anyways, good luck. 👍
|
|
|
|
|
I'm probably skipping it.
The sequential program works and can run in the background without any problems.
For me it was mostly an educational experience.
|
|
|
|
|
Are you, by any chance, using Linq?
|
|
|
|
|
|
Parent: If all your friends jumped off a cliff, would you do it too?
Machine learning model: Yes.
|
|
|
|
|
That depends very much on why they are jumping. When a T-Rex is coming after them, I would probably also decide to discuss the matter with Isaac Newton than with it. Then again, together with my old friends Heckler and Koch I might try something else.
I have lived with several Zen masters - all of them were cats.
His last invention was an evil Lasagna. It didn't kill anyone, and it actually tasted pretty good.
modified 29-Jun-20 4:59am.
|
|
|
|
|
CodeWraith wrote: When a T-Rex is comming after them, I would probably also decide to discuss the matter with Isaac Newton than with it.
Me, I'm the other way. I know Gravity works, but I also know T-Rex is extinct ...
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
You do know that they have found the remains of several dinosaurs which were not completely fossilized? Thank god a 65 million year old mummy should not contain enough intact DNA to clone anything.
Anyway, whatever I do when I see one of them, I will probably ask questions later.
I have lived with several Zen masters - all of them were cats.
His last invention was an evil Lasagna. It didn't kill anyone, and it actually tasted pretty good.
|
|
|
|
|
I will do a double check first - it's probably one of these: Dinosaur[^] and while being a fashion disaster, it's likely to be less dangerous than the cliff. 
"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
I hope they don't come with the idea of a Jurassic Park. We all know how it could end...
Life finds its way out.
M.D.V.
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
|
|
|
|
|
Why go through all that trouble? There are lots of surviving dinosaurs with complete and functional chromosome sets. Chances are good that you had a dinosaur egg for breakfast or eaten a dinosaur for lunch. Kentucky Fried T-Rex.
And they really have been trying to undo genetic changes and reactivate older DNA in chickens, so that they have a tails, snouts and teeth again.
I have lived with several Zen masters - all of them were cats.
His last invention was an evil Lasagna. It didn't kill anyone, and it actually tasted pretty good.
|
|
|
|
|
CodeWraith wrote: And they really have been trying to undo genetic changes and reactivate older DNA in chickens, so that they have a tails, snouts and teeth again.
As long as they're still the size of chicken, and tastes like chicken...
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
