|
...the mail room called. I'm afraid your sense of humor might have been lost during delivery. 
"Never attribute to malice that which can be explained by stupidity."
- Hanlon's Razor
|
|
|
|
|
I was just making a point. 
|
|
|
|
|
How much of it is unique information? (That's a rhetorical question - I don't expect anyone to know).
On any computer I have been in touch with (including my own home PC), a significant percentage of the disk is occupied by duplicated files, or duplicated parts of files. Open-source header files for C are notorious: They may have a 50+ lines license text, followed by a couple (I have seen cases of one) line of declaration; the 50+ lines are identical in all the files. Some libraries put each function definition in a separate .c file, repeating the license text from the header file. The .c file has more useful information than the .h file, but very often less lines than the license.
If you do program development: Chances are that you will find the same utilities in several directories. If you keep your photos on the PC, chances are that you have quite a few duplicates (unless you are a very orderly person ). How many backups of the same, unchanged file do you have on a multitude of USB sticks, CDs, external hard disks etc? And so on and so on.
I will spend Christmas vacation completing my self-made deduplicating backup program. Deduplication is at the file level, not disk page level, so it will save no space for those extensive license headings. But once I started looking around, I realized that at least half of my disk space is taken up by file duplicates. At work, we have some huge file servers running deduplicating at page level; those responsible for it estimates that it reduces the requrements for real disk space to less than a third.
For even more savings, a lot of information could be encoded more efficiently. Sound and video have come a long way with compression. Lots of medical information is still in uncompressed text form; much of it could be codeified, and what must remain as text compressed using well known methods. (This is on the way in, but far from completed.)
Database files are notoriously huge; they often compress quite well - if you compress them. And lots of database developers (of schemas, not code) have not been drilling in database normalization: The same attributes are repated in two, three or more tables.
Disk is so cheap nowadays that noone worries. Terabytes, petabytes, exabytes ... that's all "quite big", with little further distinction. Until you come back from vacation with ten packed 256 GB memory cards holding your HD movies, and start copying them to the hard disk (and maybe your card reader is a USB 2 device)...
Ten years ago, developers stopped being concerned about CPU load: Just buy a faster CPU! Who cares abotu O()? The same is now happening with disk space - who cares about duplication? I disklike both thrends: Algorithmic complexity is still essential, and efficient data structures with low redundancy is still essential.
|
|
|
|
|
Member 7989122 wrote: I will spend Christmas vacation completing my self-made deduplicating backup program. Deduplication is at the file level, not disk page level, so it will save no space for those extensive license headings. But once I started looking around, I realized that at least half of my disk space is taken up by file duplicates. At work, we have some huge file servers running deduplicating at page level; those responsible for it estimates that it reduces the requrements for real disk space to less than a third.
Is this something you intend to share here in CP?
I'm doing rather well myself when it comes to creating file duplicates, but I have a neighbor who's notoriously bad at it, and he's constantly asking for my help to clean up his file system. I'd love to find a fast utility that can locate duplicate files across drives, and give a number of options as to which ones to delete.
|
|
|
|
|
dandy72 wrote: Is this something you intend to share here in CP?
I might, but I have high respect for the "eating your own dog food" paradigm I need to try it out on my own mess of unstructured backups to see if it helps me as much as I hope.
Also, my backup system is explicitly for user files. I will happily backup your installers -.msi files and the like - but I have no intention of a complete backup of the OS, installed system, registry and similar files. If you experienece a total system crash: Reinstall the system. After that, I come with my user file backups. Nothing more.
Many users may not be satisfied by that. They want a total solution for everything. But for those who are most worried about their own stuff: Maybe my lightweitght backup system could be what makes them start making regular backups!
I take your suggestion to publish it on CP as something I should seriously consider. Thanks for the kick in the a**!
|
|
|
|
|
Member 7989122 wrote: my backup system is explicitly for user files. I will happily backup your installers -.msi files and the like - but I have no intention of a complete backup of the OS, installed system, registry and similar files. If you experienece a total system crash: Reinstall the system. After that, I come with my user file backups. Nothing more.
Same here.
I tried the route of taking images, but soon found that it was an enormous waste of time and disc space -- added to the fact that it restores all the cr@p you should have got rid of, anyway -- so now I just have back-ups of user files and settings.
E.g. the "My Documents" (which I don't use at all for user files) and the "ProgramData" directories contain user settings for a large number of programs, which can simply be copied into a new/restored system after the programs have been installed.
And whenever there's a portable version of an application, or an option to store settings in a file, rather than the registry, that's what I go for.
Incremental back-ups of things like that consume precious little time and resources, and while a re-image may be quicker at restoring a system than this way, how often do you have to do it, and how can you use images to set up new machines?
[edit] It's also dead handy when you're setting up VMs. [/edit]
I wanna be a eunuchs developer! Pass me a bread knife!
|
|
|
|
|
Frankly, all I want is the deduplicating part -- I just want something that'll locate identical files, and let me choose what to do with them in bulk.
|
|
|
|
|
File data duplication is necessary up to a certain point, but no further. The 3-2-1 rule for data persistence - 3 copies, 2 different storage types, 1 off sight - gives us a probability of failure that is arguably less than the mean time between failures of hard disks. But copying today occurs quite naturally as has been outlined in this thread and has led to great and ever-increasing redundancy. The real fear is not being able to find the original copy again, not that it doesn't get stored permanently somewhere.
I would like to propose a different approach to solving the redundancy problem: create a system where there is never more than one virtual copy of each of your files and folders. That is the way Hiveware for MyFiles will work.
|
|
|
|
|
There's a nice list here that goes up to yottabyte (2^80 bytes).
I'd like to think that future measurements might include the elephantyourheadabyte and the yeravingalaughinchabyte.
Whenever you find yourself on the side of the majority, it is time to pause and reflect. - Mark Twain
|
|
|
|
|
Too bad healthcare quality isn't as impressive!
I've often wondered if the reason there hasn't been a cure for cancer developed because it's more profitable to treat it then cure it?
I may not be that good looking, or athletic, or funny, or talented, or smart
I forgot where I was going with this but I do know I love bacon!
|
|
|
|
|
I keep reading those conspiracy theories, but OTOH, think of the money to be made from the patents.
|
|
|
|
|
You are a cynical old bugger!
Never underestimate the power of human stupidity -
RAH
I'm old. I know stuff - JSOP
|
|
|
|
|
Why yes I am!
I may not be that good looking, or athletic, or funny, or talented, or smart
I forgot where I was going with this but I do know I love bacon!
|
|
|
|
|
Cannot count how many blu-ray discs needed to copy the data.
TOMZ_KV
|
|
|
|
|
"...has 33 different vice presidents, each earning over 200 thousand dollars a year. Now, I have spent the last two months analyzing what all these guys do, and I still can't figure it out. One thing I do know is that our paper company lost 110 million dollars last year, and I'll bet that half of that was spent in all the paperwork going back and forth between all these vice presidents."
--Gordon Gekko, Wall Street
|
|
|
|
|
I'd be curious to know the nature of the data. Images take a lot more space than say, your chart history does.
|
|
|
|
|
There is so much important scientific data in the healthcare data (globally) the challenge nowadays is to be able to use that data for research purposes and still keep the patient's privacy. It is scattered all over and not one repository for the info to be used for research purposes.
2,314 billion gigabytes? That a lot.
|
|
|
|
|
How much of that will be in unsecured databases or places where they didn’t change the default password or install security fixes, etc.
|
|
|
|
|
Methinks a good chunk would be comprised of genetic data.
-Each of us has ~3 Billion base pairs (A unique sequence of A,C,G,or T)
-The trend (as costs go down) is moving toward patients having their entire genome sequenced instead of just the "we often see mutations in this specific area".
-For some diseases it may be prudent to get the DNA sequence multiple times (as the person's genetics change), although this would not often be the whole genome in these situations.
-Tumors (often have dramatically different gene sequences) are also now sequenced to better plan treatments for patient A vs. patient B.
-But 3 billion characters to store is the beginning of the story (although it is only known later on in the analysis) because before the sequence is known the DNA sequencers generate, yes as mentioned, images, which are used and analyzed to tell what the DNA sequence is, i.e., these are raw files that must be analyzed, compared to a reference set, etc.
-So, lots of data is generated, per genome. Lots.
Separate from the genetics aspect, hospitals I'm guessing will continue to digitize their patients' health care records.
|
|
|
|
|
I ordered laser labels yesterday for my new printer - company I've never used before. Arrived today, and ... the package feels odd, like there are lumps in it ...
Open it up and find the labels, a "10% off your next order voucher, a nice letter thanking me for using the new website - and a bag of chocolate money as a little Christmas present!
Win win! 
Sent from my Amstrad PC 1640
Never throw anything away, Griff
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
AntiTwitter: @DalekDave is now a follower!
modified 19-Dec-18 7:18am.
|
|
|
|
|
|
Nope: I may - or may not - order from them again, depends on price and availability.
But I will eat the chocolate! 
Sent from my Amstrad PC 1640
Never throw anything away, Griff
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
AntiTwitter: @DalekDave is now a follower!
|
|
|
|
|
QED!
It really is your fault that their long-term customers will be overcharged!
Shame on you!
I wanna be a eunuchs developer! Pass me a bread knife!
|
|
|
|
|
Of course they hope that you will smudge the labels with the chocolate so that you have to order new ones 
Bib on !
|
|
|
|
|
I'm suspicious. They somehow knew exactly how to bribe you. 
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
