The Weird and The Wonderful

Re: How I learned to stop worrying & love The Error

20-Mar-23 7:13

There is no such thing as a timestamp without a timezone. Does not matter how you do it. All programming languages. All databases. Even when it is not obvious. Even when it is not defined.

Even if you just write it on a piece of paper.

Myself I am not even sure if dates (not timestamps) exist without a timezone. Certainly my birthday is just as useable here as it is halfway around the world. But if I am supposed to get a something free from a international company on my birthday every year as a swag then when does it show up?

DrWalter PE26-Apr-23 18:38

DrWalter PE

26-Apr-23 18:38

raddevus wrote:
I can remember the exact instant when I realized that a large part of my life from then on was going to be spent in finding mistakes in my own programs.

I agree with whoever said,Programming is the act of debugging a blank page. Maybe Knuth?

But it feels so _diiirtyyy_!

PIEBALDconsult1-Mar-23 8:01

1-Mar-23 8:01

So I have this method ( C# , .net ), let's call it F(i) where i is an integer representing a UTF-16 character. The method determines which one of the following classes the character is a member of:
  Control (ASCII control characters)
  Delimiter (the caller can specify which characters are delimiters)
  EOF (-1)
  Escape (\)
  Non-ASCII (i > 127)
  Normal (ASCII characters which are not members of another classes)
  Quote (")

This is implemented as an array look-up with a catch for IndexOutOfRangeException which will fire for EOF and non-ASCII characters. This has been working well for a while. The data (JSON files mostly, but not exclusively) is nearly all ASCII characters with only an occasional non-ASCII character -- maybe a few "smart-quotes" or similar, which are OK, in many cases I replace those with their ASCII versions anyway.

BUT once in a while we receive a corrupt file which (in the latest case) includes a JSON value which contains more than a million non-ASCII characters (in the file they are encoded as three-byte UTF-8).
F(i) was not performing well in this case. Apparently having the catch fire occasionally is OK, but firing a million times in rapid succession is decidedly not.

Once I tracked the issue to F(i), I could try altering it to add a test for i > 127 and avoid the exception (which I am loathe to do on principle). But unit testing did show that it improved the performance considerably for the non-ASCII characters without significantly hindering the performance of ASCII characters (EOF is still handled by a catch).

That sounds like a win, except... I just don't like having the extra test which is essentially needless given that we don't expect any/many non-ASCII characters in most files we receive.

Sooo... I named the original version Fa(i) and the new version Fn(i) and I made F(i) a delegate which starts out pointing to Fa(i) but:
  If Fa(i) encounters a non_ASCII character it will re-point F(i) to Fn(i)
  If Fn(i) encounters an ASCII character it will re-point F(i) to Fa(i)

Slick as snot. Unit testing shows good performance.
  Time required to read the million non-ASCII characters with Fa == 12 seconds
  Time required to read the million non-ASCII characters with Fn == 0.06 seconds

I have integration testing running now. The current production version times out the file read after ten seconds (a protection I had to add a while back for another corrupt file), but with the new version, it should read successfully then I should get an error when trying to stuff more than a million non-ASCII characters into a database column which is defined for 500 ASCII (CP-1252) characters.

In the meantime, the people who send us this file are trying to find out what's causing the issue. So far, it's intermittent (a dozen times in the last four years), so it hasn't become critical.

I'm pretty sure I've done this sort of thing before -- having a delegate which points to one of two slightly different implementations of a method depending on what has been encountered in the data, and flipping back and forth dynamically as required. I guess I'll be code-spelunking this afternoon to review that code.

This is the way.

Mircea Neacsu1-Mar-23 8:34

Mircea Neacsu

1-Mar-23 8:34

Have you looked at the "standard" implementation of isdigit, isalpha, etc. C functions?

It uses a 128 character table in which different bits represent character class. Something like:

C++

tab['a'] = CLASS_ALPHA | CLASS_HEXDIGIT | CLASS_LOWERCASE;

For sure, you need to range limit the input.

Just a thought.

Disclaimer: Haven't looked if recent C runtime library implementations still use that "standard" implementation.

Mircea

PIEBALDconsult1-Mar-23 9:08

1-Mar-23 9:08

Of course, but the built-in methods of System.Char (.net) don't suit my needs, so I rolled my own as is my wont.
I actually have a number of places where I have to roll my own character (or byte) classing solution to meet my requirements.
The "delimiter" class in particular depends on the type of file being read -- for JSON, the delimiters are { } [ ] : , ; for CSV they are , \r \n .

Mircea Neacsu1-Mar-23 9:22

Mircea Neacsu

1-Mar-23 9:22

I was talking about the method, not the functions themselves. The basic idea is that you can assign up to 8 distinct classes to each character. Fleshing out a bit more my idea, something like:

C++

tab['{'] = CLASS_DELIM_JSON;
tab[','] = CLASS_DELIM_JSON | CLASS_DELIM_CSV;

The table would of course be statically allocated and initialized:

C++

bool is_a (int x, char what)
{
  static char tab[128] = {/*blah, blah combinations of class bits*/ };
  //preliminary tests go here
  return (tab[(char)x] & what) != 0;
}

Mircea

PIEBALDconsult1-Mar-23 9:32

1-Mar-23 9:32

Yes, and that's what I'm doing here -- but C#ishly, using an enum to define the constants and the map.

jschell20-Mar-23 7:20

20-Mar-23 7:20

PIEBALDconsult wrote:
BUT once in a while we receive a corrupt file...but firing a million times in rapid succession is decidedly not

Why does it continue after the first one? If firing once means it is corrupt and then you expect the rest to be corrupt then why not stop?

Or say stop after 10 or so?

PIEBALDconsult wrote:
to add a test for i > 127 and avoid the exception (which I am loathe to do on principle).

Not sure I understand that statement. Unicode does not fit that range and you are getting unicode. So testing for it rather than assuming it is correct is what should be happening.

PIEBALDconsult20-Mar-23 10:04

20-Mar-23 10:04

The reader has no clue whether what it's reading is corrupt or not, that's determined at a higher level. The reader's job is to simply read the characters and return them.

jschell wrote:
Or say stop after 10 or so?

That's the issue I was running into, I had it throw an Exception after ten seconds, I needed to keep that from happening.

jschell wrote:
you are getting unicode

Yes, but the reader was taking too long to read the UNICODE characters which are outside the ASCII range. It now reads them almost as quickly as UNICODE characters which are within the ASCII range.
But to do so, It's flipping between two implementations of a method -- one for ASCII and another for non-ASCII -- depending on which characters it encounters.

jschell21-Mar-23 5:35

21-Mar-23 5:35

PIEBALDconsult wrote:
what it's reading is corrupt or not, that's determined at a higher level.

That seems idealistic or odd. If it is or was throwing an exception then it seems like it did know it was corrupted.

Obviously the caller continued since you said it was failing a million times. Was the caller getting usable information? If so then the architecture indicates that it is not in general corrupted. But rather that some of the data is not useable so that is how it should be treated.

Conversely, to me, if I was seeing a million actual failures then I would question the need to attempt to retrieve 'valid' data. Corrupted generally would refer to some random process. That random process might create a character that passes your checks (regardless of how you check) but still represent bad data. Matter of fact the more data like this that exists the more likely that becomes. So "millions" would suggest that some other bad data would end up being accepted as good.

PIEBALDconsult21-Mar-23 5:50

21-Mar-23 5:50

jschell wrote:
seems like it did know it was corrupted.

No, the code had a bug whereby it "took too long" to read a value/token. I have fixed the bug so now the value/token gets read and returned and then the caller can determine whether or not the value is reasonable.

jschell wrote:
failing a million times

No, not failing a million times, failing once.

jschell wrote:
the architecture indicates that it is not in general corrupted

In general, yes, non-ASCII characters cannot be considered invalid by the reader, only the caller can determine that.
A large text value may contain a few non-ASCII characters, such as "smart quotes", and that's fine. The issue was that when a value which contains many non-ASCII characters, it took too long to read -- it also happens that the value was corrupt in this case.

jschell wrote:
Corrupted generally would refer to some random process.

Yes, and we still don't know what is causing the corruption.

jschell wrote:
random process might create a character that passes your checks (regardless of how you check) but still represent bad data

As is always the case, and again the reader can't determine that.

jschell wrote:
some other bad data would end up being accepted as good.

As is always the case, and again the reader can't determine that.

In this particular case of corruption, the process will fail when it tries to stuff a million-plus characters into a database column which allows only two-hundred. But yes, even now there may be a file on the way which is corrupt in such a way that the bad value fits and the load won't fail. The reader won't care, the loader won't care, but some other part of the process will (probably) freak out.

jschell23-Mar-23 5:55

23-Mar-23 5:55

PIEBALDconsult wrote:
No, not failing a million times, failing once.

You did say in the OP the following which seemed to me to suggest that it would have failed a million times.

"Apparently having the catch fire occasionally is OK, but firing a million times in rapid succession is decidedly not."

PIEBALDconsult wrote:
In general, yes, non-ASCII characters cannot be considered invalid by the reader, only the caller can determine that

Back to being too idealistic to me in terms of your original point of how the code should handle it.

The method is only part of this process. It is not part of a multi-use library (presumably.) And the problem being solved does involve characters that are not ascii. So nothing wrong with this method specifically dealing with that by using the 'if' solution.

You could, idealistically, make the caller deal with it before calling the method in the first place. To me even spending time on that consideration seems like overkill. It would not be worth my time for consideration.

PIEBALDconsult23-Mar-23 6:53

ChatGTP: Write me a poem about programming

23-Mar-23 6:53

Having the catch fire is not a failure.

jschell wrote:
It is not part of a multi-use library

Yes, it is, of course it is.

jschell wrote:
deal with it before calling the method

Uh, what? You can't handle characters before they've been read.

Marc Clifton1-Feb-23 3:00

Marc Clifton

1-Feb-23 3:00

I think Chris will have to update the Lounge rules - no posts about what ChatGTP says!

Me: Write me a poem about programming
Chat GPT:

Programming, so complex and so hard
It can make your head spin, like a spinning top
It can be difficult to understand
But once you do, it can never be topped

From the basics of HTML to the depths of Java
Programming is a skill that's hard to master
But with enough dedication and hard work
You can be a coding master

The possibilities are endless with programming
You can create something that's truly unique
From a simple website to a complex game
You can make something that's truly one-of-a-kind

It can be frustrating and time-consuming
But the rewards can be great
You can make something that others can use
And that's a feeling that can't be beat

So don't give up when it gets tough
And don't let the obstacles stand in your way
With enough dedication and hard work
You too can be a coding master one day!

Laugh | :laugh:

Latest Article:
SVG Grids: Squares, Triangles, Hexagons with scrolling, sprites and simple animation examples

Sean Ewington1-Feb-23 3:02

Sean Ewington

1-Feb-23 3:02

It bothers me that this would work as a reasonably structured song if you made a chorus about being a coding master.

Thanks,
Sean Ewington
CodeProject

Slacker00716-Feb-23 1:13

Slacker007

16-Feb-23 1:13

Sean Ewington wrote:
this would work as a reasonably structured song if you made a chorus about being a coding master.

Reminds me of the song: Juke Box Hero - YouTube[^]

Andre Oosthuizen28-Feb-23 2:03

Andre Oosthuizen

28-Feb-23 2:03

One of my ultimate bands and songs! Thumbs Up | :thumbsup:

Daniel Pfeffer1-Feb-23 3:22

Daniel Pfeffer

1-Feb-23 3:22

Not anywhere near as good as this :

DR. SEUSS EXPLAINS COMPUTERS
(Original author unknown) Extracted from A Grandchild's Guide to Using Grandpa's Computer by Gene Zeigler

If a packet hits a pocket on a socket on a port,
And the bus is interrupted as a very last resort,
And the address of the memory makes your floppy disc abort,
Then the socket packet pocket has an error to report.

If your cursor finds a menu item followed by a dash,
And the double-clicking icon puts your window in the trash,
And your data is corrupted 'cause the index does not flash,
Then your situation’s hopeless and your system’s gonna crash!

If the label on the cable on the table at your house,
Says the network is connected to the button on your mouse,
But your packets want to tunnel on another protocol,
That’s repeatedly rejected by the printer down the hall,

And your screen is all distorted by the side effects of gauss,
So your icons in the window are as wavy as a souse,
Then you may as well reboot and go out with a bang,
'cause as sure as I’m a poet, the suckers gonna hang.

When the copy of your floppy’s getting sloppy on the disc,
And the micro code instructions cause unnecessary risk,
Then you have to flash your memory, and you’ll want to RAM your ROM…
Quickly turn off your computer and go and tell your Mom.

Freedom is the freedom to say that two plus two make four. If that is granted, all else follows.
-- 6079 Smith W.

Slacker0071-Feb-23 4:33

Slacker007

1-Feb-23 4:33

We are no longer allowed to appreciate Dr. Seuss in America as he has been deemed a racist. Just by reading this post I could technically, literally, and figuratively be canceled.

Daniel Pfeffer1-Feb-23 6:41

Daniel Pfeffer

1-Feb-23 6:41

That is entirely USians' loss.

I followed the controversy as it developed in the US. All I can say is that if an author can be deemed racist for a couple of elements in one drawing out of N illustrated books, then something is very wrong in the US. I remember reading the controversial book (And to think I saw it on Mulberry Street, IIRC) as a child, and all I thought about the drawings of "foreigners" was that they certainly dress funnily in "foreign parts".

Freedom is the freedom to say that two plus two make four. If that is granted, all else follows.
-- 6079 Smith W.