Click here to Skip to main content
15,887,175 members
Home / Discussions / C / C++ / MFC
   

C / C++ / MFC

 
GeneralRe: c++ Pin
CPallini21-Jun-22 20:20
mveCPallini21-Jun-22 20:20 
GeneralRe: c++ Pin
tuanguyen200021-Jun-22 22:50
tuanguyen200021-Jun-22 22:50 
GeneralRe: c++ Pin
tuanguyen200021-Jun-22 22:51
tuanguyen200021-Jun-22 22:51 
GeneralRe: c++ Pin
CPallini21-Jun-22 23:08
mveCPallini21-Jun-22 23:08 
GeneralRe: c++ Pin
Richard MacCutchan22-Jun-22 1:20
mveRichard MacCutchan22-Jun-22 1:20 
QuestionMessage Closed Pin
16-Jun-22 5:53
Member 1496877116-Jun-22 5:53 
AnswerRe: parsing interactive text Pin
Mircea Neacsu16-Jun-22 8:09
Mircea Neacsu16-Jun-22 8:09 
AnswerRe: parsing interactive text Pin
trønderen16-Jun-22 10:52
trønderen16-Jun-22 10:52 
Member 14968771 wrote:
is "tokenization" a good search word ??
Fair enough, but that is only the very beginning. A precursor to parsing.

Tokenization is the chopping into atomic pieces of the input text, with no concern for how they are put together. All the tokenizer knows is how to delimit a symbol (token): That a word symbol start with a alphabetic and continues through alphanumerics but ends at the first non-alphanumeric - the tokenizer doesn't know or care whether the word is a variable name, a reserved word or something else. If it finds a digit, it devours digits. If the first non-digit is a math operator or a space, it has found an integer token. If it is a decimal point or an E (and the language permits exponents in literals), the token is a (yet incomplete) float value, and so on. The only language specific thing that the tokenizer needs to know is how to identify the end of a token. Once it has chopped the source code into pieces, its job is done.

Parsing is identifying the structures formed by the tokens. Identifying block, loops, conditional statements etc.

The borderline isn't necessarily razor sharp. Some would say that when the tokenizer finds an integer literal token, it might as well take the the task of converting it to a binary numeric token value, to be handed to the parser. That might be unsuitable in untyped languages where a numeric literal may be treated as a string. After identifying a word symbol, it might search a table of reserved words, possibly delivering it to the parser as a reserved word token. Again, in some languages this is unsuitable (and lots of people would say it goes far beyond a tokenizer's responsibility).

If you want to analyze some input, doing an initial tokenization before starting the actual parsing is a good idea. Most compilers do that.

Curious memory:
One of my fellow students was in his first job after graduation set to identify bacteria in microscope photos. That was done by parsing: They had BNF grammars for different kinds of bacteria, and the image information was parsed according to the various grammars. If the number of parsing errors was too high, the verdict was 'Nope - it surely isn't that kind of bacteria, let me try another one!' Those grammars with a low error count was handed over to a human expert for confirmation, or possibly making a choice between viable alternatives, if two or more grammars gave a low error count. This mechanism took a lot of trivial work off the medical personnel, and the computer could scan far more images for possibly dangerous bacteria than there would be human resources to do. The university lecturer in the Compilers and Compilation course certainly hadn't prepared us for compiling bacteria!
JokeRe: parsing interactive text Pin
Peter_in_278016-Jun-22 11:56
professionalPeter_in_278016-Jun-22 11:56 
AnswerRe: parsing interactive text Pin
Richard MacCutchan18-Jun-22 2:50
mveRichard MacCutchan18-Jun-22 2:50 
QuestionCan importing C++ DLL files in a C# project has a positive effect on the overall performance of the application? Pin
Code4Ever10-Jun-22 8:39
Code4Ever10-Jun-22 8:39 
AnswerRe: Can importing C++ DLL files in a C# project has a positive effect on the overall performance of the application? Pin
Dave Kreskowiak10-Jun-22 9:56
mveDave Kreskowiak10-Jun-22 9:56 
QuestionLoading DLL from file not PATH Pin
Valentinor8-Jun-22 22:59
Valentinor8-Jun-22 22:59 
AnswerRe: Loading DLL from file not PATH Pin
Victor Nijegorodov8-Jun-22 23:16
Victor Nijegorodov8-Jun-22 23:16 
GeneralRe: Loading DLL from file not PATH Pin
Valentinor8-Jun-22 23:48
Valentinor8-Jun-22 23:48 
GeneralRe: Loading DLL from file not PATH Pin
Victor Nijegorodov8-Jun-22 23:52
Victor Nijegorodov8-Jun-22 23:52 
GeneralRe: Loading DLL from file not PATH Pin
Valentinor9-Jun-22 0:04
Valentinor9-Jun-22 0:04 
AnswerRe: Loading DLL from file not PATH Pin
Richard MacCutchan9-Jun-22 3:27
mveRichard MacCutchan9-Jun-22 3:27 
GeneralRe: Loading DLL from file not PATH Pin
Valentinor9-Jun-22 6:21
Valentinor9-Jun-22 6:21 
GeneralRe: Loading DLL from file not PATH Pin
Victor Nijegorodov9-Jun-22 6:38
Victor Nijegorodov9-Jun-22 6:38 
GeneralRe: Loading DLL from file not PATH Pin
Richard MacCutchan9-Jun-22 6:43
mveRichard MacCutchan9-Jun-22 6:43 
AnswerRe: Loading DLL from file not PATH Pin
Randor 10-Jun-22 6:54
professional Randor 10-Jun-22 6:54 
QuestionRe: Loading DLL from file not PATH Pin
Valentinor14-Jun-22 3:26
Valentinor14-Jun-22 3:26 
AnswerRe: Loading DLL from file not PATH Pin
Richard MacCutchan14-Jun-22 5:10
mveRichard MacCutchan14-Jun-22 5:10 
GeneralRe: Loading DLL from file not PATH Pin
Valentinor14-Jun-22 5:41
Valentinor14-Jun-22 5:41 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.