Implementation of tokenizer. More...
#include <ctype.h>#include <stdarg.h>#include <stdbool.h>#include <stdio.h>#include <stdlib.h>#include <string.h>#include <cstdlib>#include <cstring>#include <string>#include <unordered_map>
Include dependency graph for tokenizer.h:
This graph shows which files directly or indirectly include this file:
Go to the source code of this file.
Classes | |
| struct | gymbo::Token |
| Structure representing a token. More... | |
Namespaces | |
| gymbo | |
Enumerations | |
| enum | gymbo::TokenKind { gymbo::TOKEN_RESERVED , gymbo::TOKEN_RETURN , gymbo::TOKEN_IF , gymbo::TOKEN_ELSE , gymbo::TOKEN_FOR , gymbo::TOKEN_IDENT , gymbo::TOKEN_NUM , gymbo::TOKEN_EOF } |
| Enumeration representing different token kinds. More... | |
Functions | |
| bool | gymbo::is_alpha (char c) |
| Checks if a character is an alphabetical character or underscore. More... | |
| bool | gymbo::is_alnum (char c) |
| Checks if a character is an alphanumeric character. More... | |
| void | gymbo::error (char *fmt,...) |
| Reports an error and exits the program. More... | |
| void | gymbo::error_at (char *user_input, char *loc, char *fmt,...) |
| Reports an error location and exits the program. More... | |
| bool | gymbo::consume (Token *&token, char *op) |
Consumes the current token if it matches op. More... | |
| bool | gymbo::consume_tok (Token *&token, TokenKind tok) |
Consumes the current token if it matches tok. More... | |
| Token * | gymbo::consume_ident (Token *&token) |
| Consumes the current token if it is an identifier. More... | |
| void | gymbo::expect (Token *&token, char *user_input, char *op) |
Ensures that the current token matches op. More... | |
| float | gymbo::expect_number (Token *&token, char *user_input) |
| Ensures that the current token is a number. More... | |
| bool | gymbo::at_eof (Token *token) |
| Checks if the current token is at the end of the program. More... | |
| Token * | gymbo::new_token (TokenKind kind, Token *cur, char *str, int len) |
Creates a new token and adds it as the next token of cur. More... | |
| bool | gymbo::startswith (char *p, char *q) |
Checks if the string p starts with the string q. More... | |
| Token * | gymbo::tokenize (char *user_input, std::unordered_map< std::string, int > &var_counter) |
| Tokenizes a given string and returns a linked list of tokens. More... | |
Detailed Description
Implementation of tokenizer.
