Common::U32StringTokenizer Class Reference

#include <tokenizer.h>

Public Member Functions
	U32StringTokenizer (const U32String &str, const String &delimiters=" \\\)
void	reset ()
	Resets the tokenizer to its initial state, i.e points boten token iterators to the beginning.
bool	empty () const
	Returns true if there are no more tokens left in the string, false otherwise.
U32String	nextToken ()
	Returns the next token from the string (Or an empty string if there are no more tokens)
U32StringArray	split ()
	Returns StringArray with all tokens. Beware of the memory usage.
U32String	delimitersAtTokenBegin () const
	Returns a U32String with all delimiters between the current and previous token.
U32String	delimitersAtTokenEnd () const
	Returns a U32String with all delimiters between the current and next token.

Detailed Description

A simple non-optimized unicode-string tokenizer.

Example of use: U32StringTokenizer("Now, this is a test!", " ,!") gives tokens "Now", "this", "is", "a" and "test" using nextToken(). Using non-ascii chars will also work, and is recommended to use this over StringTokenizer if string contains unicode chars.

Constructor & Destructor Documentation

U32StringTokenizer()

Common::U32StringTokenizer::U32StringTokenizer

(

const U32String &

str

)

Creates a UnicodeStringTokenizer.

Parameters

str	The unicode string to be tokenized.
delimiters	String containing all the delimiter characters (i.e. the characters to be ignored).

Note

Uses space, horizontal tab, carriage return, newline, form feed and vertical tab as delimiters by default.

---

The documentation for this class was generated from the following file:

• common/tokenizer.h