Tokenizing strings using _tcstok_s

So what is Tokenizing, eh?

Well it’s the process of splitting a string into smaller strings (called tokens) based on a given delimiter string. For e.g. If we have to split a string called “Nibu Babu Thomas” 😉 and the delimiter string is ” ” (space). So our tokens will be

1. Nibu
2. Babu
3. Thomas

So for this purpose we have a standard function called _tcstok_s. Its the safe version of _tcstok/strtok. _t stands for TCHAR version in windows. It’s nothing but a #define for unicode and non unicode.

Looks something like this…

#ifdef _UNICODE
   #define _tcstok_s wcstok_s
#else
  #define _tcstok_s strtok_s
#endif

So what’s the difference between _tcstok and _tcstok_s? _tcstok uses a static buffer internally to maintain token stream position, so that when called again it can start from this position. The problem with this approach is that during simultaneous calls this stream position is overwritten leading to unpredictable results but is safe to call from multiple threads since the static variable will be thread specific something like…

char* strtok(...)
{
    __declspec(thread) static char* _streampos_ = 0;
    ...
}

But now caller is to provide a context pointer to _tcstok_s function to maintain stream position, so this is the additional parameter for _tcstok_s.

Here is a simple function which tokenizes a given string and returns the result in a string vector.

#include <assert .h>
#include <vector>

void Tokenize( char* str, const char* splitstr, std::vector<std ::string>& Tokens)
{
    assert( str && splitstr );

    char* Pos = 0;
    char* Token = strtok_s( str, splitstr, &Pos ); // Get first token

    while( Token )
    {
        Tokens.push_back( Token );

        // Get next token, note that first parameter is NULL
        Token = strtok_s( 0, splitstr, &Pos );
    }
}

Download sample application, rename to zip. Compiled using VS2008.