Tokenizing in MEL to Remember Empty Strings

I encountered a strange thing recently.  Turns out when you tokenize a string (the act of splitting up consecutive letters by a delimiter) in MEL scripting, it doesn’t remember empty strings.  This can be a problem when you need to have the same number of elements in an array, but it resizes to an arbitrary count.

For example, using the MEL tokenize command, the string “d,s,e,,,d” would be split into an array of only four elements {“d” “s” “e” “d”}.  If you wanted to return the empty values, the method as it stands would not do this.

I’ve authored a small function that will return empty strings.  In the aforementioned example, it would return {“d” “s” “e” “” “” “d”}.   The command is called “RealTokenize.”

global proc string[] RealTokenize(string $CurStr, string $Delim) {
	//Author: Nick Pisca 2010 0001d
	//string $CurStr = "d,s,d,s,,,,e"; string $Delim = ",";
	//RealTokenize "d,s,d,,,e,e" ",";
	string $PartString;
	string $TempArr[];
	int $t = 0;
	for ($x=1;$x<size($CurStr)+1;$x++) {
		string $CurLetter = `substring $CurStr $x $x`;
		if ($CurLetter == $Delim) {
			$TempArr[$t] = $PartString;
			$t++;
			$PartString = "";
		} else {
			$PartString = $PartString + $CurLetter;
		}
	}
	return $TempArr;
}

This subroutine is pretty simple to use and unlike “tokenize,” it doesn’t use that dumb syntax that returns the “count” of elements.  I never understood why anyone would use that functionality, considering you can just “size” the array after relimiting the string.  Enjoy.