Java StreamTokenizer Example
July 20, 2014
Java.io.StreamTokenizer has been introduced in JDK 1.0. StreamTokenizer class parses input streams into token. These tokens will be read one at a time. StreamTokenizer can tokenize input stream on the basis of identifiers, numbers, quoted strings etc. To use StreamTokenizer we need to understand some static fields of it.
nval : if current token is number, nval gives that number.
sval : If current token is word, it gives the character of that word.
TT_EOF : This is the point that represents that end of file has been read.
TT_EOL : This represents that end of line has been read.
TT_NUMBER : This represents that a number has been read.
TT_WORD : This represents that word token has been read.
ttype : This contains the type of the token which has been read.
Simple Example of StreamTokenizer
Find the simple example how to use StreamTokenizer.file: D:/cp/file.txt
"Hey, 10 is my lucky number. I divide 100 by my lucky number and again I get 10."
StreamTokenizerDemoOne.java
package com.cp.io; import java.io.FileReader; import java.io.IOException; import java.io.StreamTokenizer; public class StreamTokenizerDemoOne { public static void main(String args[]) throws IOException { FileReader fileReader = new FileReader("D:/cp/file.txt"); StreamTokenizer st = new StreamTokenizer(fileReader); while(st.nextToken() != StreamTokenizer.TT_EOF) { if(st.ttype == StreamTokenizer.TT_NUMBER) { System.out.println("Number: "+st.nval); } else if(st.ttype == StreamTokenizer.TT_WORD) { System.out.println("Word: "+st.sval); }else if(st.ttype == StreamTokenizer.TT_EOL) { System.out.println("--End of Line--"); } } } }
Word: Hey Number: 10.0 Word: is Word: my Word: lucky Word: number. Word: I Word: divide Number: 100.0 Word: by Word: my Word: lucky Word: number Word: and Word: again Word: I Word: get Number: 10.0
StreamTokenizer Example Using ordinaryChar () and resetSyntax () Method
ordinaryChar() : It specifies that tokenizer has "ordinary" character arguments.resetSyntax (): It resets the tokenizer syntax table.
D:/cp/file.txt
"Add 2.0+3.0=5.0"
StreamTokenizerDemoTwo.java
package com.cp.io; import java.io.FileReader; import java.io.IOException; import java.io.StreamTokenizer; public class StreamTokenizerDemoTwo { public static void main(String args[]) throws IOException { //Using ordinaryChars method FileReader fileReader = new FileReader("D:/cp/file.txt"); StreamTokenizer st = new StreamTokenizer(fileReader); System.out.println("After Using ordinaryChar() method \n"); st.ordinaryChar('A'); printStreamTokenizer(st); fileReader.close(); //Using resetSyntax method fileReader = new FileReader("D:/cp/file.txt"); st = new StreamTokenizer(fileReader); System.out.println("\nAfter Using resetSyntax() method \n"); st.resetSyntax(); printStreamTokenizer(st); fileReader.close(); } static void printStreamTokenizer(StreamTokenizer st) throws IOException{ int token =0; while((token = st.nextToken()) != StreamTokenizer.TT_EOF) { if(st.ttype == StreamTokenizer.TT_NUMBER) { System.out.println("Number: "+st.nval); } else if(st.ttype == StreamTokenizer.TT_WORD) { System.out.println("Word: "+st.sval); }else { System.out.println("Ordinary Char: "+(char)token); } } } }
After Using ordinaryChar() method Ordinary Char: A Word: dd Number: 2.0 Ordinary Char: + Number: 3.0 Ordinary Char: = Number: 5.0 After Using resetSyntax() method Ordinary Char: A Ordinary Char: d Ordinary Char: d Ordinary Char: Ordinary Char: 2 Ordinary Char: . Ordinary Char: 0 Ordinary Char: + Ordinary Char: 3 Ordinary Char: . Ordinary Char: 0 Ordinary Char: = Ordinary Char: 5 Ordinary Char: . Ordinary Char: 0