Java StreamTokenizer Example

By Arvind Rai, July 20, 2014
Java.io.StreamTokenizer has been introduced in JDK 1.0. StreamTokenizer class parses input streams into token. These tokens will be read one at a time. StreamTokenizer can tokenize input stream on the basis of identifiers, numbers, quoted strings etc. To use StreamTokenizer we need to understand some static fields of it.

nval : if current token is number, nval gives that number.
sval : If current token is word, it gives the character of that word.
TT_EOF : This is the point that represents that end of file has been read.
TT_EOL : This represents that end of line has been read.
TT_NUMBER : This represents that a number has been read.
TT_WORD : This represents that word token has been read.
ttype : This contains the type of the token which has been read.

Simple Example of StreamTokenizer

Find the simple example how to use StreamTokenizer.
file: D:/cp/file.txt
"Hey, 10 is my lucky number.
I divide 100 by my lucky number
and again I get 10."

StreamTokenizerDemoOne.java
package com.cp.io;
import java.io.FileReader;
import java.io.IOException;
import java.io.StreamTokenizer;
public class StreamTokenizerDemoOne {
	public static void main(String args[]) throws IOException {
		FileReader fileReader = new FileReader("D:/cp/file.txt"); 
		StreamTokenizer st = new StreamTokenizer(fileReader); 
		while(st.nextToken() != StreamTokenizer.TT_EOF) { 
			if(st.ttype == StreamTokenizer.TT_NUMBER) { 
				System.out.println("Number: "+st.nval);
			} else if(st.ttype == StreamTokenizer.TT_WORD) {
				System.out.println("Word: "+st.sval); 
			}else if(st.ttype == StreamTokenizer.TT_EOL) {
				System.out.println("--End of Line--"); 
			}
		} 
	}
} 
Output will be as below.
Word: Hey
Number: 10.0
Word: is
Word: my
Word: lucky
Word: number.
Word: I
Word: divide
Number: 100.0
Word: by
Word: my
Word: lucky
Word: number
Word: and
Word: again
Word: I
Word: get
Number: 10.0

StreamTokenizer Example Using ordinaryChar () and resetSyntax () Method

ordinaryChar() : It specifies that tokenizer has "ordinary" character arguments.
resetSyntax (): It resets the tokenizer syntax table.
D:/cp/file.txt
"Add 2.0+3.0=5.0"

StreamTokenizerDemoTwo.java
package com.cp.io;
import java.io.FileReader;
import java.io.IOException;
import java.io.StreamTokenizer;
public class StreamTokenizerDemoTwo {
	public static void main(String args[]) throws IOException {
		//Using  ordinaryChars method
		FileReader fileReader = new FileReader("D:/cp/file.txt"); 
		StreamTokenizer st = new StreamTokenizer(fileReader);
		System.out.println("After Using ordinaryChar() method \n");
		st.ordinaryChar('A');
		printStreamTokenizer(st);
		fileReader.close();
		//Using  resetSyntax method
		fileReader = new FileReader("D:/cp/file.txt"); 
		st = new StreamTokenizer(fileReader);
		System.out.println("\nAfter Using resetSyntax() method \n");
		st.resetSyntax();
		printStreamTokenizer(st);
		fileReader.close();
	}
	static void printStreamTokenizer(StreamTokenizer st) throws IOException{
		int token =0; 
		while((token = st.nextToken()) != StreamTokenizer.TT_EOF) { 
			if(st.ttype == StreamTokenizer.TT_NUMBER) { 
				System.out.println("Number: "+st.nval);
			} else if(st.ttype == StreamTokenizer.TT_WORD) {
				System.out.println("Word: "+st.sval); 
			}else {
				System.out.println("Ordinary Char: "+(char)token); 
			}
		} 
	}
} 
Output is given below.
After Using ordinaryChar() method 

Ordinary Char: A
Word: dd
Number: 2.0
Ordinary Char: +
Number: 3.0
Ordinary Char: =
Number: 5.0

After Using resetSyntax() method 

Ordinary Char: A
Ordinary Char: d
Ordinary Char: d
Ordinary Char:  
Ordinary Char: 2
Ordinary Char: .
Ordinary Char: 0
Ordinary Char: +
Ordinary Char: 3
Ordinary Char: .
Ordinary Char: 0
Ordinary Char: =
Ordinary Char: 5
Ordinary Char: .
Ordinary Char: 0
POSTED BY
ARVIND RAI
ARVIND RAI







©2024 concretepage.com | Privacy Policy | Contact Us