Home  >  Core Java

Java StreamTokenizer Example

By Arvind Rai, July 20, 2014
Java.io.StreamTokenizer has been introduced in JDK 1.0. StreamTokenizer class parses input streams into token. These tokens will be read one at a time. StreamTokenizer can tokenize input stream on the basis of identifiers, numbers, quoted strings etc. To use StreamTokenizer we need to understand some static fields of it.

nval : if current token is number, nval gives that number.
sval : If current token is word, it gives the character of that word.
TT_EOF : This is the point that represents that end of file has been read.
TT_EOL : This represents that end of line has been read.
TT_NUMBER : This represents that a number has been read.
TT_WORD : This represents that word token has been read.
ttype : This contains the type of the token which has been read.

Simple Example of StreamTokenizer

Find the simple example how to use StreamTokenizer.
file: D:/cp/file.txt
"Hey, 10 is my lucky number.
I divide 100 by my lucky number
and again I get 10."

StreamTokenizerDemoOne.java
package com.cp.io;
import java.io.FileReader;
import java.io.IOException;
import java.io.StreamTokenizer;
public class StreamTokenizerDemoOne {
	public static void main(String args[]) throws IOException {
		FileReader fileReader = new FileReader("D:/cp/file.txt"); 
		StreamTokenizer st = new StreamTokenizer(fileReader); 
		while(st.nextToken() != StreamTokenizer.TT_EOF) { 
			if(st.ttype == StreamTokenizer.TT_NUMBER) { 
				System.out.println("Number: "+st.nval);
			} else if(st.ttype == StreamTokenizer.TT_WORD) {
				System.out.println("Word: "+st.sval); 
			}else if(st.ttype == StreamTokenizer.TT_EOL) {
				System.out.println("--End of Line--"); 
			}
		} 
	}
} 
Output will be as below.
Word: Hey
Number: 10.0
Word: is
Word: my
Word: lucky
Word: number.
Word: I
Word: divide
Number: 100.0
Word: by
Word: my
Word: lucky
Word: number
Word: and
Word: again
Word: I
Word: get
Number: 10.0



StreamTokenizer Example Using ordinaryChar () and resetSyntax () Method

ordinaryChar() : It specifies that tokenizer has "ordinary" character arguments.
resetSyntax (): It resets the tokenizer syntax table.
D:/cp/file.txt
"Add 2.0+3.0=5.0"

StreamTokenizerDemoTwo.java
package com.cp.io;
import java.io.FileReader;
import java.io.IOException;
import java.io.StreamTokenizer;
public class StreamTokenizerDemoTwo {
	public static void main(String args[]) throws IOException {
		//Using  ordinaryChars method
		FileReader fileReader = new FileReader("D:/cp/file.txt"); 
		StreamTokenizer st = new StreamTokenizer(fileReader);
		System.out.println("After Using ordinaryChar() method \n");
		st.ordinaryChar('A');
		printStreamTokenizer(st);
		fileReader.close();
		//Using  resetSyntax method
		fileReader = new FileReader("D:/cp/file.txt"); 
		st = new StreamTokenizer(fileReader);
		System.out.println("\nAfter Using resetSyntax() method \n");
		st.resetSyntax();
		printStreamTokenizer(st);
		fileReader.close();
	}
	static void printStreamTokenizer(StreamTokenizer st) throws IOException{
		int token =0; 
		while((token = st.nextToken()) != StreamTokenizer.TT_EOF) { 
			if(st.ttype == StreamTokenizer.TT_NUMBER) { 
				System.out.println("Number: "+st.nval);
			} else if(st.ttype == StreamTokenizer.TT_WORD) {
				System.out.println("Word: "+st.sval); 
			}else {
				System.out.println("Ordinary Char: "+(char)token); 
			}
		} 
	}
} 
Output is given below.
After Using ordinaryChar() method 

Ordinary Char: A
Word: dd
Number: 2.0
Ordinary Char: +
Number: 3.0
Ordinary Char: =
Number: 5.0

After Using resetSyntax() method 

Ordinary Char: A
Ordinary Char: d
Ordinary Char: d
Ordinary Char:  
Ordinary Char: 2
Ordinary Char: .
Ordinary Char: 0
Ordinary Char: +
Ordinary Char: 3
Ordinary Char: .
Ordinary Char: 0
Ordinary Char: =
Ordinary Char: 5
Ordinary Char: .
Ordinary Char: 0
POSTED BY
ARVIND RAI
ARVIND RAI
FIND MORE TUTORILAS






©2019 concretepage.com | Privacy Policy | Contact Us