Struck with this Task

[Rahulbatra]

I have working knowledge in C and now I have read some concepts of compiler design and I am trying to implement the basics of these in my C program. I am struck with the task of breaking the given string into token for further analyzing them in parser. Can anyone suggest me a method for accomplishing this task?

[cyrus]

Suppose you want to break the string every 2 bits and make it as tokens you can use the below code to do the same.

char a[20];
int l;
while((l = fread(a,2,f)) > 0)
display(buf,l);

By using the above code you can break the given string into token for further analyzing them in parser.