Read 'ONLY' a part of a file in Java/Scala - Part 1
Sometime before I was wondering how log analysis system works , I started to investigate from reading a log file. Assuming the size of the log files are very large and a single thread cannot afford to read the entire data at once hence ready only a chunk of data would be a better idea.
Thats the moral of this blog post, when we say reading data in chunk we should keep remind ourselves the amount of data read / or a pointer the reader could remember so it continue reading the next chunk on subsequent calls. Ok Ok I could catch your mind voice , enough talking here show me the code !!! , here we go ....
As you can see Iam reading a URL and specify the blocksize
and the skipsize. The blocksize is fixed to fit in to the array
the skipsize helps to read the stream by ommiting skipsize * blocksize
no of arrays for eg if the skipsize is 3 , the code omits 3 * 1024 bytes
from the stream. The skipsize is supposed to be given by the caller
to as to slide through the bytes to omit.
But what I get by reading a file " chunk by chunk "
By reading data by chunk the file stream could be read indefinetly while dealing
with large file
Read any chunk of a file either from beginning , this would lead to spawning
multiple thread / actors to read the file hence making faster reads.
Conclusion
I hope I have given a bit of intro but I would like to give
more details how to better use this my next posts