Entry

Quicker file processing

Jul 5th, 2000 09:59
Nathan Wallace, unknown unknown, Hans Nowak, Snippet 21, Terry Reedy

"""
Packages: files
"""

"""
>  I am using:
>fp = open('somefile', 'r')
>while 1:
>  fp.seek(22,1)
>  strtst = fp.read(8)
>  if strtst != 'ksdfsk':
>     junk = fp.readline()
>  else:
>     DoProcess_str_stuff()

Instead of seeking, reading a very small chunk, and then reading again (3 
file ops per line), try reading each line and then testing the appropriate 
slice.  (By the way, if you read 8 bytes and test for equality with 6, you 
will never match.  Also, I do not see how the loop terminates without a 
break.)
"""

FILENAME = "c:/autoexec.bat"
# change this to a file on your system

fp = open(FILENAME, "r")

while 1:
  line = fp.readline()
  if not line: break
  if line[22:28] == 'ksdfsk': process(line)

"""
>Is there anyway to speed this process up? I am taking up most of time in 
>junk = fp.readline(), which I just throw away.

On my Windows machine, py1.4 readline()s thru thousands of lines in seconds. 
 If your program takes minutes, either something is wrong or you are 
processing 100s of mbytes and should expect some waiting.
"""