python - What is the most efficient way to get first and last line of a text file?

Question

I have a text file which contains a time stamp on each line. My goal is to find the time range. All the times are in order so the first line will be the earliest time and the last line will be the latest time. I only need the very first and very last line. What would be the most efficient way to get these lines in python?

Note: These files are relatively large in length, about 1-2 million lines each and I have to do this for several hundred files.

Answer 1

To read both the first and final line of a file you could...

• open the file, ...
• ... read the first line using built-in readline(), ...
• ... seek (move the cursor) to the end of the file, ...
• ... step backwards until you encounter EOL (line break) and ...
• ... read the last line from there.

def readlastline(f):
    f.seek(-2, 2)              # Jump to the second last byte.
    while f.read(1) != b"
":  # Until EOL is found ...
        f.seek(-2, 1)          # ... jump back, over the read byte plus one more.
    return f.read()            # Read all data from this point on.
    
with open(file, "rb") as f:
    first = f.readline()
    last = readlastline(f)

Jump to the second last byte directly to prevent trailing newline characters to cause empty lines to be returned*.

The current offset is pushed ahead by one every time a byte is read so the stepping backwards is done two bytes at a time, past the recently read byte and the byte to read next.

The whence parameter passed to fseek(offset, whence=0) indicates that fseek should seek to a position offset bytes relative to...

• 0 or os.SEEK_SET = The beginning of the file.
• 1 or os.SEEK_CUR = The current position.
• 2 or os.SEEK_END = The end of the file.

* As would be expected as the default behavior of most applications, including print and echo, is to append one to every line written and has no effect on lines missing trailing newline character.

Efficiency

1-2 million lines each and I have to do this for several hundred files.

I timed this method and compared it against against the top answer.

10k iterations processing a file of 6k lines totalling 200kB: 1.62s vs 6.92s.
100 iterations processing a file of 6k lines totalling 1.3GB: 8.93s vs 86.95.

Millions of lines would increase the difference a lot more.

Exakt code used for timing:

with open(file, "rb") as f:
    first = f.readline()     # Read and store the first line.
    for last in f: pass      # Read all lines, keep final value.

Amendment

A more complex, and harder to read, variation to address comments and issues raised since.

• Return empty string when parsing empty file, raised by comment.
• Return all content when no delimiter is found, raised by comment.
• Avoid relative offsets to support text mode, raised by comment.
• UTF16/UTF32 hack, noted by comment.

Also adds support for multibyte delimiters, readlast(b'X<br>Y', b'<br>', fixed=False).

Please note that this variation is really slow for large files because of the non-relative offsets needed in text mode. Modify to your need, or do not use it at all as you're probably better off using f.readlines()[-1] with files opened in text mode.

#!/bin/python3

from os import SEEK_END

def readlast(f, sep, fixed=True):
    r"""Read the last segment from a file-like object.

    :param f: File to read last line from.
    :type  f: file-like object
    :param sep: Segment separator (delimiter).
    :type  sep: bytes, str
    :param fixed: Treat data in ``f`` as a chain of fixed size blocks.
    :type  fixed: bool
    :returns: Last line of file.
    :rtype: bytes, str
    """
    bs   = len(sep)
    step = bs if fixed else 1
    if not bs:
        raise ValueError("Zero-length separator.")
    try:
        o = f.seek(0, SEEK_END)
        o = f.seek(o-bs-step)    # - Ignore trailing delimiter 'sep'.
        while f.read(bs) != sep: # - Until reaching 'sep': Read sep-sized block
            o = f.seek(o-step)   #  and then seek to the block to read next.
    except (OSError,ValueError): # - Beginning of file reached.
        f.seek(0)
    return f.read()

def test_readlast():
    from io import BytesIO, StringIO
    
    # Text mode.
    f = StringIO("first
last
")
    assert readlast(f, "
") == "last
"
    
    # Bytes.
    f = BytesIO(b'first|last')
    assert readlast(f, b'|') == b'last'
    
    # Bytes, UTF-8.
    f = BytesIO("X
Y
".encode("utf-8"))
    assert readlast(f, b'
').decode() == "Y
"
    
    # Bytes, UTF-16.
    f = BytesIO("X
Y
".encode("utf-16"))
    assert readlast(f, b'
x00').decode('utf-16') == "Y
"
  
    # Bytes, UTF-32.
    f = BytesIO("X
Y
".encode("utf-32"))
    assert readlast(f, b'
x00x00x00').decode('utf-32') == "Y
"
    
    # Multichar delimiter.
    f = StringIO("X<br>Y")
    assert readlast(f, "<br>", fixed=False) == "Y"
    
    # Make sure you use the correct delimiters.
    seps = { 'utf8': b'
', 'utf16': b'
x00', 'utf32': b'
x00x00x00' }
    assert "
".encode('utf8' )     == seps['utf8']
    assert "
".encode('utf16')[2:] == seps['utf16']
    assert "
".encode('utf32')[4:] == seps['utf32']
    
    # Edge cases.
    edges = (
        # Text , Match
        (""    , ""  ), # Empty file, empty string.
        ("X"   , "X" ), # No delimiter, full content.
        ("
"  , "
"),
        ("

", "
"),
        # UTF16/32 encoded U+270A (b"
x00
'
x00"/utf16)
        (b'
xe2x9cx8a
'.decode(), b'xe2x9cx8a
'.decode()),
    )
    for txt, match in edges:
        for enc,sep in seps.items():
            assert readlast(BytesIO(txt.encode(enc)), sep).decode(enc) == match

if __name__ == "__main__":
    import sys
    for path in sys.argv[1:]:
        with open(path) as f:
            print(f.readline()    , end="")
            print(readlast(f,"
"), end="")