faqts : Computers : Programming : Languages : Python : Snippets

+ Search
Add Entry AlertManage Folder Edit Entry Add page to http://del.icio.us/
Did You Find This Entry Useful?

2 of 2 people (100%) answered Yes
Recently 1 of 1 people (100%) answered Yes


ASCII delimited files

Jul 5th, 2000 10:02
Nathan Wallace, Hans Nowak, Snippet 278, D'Arcy J.M. Cain

Packages: text.delimited_files
> I couldn't find this on the Python or Starship sites but you can get it
> from http://www.deja.com/getdoc.xp?AN=448769706.  One small problem with
> it is that it doesn't handle commas within quoted strings properly.  I
> am going to look at this today for another project and I will post it
> here when I have something.
And here it is.  Note that Christian Tismer wrote the original (at least
he posted it) but didn't leave his name in the script itself.  I didn't
want to add it in since he may have had his reasons but I wanted to
acknowledge that he wrote it.
One thing I didn't change was the replacement of newlines with commas
although I'm not sure that that is the best thing to replace it with.
In fact, why replace it at all?  I may change this for myself.
Oh, if anyone has a suggestion for a more efficient way to handle
embedded commas I sure would be interested.
#! /usr/bin/env python
# delimited.py
# Modified from the original by D'Arcy J.M. Cain <[email protected]>
# CT990226 V0.1
breaking tab delimited or CSV text into a list of tuples.
Original version: SMC databook plugin, readxls.py
Generalized to be usable for any delimiter but \n.
import string
def split_delimited(s, delimiter=None) :
	"""split a delimited text file string into a list of tuples.
	Quotes are obeyed, enclosed newlines are expanded to tab,
	double quotes go to quotes"""
	# 980426 finding line delimiter dynamically
	probe = s[:10000]
	eol = findlinedelimiter(probe)
	# 990226 guessing field delimiter from '\t,' if not supplied
	if not delimiter:
		candidates = [
			(string.count(probe, '\t'), '\t'),
			(string.count(probe, ','), ','),
		delimiter = candidates[-1][-1]
	del probe
	# the trick to make this handy is to use \0 as a placeholder
	# Kind of ugly but it works for embedded commas - DJMC
	inquote = 0
	for i in range(len(s)):
		if s[i] == '"': inquote = (inquote == 0)
		if inquote == 0 and s[i] == delimiter:
			s = s[:i] + "\0" + s[i + 1:]
	parts = string.split(s, '"')
	limits = (0, len(parts)-1)
	for i in range(len(parts)) :
		# may as well strip the spaces now - DJMC
		part = string.strip(parts[i])
		if i%2 :
			part = string.replace(part, eol, delimiter)
		else :
			if not part and i not in limits: part = '"'
		parts[i] = part
	# merge it back
	txt = string.join(parts, "")
	parts = string.split(txt, eol)
	# now break by \0
	for i in range(len(parts)) :
		fields = string.split(parts[i], "\0")
		parts[i] = tuple(fields)
	return parts
# utilities
def findlinedelimiter(txt) :
	provide some kb of text to this function. It will determine
	the best delimiter and therefore guess the system
	mac  = "\x0D"
	unix = "\x0A"
	dos  = mac+unix
	oses = [dos, unix, mac]
	# find the one which gives the most lines.
	# in doubt, the longest delimiter wins.
	lis = []
	while txt and txt[-1] in dos: txt = txt[:-1] # CT970904
	for delim in oses:
		lis.append((len(string.split(txt, delim)), delim))
	return lis[-1][-1]
if __name__=="__main__":
	for l in  split_delimited('''1, 2,3, "vier", "quo""te", "embedded,
comma", "this
is with a newline", here
another record'''):
		print l
# eof