Split Join

lua-users home
wiki

"split" [1] and "join" [2] are two common string operators that are essentially inverse operations of each other. Split separates a string containing a delimiter into the list of substrings between that delimiter. Join combines a list of strings into a new string by inserting a delimiter between each string.

There are various ways to design and implement these functions in Lua, as described below.

Joining list of strings

With Lua 5.x you can use table.concat[3] for joining: table.concat(tbl, delimiter_str).

table.concat({"a", "b", "c"}, ",") --> "a,b,c"

Other interfaces are possible, largely dependent on the choice of split interface since join is often intended to be the inverse operation of split.

Splitting Strings

First of all, although Lua does not have a split function is its standard library, it does have string.gmatch[4], which can be used instead of a split function in many cases. Unlike a split function, string.gmatch takes a pattern to match the non-delimiter text, instead of the delimiters themselves:

local example = "an example string"
for i in string.gmatch(example, "%S+") do
  print(i)
end

-- output:
-- an
-- example
-- string

A split[1] function separates a string into a list of substrings, breaking the original string on occurrences of some separator (character, character set, or pattern). There are various ways to design a string split function. A summary of the design decisions is listed below.

Should split return a table array, a list, or an iterator?

split("a,b,c", ",") --> {"a", "b", "c"}
split("a,b,c", ",") --> "a","b","c" (not scalable: Lua has a limit of a few thousand return values)
for x in split("a,b,c", ",") do ..... end

Should the separator be a string, Lua pattern, LPeg pattern, or regular expression?

split("a  +b c", " +") --> {"a ", "b c"}
split("a  +b c", " +") --> {"a", "+b", "c"}
split("a  +b c", some_other_object) --> .....

How should empty separators be handled?

split("abc", "") --> {"a", "b", "c"} 
split("abc", "") --> {"", "a", "b", "c", ""}
split("abc", "") --> error
split("abc", "%d*") --> what about patterns that can evaluate to empty strings?

Note: split(s,"") is a convenient idiom for splitting a string into characters. In Lua, we can alternately do for c in s:gmatch"." do ..... end.

How should empty values be handled?

split(",,a,b,c,", ",") --> {"a", "b", "c"}
split(",,a,b,c,", ",") --> {"", "", "a", "b", "c", ""}
split(",", ",") --> {} or {""} or {"", ""} ?
split("", ",") --> {} or {""} ?

Note: Although splitting and joining are roughly inverses, the operations are not necessarily uniquely determined, particularly when there are empty strings. join({"",""}, ""), join({""}, "") and join({}, "") all result in the same string "". Therefore, the choice of what the inverse operation split("", "") should return is not immediately clear.

Note: completely ignoring empty values can be undesirable, such as for rows in a CSV file where column positions matter. A CSV file with empty rows "" is unclear: is this an column containing an empty value or are there zero columns? A zero column CSV file is unlikely though perhaps not impossible.

Should there be an argument to limit the number of splits?

split("a,b,c", ",", 2) --> {"a", "b,c"}

Should the separator be returned? This is more useful when the separator is a pattern, in which case the separator can vary:

split("a  b c", " +") --> {"a", "  ", "b", " ", "c"}

Note: Note also that string.gmatch [5] is in a way a dual of split, returning the substrings that match a pattern and discarding strings between them rather than the other way around. A function that returns both is sometimes called partition [6].


Method: Split by pattern using string.gsub/string.match

Break a string up at occurrences of a single character. If the number of fields is known:

str:match( ("([^"..sep.."]*)"..sep):rep(nsep) )

If the number of fields is not known

fields = {str:match((str:gsub("[^"..sep.."]*"..sep, "([^"..sep.."]*)"..sep)))}

Some might call the above a hack :) sep will need to be escaped if it is a pattern metacharacter, and you'd probably be better off precomputing and/or memoizing the patterns.


Method: Using only string.gsub

fields = {}
str:gsub("([^"..sep.."]*)"..sep, function(c) table.insert(fields, c) end)

Does not work as expected:

str, sep = "1:2:3", ":"
fields = {}
str:gsub("([^"..sep.."]*)"..sep, function(c) table.insert(fields, c) end)
for i,v in ipairs(fields) do print(i,v) end

-- output:
-- 1        1
-- 2        2

Fix:

function string:split(sep)
        local sep, fields = sep or ":", {}
        local pattern = string.format("([^%s]+)", sep)
        self:gsub(pattern, function(c) fields[#fields+1] = c end)
        return fields
end

Example: split a string into words, or return nil

function justWords(str)
  local t = {}
  local function helper(word) table.insert(t, word) return "" end
  if not str:gsub("%w+", helper):find"%S" then return t end
end


Method: Split a string with a pattern, Take One

This splits a string using the pattern sep. It calls func for each segment. When func is called, the first argument is the segment and the remaining arguments are the captures from sep, if any. On the last segment, func will be called with just one argument. (This could be used as a flag, or you could use two different functions). sep must not match the empty string. Enhancements are left as an exercise :)

func((str:gsub("(.-)("..sep..")", func)))

Example: Split a string into lines separated by either DOS or Unix line endings, creating a table out of the results.

function lines(str)
  local t = {}
  local function helper(line) table.insert(t, line) return "" end
  helper((str:gsub("(.-)\r?\n", helper)))
  return t
end

Function: Split a string with a pattern, Take Two

The problem with using gsub as above is that it can't handle the case when the separator pattern doesn't appear at the end of the string. In that case the final "(.-)" never gets to capture the end of the string, because the overall pattern fails to match. To handle that case you have to do something a little more complicated. The split function below behaves more or less like split in perl or python. In particular, single matches at the beginning and end of the string do not create new elements. Multiple matches in a row create empty string elements.

-- Compatibility: Lua-5.1
function split(str, pat)
   local t = {}  -- NOTE: use {n = 0} in Lua-5.0
   local fpat = "(.-)" .. pat
   local last_end = 1
   local s, e, cap = str:find(fpat, 1)
   while s do
      if s ~= 1 or cap ~= "" then
	 table.insert(t,cap)
      end
      last_end = e+1
      s, e, cap = str:find(fpat, last_end)
   end
   if last_end <= #str then
      cap = str:sub(last_end)
      table.insert(t, cap)
   end
   return t
end

Example: Split a file path string into components.

function split_path(str)
   return split(str,'[\\/]+')
end

parts = split_path("/usr/local/bin")
--> {'usr','local','bin'}

Test Cases:

split('foo/bar/baz/test','/')
--> {'foo','bar','baz','test'}
split('/foo/bar/baz/test','/')
--> {'foo','bar','baz','test'}
split('/foo/bar/baz/test/','/')
--> {'foo','bar','baz','test'}
split('/foo/bar//baz/test///','/')
--> {'foo','bar','','baz','test','',''}
split('//foo////bar/baz///test///','/+')
--> {'foo','bar','baz','test'}
split('foo','/+')
--> {'foo'}
split('','/+')
--> {}
split('foo','')  -- opps! infinite loop!

Function: Split a string with a pattern, Take Three

After a discussion on this topic in the mailing list, I made my own function... I took, unknowingly, a way similar to the function above, except I use gfind to iterate, and I see the single matches at beginning and end of string as empty fields. As above, multiple successive delimiters create empty string elements.

-- Compatibility: Lua-5.0
function Split(str, delim, maxNb)
    -- Eliminate bad cases...
    if string.find(str, delim) == nil then
        return { str }
    end
    if maxNb == nil or maxNb < 1 then
        maxNb = 0    -- No limit
    end
    local result = {}
    local pat = "(.-)" .. delim .. "()"
    local nb = 0
    local lastPos
    for part, pos in string.gfind(str, pat) do
        nb = nb + 1
        result[nb] = part
        lastPos = pos
        if nb == maxNb then break end
    end
    -- Handle the last field
    if nb ~= maxNb then
        result[nb + 1] = string.sub(str, lastPos)
    end
    return result
end

Test Cases:

ShowSplit("abc", '')
--> { [1] = "", [2] = "", [3] = "", [4] = "", [5] = "" }
-- No infite loop... but garbage in, garbage out...
ShowSplit("", ',')
--> { [1] = "" }
ShowSplit("abc", ',')
--> { [1] = "abc" }
ShowSplit("a,b,c", ',')
--> { [1] = "a", [2] = "b", [3] = "c" }
ShowSplit("a,b,c,", ',')
--> { [1] = "a", [2] = "b", [3] = "c", [4] = "" }
ShowSplit(",a,b,c,", ',')
--> { [1] = "", [2] = "a", [3] = "b", [4] = "c", [5] = "" }
ShowSplit("x,,,y", ',')
--> { [1] = "x", [2] = "", [3] = "", [4] = "y" }
ShowSplit(",,,", ',')
--> { [1] = "", [2] = "", [3] = "", [4] = "" }
ShowSplit("x!yy!zzz!@", '!', 4)
--> { [1] = "x", [2] = "yy", [3] = "zzz", [4] = "@" }
ShowSplit("x!yy!zzz!@", '!', 3)
--> { [1] = "x", [2] = "yy", [3] = "zzz" }
ShowSplit("x!yy!zzz!@", '!', 1)
--> { [1] = "x" }

ShowSplit("a:b:i:p:u:random:garbage", ":", 5)
--> { [1] = "a", [2] = "b", [3] = "i", [4] = "p", [5] = "u" }
ShowSplit("hr , br ;  p ,span, div", '%s*[;,]%s*')
--> { [1] = "hr", [2] = "br", [3] = "p", [4] = "span", [5] = "div" }

(PhilippeLhoste)


Function: Perl-like split/join

Many people miss Perl-like split/join functions in Lua. Here are mine:

-- Concat the contents of the parameter list,
-- separated by the string delimiter (just like in perl)
-- example: strjoin(", ", {"Anna", "Bob", "Charlie", "Dolores"})
function strjoin(delimiter, list)
  local len = getn(list)
  if len == 0 then 
    return "" 
  end
  local string = list[1]
  for i = 2, len do 
    string = string .. delimiter .. list[i] 
  end
  return string
end

-- Split text into a list consisting of the strings in text,
-- separated by strings matching delimiter (which may be a pattern). 
-- example: strsplit(",%s*", "Anna, Bob, Charlie,Dolores")
function strsplit(delimiter, text)
  local list = {}
  local pos = 1
  if strfind("", delimiter, 1) then -- this would result in endless loops
    error("delimiter matches empty string!")
  end
  while 1 do
    local first, last = strfind(text, delimiter, pos)
    if first then -- found?
      tinsert(list, strsub(text, pos, first-1))
      pos = last+1
    else
      tinsert(list, strsub(text, pos))
      break
    end
  end
  return list
end

(PeterPrade)


Function: Perl-like split/join, alternative

Here's my own split function, for comparison. It's largely the same as the above; not quite as DRY but (IMO) slightly cleaner. It doesn't use gfind (as suggested below) because I wanted to be able to specify a pattern for the split string, not a pattern for the data sections. If speed is paramount, it might be made faster by caching string.find as a local 'strfind' variable, as the above does.

--Written for 5.0; could be made slightly cleaner with 5.1
--Splits a string based on a separator string or pattern;
--returns an array of pieces of the string.
--(May optionally supply a table as the third parameter which will be filled 
with the results.)
function string:split( inSplitPattern, outResults )
  if not outResults then
    outResults = { }
  end
  local theStart = 1
  local theSplitStart, theSplitEnd = string.find( self, inSplitPattern, 
theStart )
  while theSplitStart do
    table.insert( outResults, string.sub( self, theStart, theSplitStart-1 ) )
    theStart = theSplitEnd + 1
    theSplitStart, theSplitEnd = string.find( self, inSplitPattern, theStart )
  end
  table.insert( outResults, string.sub( self, theStart ) )
  return outResults
end

(GavinKistner)


Function: PHP-like explode

Explode string into table with seperator (moved from TableUtils):

-- explode(seperator, string)
function explode(d,p)
  local t, ll
  t={}
  ll=0
  if(#p == 1) then return {p} end
    while true do
      l=string.find(p,d,ll,true) -- find the next d in the string
      if l~=nil then -- if "not not" found then..
        table.insert(t, string.sub(p,ll,l-1)) -- Save it in our array.
        ll=l+1 -- save just after where we found it for searching next time.
      else
        table.insert(t, string.sub(p,ll)) -- Save what's left in our array.
        break -- Break at end, as it should be, according to the lua manual.
      end
    end
  return t
end

Here's my version of PHP style explode, supporting limit

function explode(sep, str, limit)
    if not sep or sep == "" then return false end
    if not str then return false end
    limit = limit or mhuge
    if limit == 0 or limit == 1 then return {str},1 end

    local r = {}
    local n, init = 0, 1

    while true do
        local s,e = strfind(str, sep, init, true)
        if not s then break end
        r[#r+1] = strsub(str, init, s - 1)
        init = e + 1
        n = n + 1
        if n == limit - 1 then break end
    end

    if init <= strlen(str) then
        r[#r+1] = strsub(str, init)
    else
        r[#r+1] = ""
    end
    n = n + 1

    if limit < 0 then
        for i=n, n + limit + 1, -1 do r[i] = nil end
        n = n + limit
    end

    return r, n
end
(Lance Li)


Function: Using metatable and __index

This function uses a metatable's __index function to populate the table of split parts. This function does not try to (correctly) invert the pattern, and so really doesn't work as most string split functions do.

--[[ written for Lua 5.1
split a string by a pattern, take care to create the "inverse" pattern 
yourself. default pattern splits by white space.
]]
string.split = function(str, pattern)
  pattern = pattern or "[^%s]+"
  if pattern:len() == 0 then pattern = "[^%s]+" end
  local parts = {__index = table.insert}
  setmetatable(parts, parts)
  str:gsub(pattern, parts)
  setmetatable(parts, nil)
  parts.__index = nil
  return parts
end
-- example 1
str = "no separators in this string"
parts = str:split( "[^,]+" )
print( # parts )
table.foreach(parts, print)
--[[ output:
1
1	no separators in this string
]]

-- example 2
str = "   split, comma, separated  , , string   "
parts = str:split( "[^,%s]+" )
print( # parts )
table.foreach(parts, print)
--[[ output:
4
1	split
2	comma
3	separated
4	string
]]


Function: true Python semantics for split

This is the Python behavior:

Python 2.5.1 (r251:54863, Jun 15 2008, 18:24:51) 
[GCC 4.3.0 20080428 (Red Hat 4.3.0-8)] on linux2
>>> 'x!yy!zzz!@'.split('!')
['x', 'yy', 'zzz', '@']
>>> 'x!yy!zzz!@'.split('!', 3)
['x', 'yy', 'zzz', '@']
>>> 'x!yy!zzz!@'.split('!', 2)
['x', 'yy', 'zzz!@']
>>> 'x!yy!zzz!@'.split('!', 1)
['x', 'yy!zzz!@']

And IMHO this Lua function implements this semantics:

function string:split(sSeparator, nMax, bRegexp)
	assert(sSeparator ~= '')
	assert(nMax == nil or nMax >= 1)

	local aRecord = {}

	if self:len() > 0 then
		local bPlain = not bRegexp
		nMax = nMax or -1

		local nField=1 nStart=1
		local nFirst,nLast = self:find(sSeparator, nStart, bPlain)
		while nFirst and nMax ~= 0 do
			aRecord[nField] = self:sub(nStart, nFirst-1)
			nField = nField+1
			nStart = nLast+1
			nFirst,nLast = self:find(sSeparator, nStart, bPlain)
			nMax = nMax-1
		end
		aRecord[nField] = self:sub(nStart)
	end

	return aRecord
end

Observe the possibility to use simple strings or regular expressions as delimiters.

Test Cases:

Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio
...
> for k,v in next, string.split('x!yy!zzz!@', '!') do print(v) end
x
yy
zzz
@
> for k,v in next, string.split('x!yy!zzz!@', '!', 3) do print(v) end
x
yy
zzz
@
> for k,v in next, string.split('x!yy!zzz!@', '!', 2) do print(v) end
x
yy
zzz!@
> for k,v in next, string.split('x!yy!zzz!@', '!', 1) do print(v) end
x
yy!zzz!@

(JoanOrdinas)


If we define split simply as "return all 0-n character occurences followed by a separator, plus what's left of the string", which I think results in the most intuitive splitting logic, then we get a simple implementation using just gmatch which covers all cases and still allows the separator to be a pattern:

function gsplit(s,sep)
	return coroutine.wrap(function()
		if s == '' or sep == '' then coroutine.yield(s) return end
		local lasti = 1
		for v,i in s:gmatch('(.-)'..sep..'()') do
		   coroutine.yield(v)
		   lasti = i
		end
		coroutine.yield(s:sub(lasti))
	end)
end

--same idea without coroutines

function gsplit2(s,sep)
	local lasti, done, g = 1, false, s:gmatch('(.-)'..sep..'()')
	return function()
		if done then return end
		local v,i = g()
		if s == '' or sep == '' then done = true return s end
		if v == nil then done = true return s:sub(lasti) end
		lasti = i
		return v
	end
end

The gsplit() above returns an iterator, so other API variants can be easily derived from it:

{{{!Lua
function iunpack(i,s,v1)
   local function pass(...)
	  local v1 = i(s,v1)
	  if v1 == nil then return ... end
	  return v1, pass(...)
   end
   return pass()
end

function split(s,sep)
   return iunpack(gsplit(s,sep))
end

function accumulate(t,i,s,v)
    for v in i,s,v do
        t[#t+1] = v
    end
    return t
end

function tsplit(s,sep)
   return accumulate({}, gsplit(s,sep))
end

Note that the above implementation does not allow captures in the separator. To allow for that, another closure must be created to pass along the additional captured strings (see VarargTheSecondClassCitizen). The semantics also get muddy (I suppose one use case would be wanting to know what the actual separator was for each string, eg. for a separator pattern like [%.,;]).

function gsplit(s,sep)
	local i, done, g = 1, false, s:gmatch('(.-)'..sep..'()')
	local function pass(...)
		if ... == nil then done = true return s:sub(i) end
		i = select(select('#',...),...)
		return ...
	end
	return function()
		if done then return end
		if s == '' or sep == '' then done = true return s end
		return pass(g())
	end
end

The problem with the above implementation is that however easy to read, the (.-) pattern in Lua has awful performance, hence the following implementation based on only string.find (allows for captures in the separator and adds a third argument "plain", similar to string.find):

function string.gsplit(s, sep, plain)
	local start = 1
	local done = false
	local function pass(i, j, ...)
		if i then
			local seg = s:sub(start, i - 1)
			start = j + 1
			return seg, ...
		else
			done = true
			return s:sub(start)
		end
	end
	return function()
		if done then return end
		if sep == '' then done = true return s end
		return pass(s:find(sep, start, plain))
	end
end

Unit testing:

local function test(s,sep,expect)
	local t={} for c in s:gsplit(sep) do table.insert(t,c) end
	assert(#t == #expect)
	for i=1,#t do assert(t[i] == expect[i]) end
	test(t, expect)
end
test('','',{''})
test('','asdf',{''})
test('asdf','',{'asdf'})
test('', ',', {''})
test(',', ',', {'',''})
test('a', ',', {'a'})
test('a,b', ',', {'a','b'})
test('a,b,', ',', {'a','b',''})
test(',a,b', ',', {'','a','b'})
test(',a,b,', ',', {'','a','b',''})
test(',a,,b,', ',', {'','a','','b',''})
test('a,,b', ',', {'a','','b'})
test('asd  ,   fgh  ,;  qwe, rty.   ,jkl', '%s*[,.;]%s*', {'asd','fgh','','qwe','rty','','jkl'})
test('Spam eggs spam spam and ham', 'spam', {'Spam eggs ',' ',' and ham'})

(CosminApreutesei)



-- single char string splitter, sep *must* be a single char pattern
-- *probably* escaped with % if it has any special pattern meaning, eg "%." not "."
-- so good for splitting paths on "/" or "%." which is a common need

local function csplit(str,sep)
        local ret={}
        local n=1
        for w in str:gmatch("([^"..sep.."]*)") do
                        ret[n]=ret[n] or w -- only set once (so the blank after a string is ignored)
                        if w=="" then n=n+1 end -- step forwards on a blank but not a string
        end
        return ret
end

-- the following is true of any string, csplit will do the reverse of a concat

local str=""
print(str , assert( table.concat( csplit(str,"/") , "/" ) == str ) )

local str="only"
print(str , assert( table.concat( csplit(str,"/") , "/" ) == str ) )

local str="/test//ok/"
print(str , assert( table.concat( csplit(str,"/") , "/" ) == str ) )

local str=".test..ok."
print(str , assert( table.concat( csplit(str,"%.") , "." ) == str ) )


Comparison to Other Languages

User Comments

I mean no disrespect, of course, but.. does anyone actually have a working split function without glitches like infinite loops, wrong matches, or error cases? Are all those "takes" of any help here? -- CosminApreutesei

Try Rici Lake's split function: LuaList:2006-12/msg00414.html -- Jörg Richter

That version fails again when the pattern is an empty string. Specifications for split functions in other languages define how these corner cases should behave (see "Comparison to Other Languages" above). --David Manura

See Also


RecentChanges · preferences
edit · history
Last edited May 11, 2013 8:10 pm GMT (diff)