In your constructor, if you don't pass in a o table, it makes an empty table.
You're then doing an operation o.r = o.r % 255 which is equivalent to o.r = nil % 255
The default values of r, g, and b are set on the first line in the Color table which means to access them you'd need to this
function Color:new(o)
local o = o or {}
setmetatable(o, self)
self.__index = self
o.r = o.r % 255
o.g = o.g % 255
o.b = o.b % 255
return o
end
or
function Color:new(o)
local o = o or {}
o.r = self.r % 255
o.g = self.g % 255
o.b = self.b % 255
setmetatable(o, self)
self.__index = self
return o
end