Note that Lua patterns do not support alternatives with '|', but they support '?' for 0 or 1 occurrence (not just '*' and '+' for 0+ and 1+ occurences).
Note also that you want to match the dot for the extension literally, not arbitrary characters, so you must use '%.', not '.'. The '#' does not need to be escaped by a '%' in the search pattern (however it still works if you do).
Just create a capture group for the optional anchor (*including* the leading '#').
function Link (link)
link.target = link.target:gsub('(.+)%.md(#.+)?', '%1.html%2')
return link
end