Date: Fri, 12 Jan 2001 19:15:04 +0200 From: Maxim Sobolev <sobomax@FreeBSD.org> To: ports@FreeBSD.org Subject: RFC: tool to automate common operations with patchfiles Message-ID: <3A5F3B98.954D80A5@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format.
--------------0CE4933781104F1CFCC44A6F
Content-Type: text/plain; charset=koi8-r
Content-Transfer-Encoding: 7bit
Hi folks,
With this message I'm attaching a new tool aimed to significantly simplify
common operations with ports' patches. I modelled it based on my own experience
with the subject, so I hope it would be useful for all hardcore porters and
would like to hear your comments/suggestions/patches/flames etc.
The tool has the following two modes of operation:
1. Generate a patchfile(a) from the user supplied file(s).
----------------------------------------------------------
Following example session should give you an idea on how it operates:
$ pwd
/somewhere/myport/work/myport-1.0/src/lib/bar
$ cp foo.c foo.c.orig
$ vi foo.c
[..] <-make your changes
$ patchtool foo.c | more
[..] <-read the diff
$ patchtool -a foo.c
Generating patchfile: patch-src_lib_bar_foo.c <- now you have patchfile ready
in /somewhere/myport/files
[..]
The tool tries hard to guess correct path of the file relatively to WRKSRC, so
you current directory does not matter and patch should be generated properly
wherever you are, so example, the following will work all the same:
$ cd /to/the/hell
$ patchtool -a /somewhere/myport/work/myport-1.0/src/lib/bar/foo.c
Generating patchfile: patch-src_lib_bar_foo.c...
$ patchtool -a ../../../somewhere/myport/work/myport-1.0/src/lib/bar/foo.c
Generating patchfile: patch-src_lib_bar_foo.c...
Several notes:
- `-a' option instructs the tool to automatically save resulting diff into
appropriate patchfile instead of dumping it to the stdout;
- the tool is smart enough to check your existing patches before generating
patchfile, so for example if in the previous case you already had a patch for
src/lib/bar/foo.c called patch-xxx, then it would update that existing
patchfile instead of creating a new one;
- the tool honours WRKDIRPREFIX;
- you can supply arbitrary number of files to generate patches from, you can
even mix files from the different ports - the resulting patchfiles will be
placed into the appropriate PATCHDIRs properly, for example:
$ patchtool -a port1/work/port1-1.0/1.c port2/work/port2-1.0/2.c
Generating patchfile: patch-1.c...
Generating patchfile: patch-2.c...
will generate port1/files/patch-1.c and port2/files/patch-2.c.
2. Update pointed by the user patchfile(s).
-------------------------------------------
Another mode of operation is updating of already existing patchfiles. In this
mode the tool updates all patchfiles specified in the command line, for
example:
$ patchtool -u files/patch-aa files/patch-be
Updating patchfile: patch-aa...
Updating patchfile: patch-be...
Alternatively you can specify any directory within a tree of a particular port,
in that case all patches that belong to this port will be updated, for example:
$ ls /somewhere/myport/files
patch-1 patch-8
$ patchtool -u /somewhere/myport
Updating patchfile: patch-1...
Updating patchfile: patch-8...
$ patchtool -u /somewhere/myport/files
Updating patchfile: patch-1...
Updating patchfile: patch-8...
$ cd /somewhere/myport/work
$ patchtool -u ./
Updating patchfile: patch-1...
Updating patchfile: patch-8...
[etc]
Final notes:
- you can tweak many aspects of behaviour, e.g. default diff flags (/me hands
will Will PT_DIFF_ARGS=-u0), patchfile prefix, suffix for saved version of
files etc., by setting appropriate environment variable. See comments at the
very beginning of the file;
- error handling may not be ideal at the moment - please report any problems
you may have with it;
- the tool currently doesn't try to make sure that the patch is in fact needs
to be updated, so use -u for existing ports with prejudice (it may generate
unnecessary repobloat);
- sourcecode is best viewed with tabsize=4.
Regards,
Maxim
P.S. Any flames/holywars regarding Python vs. Perl/Ruby/whatever will be
silently dumped to /dev/null! :-)
--------------0CE4933781104F1CFCC44A6F
Content-Type: text/plain; charset=koi8-r;
name="patchtool"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="patchtool"
#!/usr/local/bin/python
#
# ----------------------------------------------------------------------------
# "THE BEER-WARE LICENSE" (Revision 42, (c) Poul-Henning Kamp):
# Maxim Sobolev <sobomax@FreeBSD.org> wrote this file. As long as you retain
# this notice you can do whatever you want with this stuff. If we meet some
# day, and you think this stuff is worth it, you can buy me a beer in return.
#
# Maxim Sobolev
# ----------------------------------------------------------------------------
#
# $FreeBSD$
#
# MAINTAINER= sobomax@FreeBSD.org
#
import os, os.path, popen2, sys, getopt, glob, errno, types
# Some global variables used as constants
True = 1
False = 0
Error = -1
# Tweakable global variables. User is able to override any of these by setting
# appropriate environment variable prefixed by `PT_', eg:
# $ export PT_CVS_ID="FooOS"
# $ export PT_DIFF_CMD="/usr/local/bin/mydiff"
# will force script to use "FooOS" as a CVS_ID instead of "FreeBSD" and
# "/usr/local/bin/mydiff" as a command to generate diffs.
class Vars:
CVS_ID = 'FreeBSD'
DIFF_ARGS = '-du'
DIFF_SUFX = '.orig'
PATCH_PREFIX = 'patch-'
RCSDIFF_SUFX = ',v'
CD_CMD = 'cd'
DIFF_CMD = '/usr/bin/diff'
MAKE_CMD = '/usr/bin/make'
PRINTF_CMD = '/usr/bin/printf'
RCSDIFF_CMD = '/usr/bin/rcsdiff'
DEFAULT_MAKEFILE = 'Makefile'
DEV_NULL = '/dev/null'
ETC_MAKE_CONF = '/etc/make.conf'
#
# Check if the supplied patch refers to a port's directory.
#
def isportdir(path, soft = False):
REQ_FILES = ('Makefile', 'pkg-comment', 'pkg-descr', 'pkg-plist', \
'distinfo')
if not os.path.isdir(path) and soft != True:
raise IOError(errno.ENOENT, path)
# Not reached #
try:
content = os.listdir(path)
except OSError:
return False
for file in REQ_FILES:
if file not in content:
return False
return True
#
# Traverse directory tree up from the path pointed by argument and return if
# root directory of a port is found.
#
def locateportdir(path, wrkdirprefix= ''):
# Flag to relax error checking in isportdir() function. It required when
# WRKDIRPREFIX is defined.
softisport = False
path = os.path.abspath(path)
if wrkdirprefix != '':
wrkdirprefix= os.path.abspath(wrkdirprefix)
commonprefix = os.path.commonprefix((path, wrkdirprefix))
if commonprefix != wrkdirprefix:
return ''
path = path[len(wrkdirprefix):]
softisport = True
while path != '/':
if isportdir(path, softisport) == True:
return path
path = os.path.abspath(os.path.join(path, '..'))
return ''
#
# Get value of a make(1) variable called varname. Optionally maintain a cache
# for resolved varname:makepath pairs to speed-up operation if the same variable
# from the exactly same file is requiested repeatedly (invocation of make(1) is
# very expensive operation...)
#
def querymakevar(varname, path = 'Makefile', cache = {}):
path = os.path.abspath(path)
if cache.has_key((varname, path)) == 1:
return cache[(varname, path)]
origpath = path
if os.path.isdir(path):
path = os.path.join(path, Vars.DEFAULT_MAKEFILE)
if not os.path.isfile(path):
raise IOError(errno.ENOENT, path)
# Not reached #
dir = os.path.dirname(path)
CMDLINE = '%s %s && %s -f %s -V %s' % (Vars.CD_CMD, dir, Vars.MAKE_CMD, \
path, varname)
pipe = popen2.popen3(CMDLINE)
retval = ''
for line in pipe[0].readlines():
retval = retval + line.strip() + ' '
for fd in pipe:
fd.close()
retval = retval[:-1]
cache[(varname, origpath)] = retval
return retval
#
# Get a path of `path' relatively to wrksrc. For example:
# path: /foo/bar
# wrksrc: /foo/bar/baz/somefile.c
# getrelpath: baz/somefile.c
# Most of the code here is to handle cases when ../ operation is required to
# reach wrksrc from path, for example:
# path: /foo/bar
# wrksrc: /foo/baz/somefile.c
# getrelpath: ../baz/somefile.c
#
def getrelpath(path, wrksrc):
path = os.path.abspath(path)
wrksrc = os.path.abspath(wrksrc)
commonpart = os.path.commonprefix((path, wrksrc))
path = path[len(commonpart):]
wrksrc = wrksrc[len(commonpart):]
if wrksrc == '':
path = path[1:]
adjust = ''
while os.path.normpath(os.path.join(wrksrc, adjust)) != '.':
adjust = os.path.join(adjust, '..')
relpath = os.path.join(adjust, path)
return relpath
#
# Generare a diff between saved and current versions of the file pointed by the
# wrksrc+path. Apply heuristics to locate saved version of the file in question
# and if it fails assume that file is new, so /dev/null is to be used as
# original file. Optionally save generated patch into `outfile' instead of
# dumping it to stdout. Generated patches automatically being tagged with
# "FreeBSD" cvs id.
#
def gendiff(path, wrksrc, outfile = ''):
IDGEN_CMD = '%s "\n\$%s\$\n\n"' % (Vars.PRINTF_CMD, Vars.CVS_ID)
fullpath = os.path.join(wrksrc, path)
if not os.path.isfile(fullpath):
raise IOError(errno.ENOENT, fullpath)
# Not reached #
retval = Error
cmdline = ''
if os.path.isfile(fullpath + Vars.DIFF_SUFX): # Normal diff
cmdline = '%s %s %s%s %s' % (Vars.DIFF_CMD, Vars.DIFF_ARGS, path, \
Vars.DIFF_SUFX, path)
elif os.path.isfile(fullpath + Vars.RCSDIFF_SUFX): # RCS diff
cmdline = '%s %s %s' % (Vars.RCSDIFF_CMD, Vars.DIFF_ARGS, path)
else: # New file
cmdline = '%s %s %s %s' % (Vars.DIFF_CMD, Vars.DIFF_ARGS, \
Vars.DEV_NULL, path)
if cmdline != '':
if outfile != '':
cmdline = '( %s && %s ) 1>%s 2>%s' % (IDGEN_CMD, cmdline, outfile, \
Vars.DEV_NULL)
savedir = os.getcwd()
os.chdir(wrksrc)
exitstat = os.system(cmdline)
if os.WIFEXITED(exitstat):
exitval = os.WEXITSTATUS(exitstat)
if exitval == 0: # No differences were found
if outfile != '':
os.unlink(outfile)
retval = False
elif exitval == 1: # Some differences were found
retval = True
else: # Error occured
raise ECmdError('"%s"' % cmdline, \
'external command returned non-zero error code')
os.chdir(savedir)
return retval
#
# Automatically generate a name for a patch based on its path relative to
# wrksrc. Use simple scheme to ensute 1-to-1 mapping between path and
# patchname - replace all '_' with '__' and all '/' with '_'.
#
def makepatchname(path, patchdir = ''):
retval = Vars.PATCH_PREFIX + \
path.replace('_', '__').replace('/', '_')
retval = os.path.join(patchdir, retval)
return retval
#
# Write a specified message to stderr.
#
def write_msg(message):
if type(message) == types.StringType:
message = message,
sys.stderr.writelines(message)
#
# Print specified message to stdout and ask user [y/N]?. Optionally allow
# specify default answer, i.e. return value if user typed only <cr>
#
def query_yn(message, default = False):
while True:
if default == True:
yn = 'Y/n'
elif default == False:
yn = 'y/N'
else:
yn = 'Y/N'
reply = raw_input('%s [%s]: ' % (message, yn))
if reply == 'y' or reply == 'Y':
return True
elif reply == 'n' or reply == 'N':
return False
elif reply == '' and default in (True, False):
return default
print 'Wrong answer "%s", please try again' % reply
#
# Print optional message and usage information and exit with specified exit
# code.
#
def usage(code, msg = ''):
myname = os.path.basename(sys.argv[0])
write_msg((str(msg), """
Usage: %s [-ai] file ...
%s -u [ai] [patchfile|patchdir ...]
""" % (myname, myname)))
sys.exit(code)
#
# Simple custom exception
#
class MyError:
msg = 'error'
def __init__(self, file, msg=''):
self.file = file
if msg != '':
self.msg = msg
def __str__(self):
return '%s: %s' % (self.file, self.msg)
#
# Error parsing patchfile
#
class PatchError(MyError):
msg = 'corrupt patchfile'
#
# Error executing external command
#
class ECmdError(MyError):
pass
class Patch:
fullpath = ''
minus3file = ''
plus3file = ''
wrksrc = ''
def __init__(self, path, wrksrc):
MINUS3_DELIM = '--- '
PLUS3_DELIM = '+++ '
path = os.path.abspath(path)
if not os.path.isfile(path):
raise IOError(errno.ENOENT, path)
# Not reached #
self.fullpath = path
file = open(path)
for line in file.readlines():
if self.minus3file == '':
if line[:len(MINUS3_DELIM)] == MINUS3_DELIM:
lineparts = line.split()
try:
self.minus3file = lineparts[1]
except IndexError:
raise PatchError(path)
# Not reached #
continue
elif line[:len(PLUS3_DELIM)] == PLUS3_DELIM:
lineparts = line.split()
try:
self.plus3file = lineparts[1]
except IndexError:
raise PatchError(path)
# Not reached #
break
file.close()
if self.minus3file == '' or self.plus3file == '':
raise PatchError(path)
# Not reached #
self.wrksrc = os.path.abspath(wrksrc)
def update(self):
targetfile = os.path.join(self.wrksrc, self.plus3file)
if not os.path.isfile(targetfile):
raise IOError(errno.ENOENT, targetfile)
patchdir = os.path.dirname(self.fullpath)
if not os.path.isdir(patchdir):
os.mkdir(patchdir)
return gendiff(self.plus3file, self.wrksrc, self.fullpath)
class NewPatch(Patch):
def __init__(self, patchdir, wrksrc, relpath):
self.fullpath = makepatchname(relpath, os.path.abspath(patchdir))
self.wrksrc = os.path.abspath(wrksrc)
self.plus3file = relpath
self.minus3file = relpath
class PatchesCollection:
patches = {}
def __init__(self):
self.patches = {}
pass
def adddir(self, patchdir, wrksrc):
if not os.path.isdir(patchdir):
raise IOError(errno.ENOENT, patchdir)
# Not reached #
for file in glob.glob(os.path.join(patchdir, Vars.PATCH_PREFIX + '*')):
self.addpatchfile(file, wrksrc)
def addpatchfile(self, path, wrksrc):
path = os.path.abspath(path)
if not self.patches.has_key(path):
self.addpatchobj(Patch(path, wrksrc))
def addpatchobj(self, patchobj):
self.patches[patchobj.fullpath] = patchobj
def lookupbyname(self, path):
path = os.path.abspath(path)
if self.patches.has_key(path):
return self.patches[path]
return None
def lookupbytarget(self, wrksrc, relpath):
wrksrc = os.path.abspath(wrksrc)
for patch in self.patches.values():
if wrksrc == patch.wrksrc and relpath == patch.plus3file:
return patch
return None
def getpatchobjs(self):
return self.patches.values()
def main():
try:
opts, args = getopt.getopt(sys.argv[1:], 'afu')
except getopt.GetoptError, msg:
usage(2, msg)
automatic = False
force = False
mode = generate
for o, a in opts:
if o == '-a':
automatic = True
elif o == '-f':
force = True
elif o == '-u':
mode = update
else:
usage(2)
# Allow user to override internal constants
for varname in dir(Vars):
if varname[:2] == '__' and varname[-2:] == '__':
continue
try:
value = os.environ['PT_' + varname]
setattr(Vars, varname, value)
except KeyError:
pass
mode(args, automatic, force)
sys.exit(0)
def generate(args, automatic, force):
if len(args) == 0:
usage(2, "ERROR: no input files specified")
patches = PatchesCollection()
for filepath in args:
if not os.path.isfile(filepath):
raise IOError(errno.ENOENT, filepath)
# Not reached #
wrkdirprefix = querymakevar('WRKDIRPREFIX', Vars.ETC_MAKE_CONF)
portdir = locateportdir(os.path.dirname(filepath), wrkdirprefix)
if portdir == '':
sys.exit("ERROR: can't locate portdir")
# Not reached #
wrksrc = querymakevar('WRKSRC', portdir)
if wrksrc == '':
sys.exit("ERROR: can't get WRKSRC value")
# Not reached #
relpath = getrelpath(filepath, wrksrc)
if automatic == True:
patchdir = querymakevar('PATCHDIR', portdir)
if patchdir == '':
sys.exit("ERROR: can't get PATCHDIR value")
# Not reached #
if os.path.isdir(patchdir):
patches.adddir(patchdir, wrksrc)
patchobj = patches.lookupbytarget(wrksrc, relpath)
if patchobj == None:
patchobj = NewPatch(patchdir, wrksrc, relpath)
patches.addpatchobj(patchobj)
write_msg('Generating patchfile: %s...\n' % \
os.path.basename(patchobj.fullpath))
if not force and os.path.exists(patchobj.fullpath) and \
os.path.getsize(patchobj.fullpath) > 0:
try:
retval = query_yn('Target patchfile "%s" already ' \
'exists, do you want to replace it?' % \
os.path.basename(patchobj.fullpath))
except KeyboardInterrupt:
sys.exit('\nAction aborted')
# Not reached #
if retval == False:
continue
retval = patchobj.update()
else: # automatic != True
retval = gendiff(relpath, wrksrc)
if retval == Error:
sys.exit("ERROR: can't generate diff for \"%s\"" % filepath)
elif retval == False:
write_mss('WARNING: no differencies found between original and '
'current version of "%s"\n' % filepath)
def update(args, automatic, force):
if len(args) == 0:
args = './',
for path in args:
if not os.path.exists(path):
raise IOError(errno.ENOENT, path)
# Not reached #
patches = PatchesCollection()
if os.path.isdir(path):
for wrkdirprefix in (querymakevar('WRKDIRPREFIX', \
Vars.ETC_MAKE_CONF), ''):
portdir = locateportdir(path, wrkdirprefix)
if portdir != '':
break
if portdir == '':
sys.exit("ERROR: can't locate portdir")
# Not reached #
wrksrc = querymakevar('WRKSRC', portdir)
if wrksrc == '':
sys.exit("ERROR: can't get WRKSRC value")
# Not reached #
patchdir = querymakevar('PATCHDIR', portdir)
if patchdir == '':
sys.exit("ERROR: can't get PATCHDIR value")
# Not reached #
if os.path.isdir(patchdir):
patches.adddir(patchdir, wrksrc)
else:
continue
elif os.path.isfile(path):
patches.addpatchfile(path)
for patchobj in patches.getpatchobjs():
write_msg('Updating patchfile: %s...\n' % \
patchobj.fullpath)
patchobj.update()
if __name__ == '__main__':
try:
main()
except (PatchError, ECmdError), msg:
sys.exit('ERROR: ' + str(msg))
except IOError, (code, msg):
sys.exit('ERROR: %s: %s' % (str(msg), os.strerror(code)))
--------------0CE4933781104F1CFCC44A6F--
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-ports" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3A5F3B98.954D80A5>