Date: Wed, 4 Jul 2012 12:50:20 GMT From: Volodymyr Kostyrko <c.kworr@gmail.com> To: freebsd-gnats-submit@FreeBSD.org Subject: ports/169643: New port: databases/py-mysql2pgsql [redports] Message-ID: <201207041250.q64CoKLD085709@red.freebsd.org> Resent-Message-ID: <201207041300.q64D0LTp053174@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 169643 >Category: ports >Synopsis: New port: databases/py-mysql2pgsql [redports] >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-ports-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: maintainer-update >Submitter-Id: current-users >Arrival-Date: Wed Jul 04 13:00:21 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Volodymyr Kostyrko >Release: RELENG_9 >Organization: None >Environment: FreeBSD green.tandem.local 9.0-STABLE FreeBSD 9.0-STABLE #0 r238050M: Tue Jul 3 11:58:06 EEST 2012 arcade@green.tandem.local:/usr/obj/usr/src/sys/MINIMAL amd64 >Description: Tool for migrating/converting from mysql to postgresql RedPorts: http://redports.org/buildarchive/20120704124110-14810/ >How-To-Repeat: >Fix: Patch attached with submission follows: # This is a shell archive. Save it in a file, remove anything before # this line, and then unpack it by entering "sh file". Note, it may # create directories; files and directories will be owned by you and # have default permissions. # # This archive contains: # # py-mysql2pgsql # py-mysql2pgsql/pkg-plist # py-mysql2pgsql/distinfo # py-mysql2pgsql/Makefile # py-mysql2pgsql/files # py-mysql2pgsql/files/extra-patch # py-mysql2pgsql/pkg-descr # echo c - py-mysql2pgsql mkdir -p py-mysql2pgsql > /dev/null 2>&1 echo x - py-mysql2pgsql/pkg-plist sed 's/^X//' >py-mysql2pgsql/pkg-plist << '48968953d7ef2d4a4eb975f071270e24' Xbin/py-mysql2pgsql X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/EGG-INFO/PKG-INFO X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/EGG-INFO/SOURCES.txt X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/EGG-INFO/dependency_links.txt X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/EGG-INFO/not-zip-safe X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/EGG-INFO/requires.txt X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/EGG-INFO/scripts/py-mysql2pgsql X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/EGG-INFO/top_level.txt X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/__init__.py X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/__init__.pyc X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/__init__.pyo X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/__init__.py X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/__init__.pyc X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/__init__.pyo X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/config.py X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/config.pyc X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/config.pyo X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/converter.py X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/converter.pyc X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/converter.pyo X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/errors.py X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/errors.pyc X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/errors.pyo X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/mysql_reader.py X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/mysql_reader.pyc X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/mysql_reader.pyo X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/postgres_db_writer.py X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/postgres_db_writer.pyc X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/postgres_db_writer.pyo X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/postgres_file_writer.py X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/postgres_file_writer.pyc X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/postgres_file_writer.pyo X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/postgres_writer.py X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/postgres_writer.pyc X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/postgres_writer.pyo X%%EXTRA%%%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/writer.py X%%EXTRA%%%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/writer.pyc X%%EXTRA%%%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib/writer.pyo X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/mysql2pgsql.py X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/mysql2pgsql.pyc X%%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/mysql2pgsql.pyo X@dirrm %%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/EGG-INFO/scripts X@dirrm %%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/EGG-INFO X@dirrm %%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql/lib X@dirrm %%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%%/mysql2pgsql X@dirrm %%PYTHON_SITELIBDIR%%/%%PYEASYINSTALL_EGG%% 48968953d7ef2d4a4eb975f071270e24 echo x - py-mysql2pgsql/distinfo sed 's/^X//' >py-mysql2pgsql/distinfo << '8fb20bf151eb4573a259e9807f986359' XSHA256 (postgresql/mysql2pgsql/v0.1.2) = 76354d3533adb70757cd1861f7fb68264e0121cd47b32d6050702bb0951b181f XSIZE (postgresql/mysql2pgsql/v0.1.2) = 105971 8fb20bf151eb4573a259e9807f986359 echo x - py-mysql2pgsql/Makefile sed 's/^X//' >py-mysql2pgsql/Makefile << '1d275f4ae6d6996bd614d0f92514c394' X# New ports collection makefile for: py-mysql2pgsql X# Date created: 29 May 2011 X# Whom: Volodymyr Kostyrko <c.kworr@gmail.com> X# vim:ts=8 X# X# $FreeBSD$ X# X XPORTNAME= mysql2pgsql XPORTVERSION= 0.1.2 XCATEGORIES= databases python XMASTER_SITES= https://nodeload.github.com/${GITHUB_USER}/py-${PORTNAME}/tarball/ XPKGNAMEPREFIX= ${PYTHON_PKGNAMEPREFIX} XPKGNAMESUFFIX?= ${EXTRA_SUFFIX} XDISTNAME= v${PORTVERSION} XEXTRACT_SUFX= XDIST_SUBDIR= postgresql/${PORTNAME} X XMAINTAINER= c.kworr@gmail.com XCOMMENT= Tool for migrating/converting from mysql to postgresql X XLICENSE= MIT # ?dunno, looks like MIT XLICENSE_FILE= ${WRKSRC}/LICENSE X XRUN_DEPENDS= ${PYTHON_PKGNAMEPREFIX}MySQLdb>0:${PORTSDIR}/databases/py-MySQLdb \ X ${PYTHON_PKGNAMEPREFIX}psycopg2>0:${PORTSDIR}/databases/py-psycopg2 \ X ${PYTHON_PKGNAMEPREFIX}termcolor>0:${PORTSDIR}/devel/py-termcolor \ X ${PYTHON_PKGNAMEPREFIX}yaml>0:${PORTSDIR}/devel/py-yaml X XUSE_PYTHON= yes XUSE_PYDISTUTILS= easy_install XPYDISTUTILS_PKGNAME= py_${PORTNAME} XGITHUB_USER= philipsoutham XGITHUB_HASH= 74de1e0 XWRKSRC= ${WRKDIR}/${GITHUB_USER}-py-${PORTNAME}-${GITHUB_HASH} X XOPTIONS_DEFINE= EXTRA XEXTRA_DESC= Extra patches from github repo not tagged separately X# Actually includes a lot of my own patches, I'm trying to stuff it to the master branch X X.include <bsd.port.pre.mk> X X.if ${PORT_OPTIONS:MEXTRA} XPLIST_SUB+= EXTRA="@comment " XEXTRA_PATCHES+= ${FILESDIR}/extra-patch XEXTRA_SUFFIX?= +extra X.else XPLIST_SUB+= EXTRA="@comment " X.endif X X.include <bsd.port.post.mk> 1d275f4ae6d6996bd614d0f92514c394 echo c - py-mysql2pgsql/files mkdir -p py-mysql2pgsql/files > /dev/null 2>&1 echo x - py-mysql2pgsql/files/extra-patch sed 's/^X//' >py-mysql2pgsql/files/extra-patch << '0c3925eddecf298c646bd0c56b7311c4' X--- .travis.yml Thu Jan 01 00:00:00 1970 +0000 X+++ .travis.yml Wed Jul 04 12:03:14 2012 +0300 X@@ -0,0 +1,21 @@ X+language: python X+python: X+ - "2.6" X+ - "2.7" X+install: X+ - pip install . --use-mirrors X+ - pip install -r requirements.txt --use-mirrors X+script: nosetests X+before_script: X+ - mysql -e 'create database mysql2pgsql;' X+ - psql -c 'create database mysql2pgsql;' -U postgres X+env: X+ - DB=mysql X+ - DB=postgres X+branches: X+ only: X+ - master X+ - develop X+notifications: X+ email: X+ - philipsoutham@gmail.com X--- bin/py-mysql2pgsql Thu Aug 18 13:23:12 2011 -0700 X+++ bin/py-mysql2pgsql Wed Jul 04 12:03:14 2012 +0300 X@@ -1,54 +1,33 @@ X #! /usr/bin/env python X import os X import sys X+import argparse X import mysql2pgsql X from mysql2pgsql.lib.errors import ConfigurationFileInitialized X X if __name__ == '__main__': X description = 'Tool for migrating/converting data from mysql to postgresql.' X epilog = 'https://github.com/philipsoutham/py-mysql2pgsql' X- try: X- import argparse X- parser = argparse.ArgumentParser( X- description=description, X- epilog=epilog) X- parser.add_argument( X- '-v', '--verbose', X- action='store_true', X- help='Show progress of data migration.' X- ) X- parser.add_argument( X- '-f', '--file', X- default='mysql2pgsql.yml', X- help='Location of configuration file (default: %(default)s). If none exists at that path, one will be created for you.', X- ) X- parser.add_argument( X- '-V', '--version', X- action='store_true', X- help='Print version and exit.' X- ) X- options = parser.parse_args() X- except ImportError: X- import optparse X- parser = optparse.OptionParser( X- description=description, X- epilog=epilog) X- parser.add_argument( X- '-v', '--verbose', X- action='store_true', X- help='Show progress of data migration.' X- ) X- parser.add_argument( X- '-f', '--file', X- default='mysql2pgsql.yml', X- help='Location of configuration file (default: %default). If none exists at that path, one will be created for you.', X- ) X- parser.add_argument( X- '-V', '--version', X- action='store_true', X- help='Print version and exit.' X- ) X- options, args = parser.parse_args() X+ X+ parser = argparse.ArgumentParser( X+ description=description, X+ epilog=epilog) X+ parser.add_argument( X+ '-v', '--verbose', X+ action='store_true', X+ help='Show progress of data migration.' X+ ) X+ parser.add_argument( X+ '-f', '--file', X+ default='mysql2pgsql.yml', X+ help='Location of configuration file (default: %(default)s). If none exists at that path, one will be created for you.', X+ ) X+ parser.add_argument( X+ '-V', '--version', X+ action='store_true', X+ help='Print version and exit.' X+ ) X+ options = parser.parse_args() X X if options.version: X # Someone wants to know the version, print and exit X--- mysql2pgsql/lib/__init__.py Thu Aug 18 13:23:12 2011 -0700 X+++ mysql2pgsql/lib/__init__.py Wed Jul 04 12:03:14 2012 +0300 X@@ -5,7 +5,7 @@ X X from .mysql_reader import MysqlReader X try: X- from termcolor import colored, cprint X+ from termcolor import cprint X except ImportError: X pass X X@@ -89,4 +89,3 @@ X else: X return f(*args, **kwargs) X return decorated_function X- X--- mysql2pgsql/lib/config.py Thu Aug 18 13:23:12 2011 -0700 X+++ mysql2pgsql/lib/config.py Wed Jul 04 12:03:14 2012 +0300 X@@ -2,7 +2,7 @@ X X import os.path X X-from yaml import load, dump X+from yaml import load X X try: X from yaml import CLoader as Loader, CDumper as Dumper X@@ -17,6 +17,7 @@ X def __init__(self, config_file_path): X self.options = load(open(config_file_path)) X X+ X class Config(ConfigBase): X def __init__(self, config_file_path, generate_if_not_found=True): X if not os.path.isfile(config_file_path): X@@ -34,7 +35,7 @@ X def reset_configfile(self, file_path): X with open(file_path, 'w') as f: X f.write(CONFIG_TEMPLATE) X- X+ X CONFIG_TEMPLATE = """ X # if a socket is specified we will use that X # if tcp is chosen you can use compression X--- mysql2pgsql/lib/converter.py Thu Aug 18 13:23:12 2011 -0700 X+++ mysql2pgsql/lib/converter.py Wed Jul 04 12:03:14 2012 +0300 X@@ -20,7 +20,9 @@ X print_start_table('>>>>>>>>>> STARTING <<<<<<<<<<\n\n') X X tables = [t for t in (t for t in self.reader.tables if t.name not in self.exclude_tables) if not self.only_tables or t.name in self.only_tables] X- X+ if self.only_tables: X+ tables.sort(key=lambda t: self.only_tables.index(t.name)) X+ X if not self.supress_ddl: X if self.verbose: X print_start_table('START CREATING TABLES') X@@ -57,12 +59,13 @@ X X for table in tables: X self.writer.write_indexes(table) X+ X+ for table in tables: X self.writer.write_constraints(table) X X if self.verbose: X print_start_table('DONE CREATING INDEXES AND CONSTRAINTS') X X- X if self.verbose: X print_start_table('\n\n>>>>>>>>>> FINISHED <<<<<<<<<<') X X--- mysql2pgsql/lib/mysql_reader.py Thu Aug 18 13:23:12 2011 -0700 X+++ mysql2pgsql/lib/mysql_reader.py Wed Jul 04 12:03:14 2012 +0300 X@@ -11,7 +11,8 @@ X re_column_precision = re.compile(r'\((\d+),(\d+)\)') X re_key_1 = re.compile(r'CONSTRAINT `(\w+)` FOREIGN KEY \(`(\w+)`\) REFERENCES `(\w+)` \(`(\w+)`\)') X re_key_2 = re.compile(r'KEY `(\w+)` \((.*)\)') X-re_key_3 = re.compile(r'PRIMARY KEY .*\((.*)\)') X+re_key_3 = re.compile(r'PRIMARY KEY \((.*)\)') X+ X X class DB: X """ X@@ -21,6 +22,7 @@ X helper functions. X """ X conn = None X+ X def __init__(self, options): X args = { X 'user': options.get('username', 'root'), X@@ -42,7 +44,7 @@ X self.options = args X X def connect(self): X- self.conn = MySQLdb.connect(**self.options) X+ self.conn = MySQLdb.connect(**self.options) X X def close(self): X self.conn.close() X@@ -86,37 +88,43 @@ X X def _convert_type(self, data_type): X """Normalize MySQL `data_type`""" X- if 'varchar' in data_type: X+ if data_type.startswith('varchar'): X return 'varchar' X- elif 'char' in data_type: X+ elif data_type.startswith('char'): X return 'char' X elif data_type in ('bit(1)', 'tinyint(1)', 'tinyint(1) unsigned'): X return 'boolean' X- elif re.search(r'smallint.* unsigned', data_type) or 'mediumint' in data_type: X+ elif re.search(r'^smallint.* unsigned', data_type) or data_type.startswith('mediumint'): X return 'integer' X- elif 'smallint' in data_type: X+ elif data_type.startswith('smallint'): X return 'tinyint' X- elif 'tinyint' in data_type or 'year(' in data_type: X+ elif data_type.startswith('tinyint') or data_type.startswith('year('): X return 'tinyint' X- elif 'bigint' in data_type and 'unsigned' in data_type: X+ elif data_type.startswith('bigint') and 'unsigned' in data_type: X return 'numeric' X- elif re.search(r'int.* unsigned', data_type) or\ X+ elif re.search(r'^int.* unsigned', data_type) or\ X ('bigint' in data_type and 'unsigned' not in data_type): X return 'bigint' X- elif 'int' in data_type: X+ elif data_type.startswith('int'): X return 'integer' X- elif 'float' in data_type: X+ elif data_type.startswith('float'): X return 'float' X- elif 'decimal' in data_type: X+ elif data_type.startswith('decimal'): X return 'decimal' X- elif 'double' in data_type: X+ elif data_type.startswith('double'): X return 'double precision' X else: X return data_type X X def _load_columns(self): X fields = [] X- for res in self.reader.db.query('EXPLAIN `%s`' % self.name): X+ for row in self.reader.db.query('EXPLAIN `%s`' % self.name): X+ res = () X+ for field in row: X+ if type(field) == unicode: X+ res += field.encode('utf8'), X+ else: X+ res += field, X length_match = re_column_length.search(res[1]) X precision_match = re_column_precision.search(res[1]) X length = length_match.group(1) if length_match else \ X@@ -138,7 +146,7 @@ X res = self.reader.db.query('SELECT MAX(`%s`) FROM `%s`;' % (field['name'], self.name), one=True) X field['maxval'] = int(res[0]) if res[0] else 0 X return fields X- X+ X def _load_indexes(self): X explain = self.reader.db.query('SHOW CREATE TABLE `%s`' % self.name, one=True) X explain = explain[1] X@@ -164,7 +172,7 @@ X match_data = re_key_3.search(line) X if match_data: X index['primary'] = True X- index['columns'] = [col.replace('`', '') for col in match_data.group(1).split(',')] X+ index['columns'] = [re.sub(r'\(\d+\)', '', col.replace('`', '')) for col in match_data.group(1).split(',')] X self._indexes.append(index) X continue X X@@ -189,7 +197,7 @@ X return 'SELECT %(column_names)s FROM `%(table_name)s`' % { X 'table_name': self.name, X 'column_names': ', '. join(("`%s`" % c['name']) for c in self.columns)} X- X+ X def __init__(self, options): X self.db = DB(options) X X--- mysql2pgsql/lib/postgres_db_writer.py Thu Aug 18 13:23:12 2011 -0700 X+++ mysql2pgsql/lib/postgres_db_writer.py Wed Jul 04 12:03:14 2012 +0300 X@@ -1,15 +1,14 @@ X from __future__ import with_statement, absolute_import X X-import sys X import time X from contextlib import closing X X import psycopg2 X-from psycopg2.extensions import QuotedString X X from . import print_row_progress, status_logger X from .postgres_writer import PostgresWriter X X+ X class PostgresDbWriter(PostgresWriter): X """Class used to stream DDL and/or data X from a MySQL server to a PostgreSQL. X@@ -53,7 +52,7 @@ X try: X return '%s\n' % ('\t'.join(row)) X except UnicodeDecodeError: X- return '%s\n' % ('\t'.join(row)).decode('utf-8') X+ return '%s\n' % ('\t'.join(r.decode('utf8') for r in row)) X finally: X if self.verbose: X if (self.idx % 20000) == 0: X@@ -69,8 +68,8 @@ X def read(self, *args, **kwargs): X return self.readline(*args, **kwargs) X X- X def __init__(self, db_options, verbose=False): X+ super(PostgresDbWriter, self).__init__() X self.verbose = verbose X self.db_options = { X 'host': db_options['hostname'], X@@ -80,7 +79,7 @@ X 'user': db_options['username'], X } X if ':' in db_options['database']: X- self.db_options['database'], self.schema = self.db_options['database'].split(':') X+ self.db_options['database'], self.schema = self.db_options['database'].split(':') X else: X self.schema = None X self.open() X@@ -115,7 +114,7 @@ X table=table_name, X columns=columns X ) X- X+ X self.conn.commit() X X def close(self): X@@ -129,13 +128,13 @@ X @status_logger X def truncate(self, table): X """Send DDL to truncate the specified `table` X- X+ X :Parameters: X - `table`: an instance of a :py:class:`mysql2pgsql.lib.mysql_reader.MysqlReader.Table` object that represents the table to read/write. X X Returns None X """ X- truncate_sql, serial_key_sql = super(self.__class__, self).truncate(table) X+ truncate_sql, serial_key_sql = super(PostgresDbWriter, self).truncate(table) X self.execute(truncate_sql) X if serial_key_sql: X self.execute(serial_key_sql) X@@ -149,10 +148,10 @@ X X Returns None X """ X- table_sql, serial_key_sql = super(self.__class__, self).write_table(table) X+ table_sql, serial_key_sql = super(PostgresDbWriter, self).write_table(table) X for sql in serial_key_sql + table_sql: X self.execute(sql) X- X+ X @status_logger X def write_indexes(self, table): X """Send DDL to create the specified `table` indexes X@@ -162,7 +161,7 @@ X X Returns None X """ X- index_sql = super(self.__class__, self).write_indexes(table) X+ index_sql = super(PostgresDbWriter, self).write_indexes(table) X for sql in index_sql: X self.execute(sql) X X@@ -175,7 +174,7 @@ X X Returns None X """ X- constraint_sql = super(self.__class__, self).write_constraints(table) X+ constraint_sql = super(PostgresDbWriter, self).write_constraints(table) X for sql in constraint_sql: X self.execute(sql) X X--- mysql2pgsql/lib/postgres_file_writer.py Thu Aug 18 13:23:12 2011 -0700 X+++ mysql2pgsql/lib/postgres_file_writer.py Wed Jul 04 12:03:14 2012 +0300 X@@ -1,16 +1,13 @@ X from __future__ import absolute_import X X import time X-import sys X X-from cStringIO import StringIO X- X-from psycopg2.extensions import QuotedString X X from .postgres_writer import PostgresWriter X X from . import print_row_progress, status_logger X X+ X class PostgresFileWriter(PostgresWriter): X """Class used to ouput the PostgreSQL X compatable DDL and/or data to the specified X@@ -19,10 +16,12 @@ X :Parameters: X - `output_file`: the output :py:obj:`file` to send the DDL and/or data X - `verbose`: whether or not to log progress to :py:obj:`stdout` X- X+ X """ X verbose = None X+ X def __init__(self, output_file, verbose=False): X+ super(PostgresFileWriter, self).__init__() X self.verbose = verbose X self.f = output_file X self.f.write(""" X@@ -42,7 +41,7 @@ X X Returns None X """ X- truncate_sql, serial_key_sql = super(self.__class__, self).truncate(table) X+ truncate_sql, serial_key_sql = super(PostgresFileWriter, self).truncate(table) X self.f.write(""" X -- TRUNCATE %(table_name)s; X %(truncate_sql)s X@@ -63,9 +62,9 @@ X X Returns None X """ X- table_sql, serial_key_sql = super(self.__class__, self).write_table(table) X+ table_sql, serial_key_sql = super(PostgresFileWriter, self).write_table(table) X if serial_key_sql: X- self.f.write(""" X+ self.f.write(""" X %(serial_key_sql)s X """ % { X 'serial_key_sql': '\n'.join(serial_key_sql) X@@ -88,7 +87,7 @@ X X Returns None X """ X- self.f.write('\n'.join(super(self.__class__, self).write_indexes(table))) X+ self.f.write('\n'.join(super(PostgresFileWriter, self).write_indexes(table))) X X @status_logger X def write_constraints(self, table): X@@ -99,7 +98,7 @@ X X Returns None X """ X- self.f.write('\n'.join(super(self.__class__, self).write_constraints(table))) X+ self.f.write('\n'.join(super(PostgresFileWriter, self).write_constraints(table))) X X @status_logger X def write_contents(self, table, reader): X@@ -119,7 +118,7 @@ X X f_write(""" X -- X--- Data for Name: %(table_name)s; Type: TABLE DATA; X+-- Data for Name: %(table_name)s; Type: TABLE DATA; X -- X X COPY "%(table_name)s" (%(column_names)s) FROM stdin; X@@ -131,22 +130,22 @@ X start_time = tt() X prev_val_len = 0 X prev_row_count = 0 X- for i, row in enumerate(reader.read(table)): X+ for i, row in enumerate(reader.read(table), 1): X row = list(row) X pr(table, row) X try: X- f_write('%s\n' % ('\t'.join(row))) X+ f_write(u'%s\n' % (u'\t'.join(row))) X except UnicodeDecodeError: X- f_write('%s\n' % ('\t'.join(row)).decode('utf-8')) X+ f_write(u'%s\n' % (u'\t'.join(r.decode('utf-8') for r in row))) X if verbose: X- if ((i + 1) % 20000) == 0: X+ if (i % 20000) == 0: X now = tt() X elapsed = now - start_time X- val = '%.2f rows/sec [%s] ' % (((i + 1) - prev_row_count) / elapsed, (i + 1)) X+ val = '%.2f rows/sec [%s] ' % ((i - prev_row_count) / elapsed, i) X print_row_progress('%s%s' % (("\b" * prev_val_len), val)) X prev_val_len = len(val) + 3 X start_time = now X- prev_row_count = i + 1 X+ prev_row_count = i X X f_write("\\.\n\n") X if verbose: X--- mysql2pgsql/lib/postgres_writer.py Thu Aug 18 13:23:12 2011 -0700 X+++ mysql2pgsql/lib/postgres_writer.py Wed Jul 04 12:03:14 2012 +0300 X@@ -6,18 +6,21 @@ X X from psycopg2.extensions import QuotedString, Binary, AsIs X X-from .writer import Writer X X- X-class PostgresWriter(Writer): X+class PostgresWriter(object): X """Base class for :py:class:`mysql2pgsql.lib.postgres_file_writer.PostgresFileWriter` X and :py:class:`mysql2pgsql.lib.postgres_db_writer.PostgresDbWriter`. X """ X+ def __init__(self): X+ self.column_types = {} X+ X def column_description(self, column): X return '"%s" %s' % (column['name'], self.column_type_info(column)) X X def column_type(self, column): X- return self.column_type_info(column).split(" ")[0] X+ hash_key = hash(frozenset(column.items())) X+ self.column_types[hash_key] = self.column_type_info(column).split(" ")[0] X+ return self.column_types[hash_key] X X def column_type_info(self, column): X """ X@@ -26,15 +29,14 @@ X return 'integer DEFAULT nextval(\'%s_%s_seq\'::regclass) NOT NULL' % ( X column['table_name'], column['name']) X X- X null = "" if column['null'] else " NOT NULL" X- X+ X def get_type(column): X """This in conjunction with :py:class:`mysql2pgsql.lib.mysql_reader.MysqlReader._convert_type` X determines the PostgreSQL data type. In my opinion this is way too fugly, will need X to refactor one day. X """ X- def t(v): return not v == None X+ t = lambda v: not v == None X default = (' DEFAULT %s' % QuotedString(column['default']).getquoted()) if t(column['default']) else None X X if column['type'] == 'char': X@@ -74,11 +76,13 @@ X default = None X return default, 'date' X elif column['type'] == 'timestamp': X- if "CURRENT_TIMESTAMP" in column['default']: X+ if column['default'] == None: X+ default = None X+ elif "CURRENT_TIMESTAMP" in column['default']: X default = ' DEFAULT CURRENT_TIMESTAMP' X- if "0000-00-00 00:00" in column['default']: X+ elif "0000-00-00 00:00" in column['default']: X default = " DEFAULT '1970-01-01 00:00'" X- if "0000-00-00 00:00:00" in column['default']: X+ elif "0000-00-00 00:00:00" in column['default']: X default = " DEFAULT '1970-01-01 00:00:00'" X return default, 'timestamp without time zone' X elif column['type'] == 'time': X@@ -90,8 +94,8 @@ X return default, 'text' X elif re.search(r'^enum', column['type']): X default = (' %s::character varying' % default) if t(default) else None X- enum = re.sub(r'enum|\(|\)', '', column['type']) X- max_enum_size = max([(len(e) - 2) for e in enum.split(',')]) X+ enum = re.sub(r'^enum\(|\)$', '', column['type']) X+ max_enum_size = max([len(e.replace("''", "'")) for e in enum.split("','")]) X return default, ' character varying(%s) check(%s in (%s))' % (max_enum_size, column['name'], enum) X elif 'bit(' in column['type']: X return ' DEFAULT %s' % column['default'].upper() if column['default'] else column['default'], 'varbit(%s)' % re.search(r'\((\d+)\)', column['type']).group(1) X@@ -111,14 +115,15 @@ X sending to PostgreSQL via the copy command X """ X for index, column in enumerate(table.columns): X- column_type = self.column_type(column) X+ hash_key = hash(frozenset(column.items())) X+ column_type = self.column_types[hash_key] if hash_key in self.column_types else self.column_type(column) X if row[index] == None and ('timestamp' not in column_type or not column['default']): X row[index] = '\N' X elif row[index] == None and column['default']: X row[index] = '1970-01-01 00:00:00' X elif 'bit' in column_type: X row[index] = bin(ord(row[index]))[2:] X- elif row[index].__class__ in (str, unicode): X+ elif isinstance(row[index], (str, unicode, basestring)): X if column_type == 'bytea': X row[index] = Binary(row[index]).getquoted()[1:-8] if row[index] else row[index] X elif 'text[' in column_type: X@@ -126,10 +131,11 @@ X else: X row[index] = row[index].replace('\\', r'\\').replace('\n', r'\n').replace('\t', r'\t').replace('\r', r'\r').replace('\0', '') X elif column_type == 'boolean': X- row[index] = 't' if row[index] == 1 else 'f' if row[index] == 0 else row[index] X- elif row[index].__class__ in (date, datetime): X+ # We got here because you used a tinyint(1), if you didn't want a bool, don't use that type X+ row[index] = 't' if row[index] not in (None, 0) else 'f' if row[index] == 0 else row[index] X+ elif isinstance(row[index], (date, datetime)): X row[index] = row[index].isoformat() X- elif row[index].__class__ is timedelta: X+ elif isinstance(row[index], timedelta): X row[index] = datetime.utcfromtimestamp(row[index].total_seconds()).time().isoformat() X else: X row[index] = AsIs(row[index]).getquoted() X@@ -149,7 +155,6 @@ X columns.write(' %s,\n' % self.column_description(column)) X return primary_keys, serial_key, maxval, columns.getvalue()[:-2] X X- X def truncate(self, table): X serial_key = None X maxval = None X@@ -182,7 +187,7 @@ X serial_key_sql.append('SELECT pg_catalog.setval(%s, %s, true);' % (QuotedString(serial_key_seq).getquoted(), maxval)) X X table_sql.append('DROP TABLE IF EXISTS "%s" CASCADE;' % table.name) X- table_sql.append('CREATE TABLE "%s" (\n%s\n)\nWITHOUT OIDS;' % (table.name, columns)) X+ table_sql.append('CREATE TABLE "%s" (\n%s\n)\nWITHOUT OIDS;' % (table.name.encode('utf8'), columns)) X return (table_sql, serial_key_sql) X X def write_indexes(self, table): X@@ -191,7 +196,7 @@ X if primary_index: X index_sql.append('ALTER TABLE "%(table_name)s" ADD CONSTRAINT "%(index_name)s_pkey" PRIMARY KEY(%(column_names)s);' % { X 'table_name': table.name, X- 'index_name': '%s_%s' % (table.name, '_'.join(primary_index[0]['columns'])), X+ 'index_name': '%s_%s' % (table.name, '_'.join(re.sub('[\W]+', '', c) for c in primary_index[0]['columns'])), X 'column_names': ', '.join('"%s"' % col for col in primary_index[0]['columns']), X }) X for index in table.indexes: X@@ -206,7 +211,7 @@ X 'table_name': table.name, X 'column_names': ', '.join('"%s"' % col for col in index['columns']), X }) X- X+ X return index_sql X X def write_constraints(self, table): X--- mysql2pgsql/lib/writer.py Thu Aug 18 13:23:12 2011 -0700 X+++ /dev/null Thu Jan 01 00:00:00 1970 +0000 X@@ -1,2 +0,0 @@ X-class Writer(object): X- pass X--- /dev/null Thu Jan 01 00:00:00 1970 +0000 X+++ requirements.txt Wed Jul 04 12:03:14 2012 +0300 X@@ -0,0 +1,5 @@ X+mysql-python X+psycopg2 X+pyyaml X+termcolor X+argparse X--- setup.py Thu Aug 18 13:23:12 2011 -0700 X+++ setup.py Wed Jul 04 12:03:14 2012 +0300 X@@ -5,6 +5,7 @@ X 'mysql-python>=1.2.3', X 'psycopg2>=2.4.2', X 'pyyaml>=3.10.0', X+ 'argparse', X ] X X if os.name == 'posix': X--- /dev/null Thu Jan 01 00:00:00 1970 +0000 X+++ tests/mysql2pgsql-test.yml Wed Jul 04 12:03:14 2012 +0300 X@@ -0,0 +1,38 @@ X+ X+# if a socket is specified we will use that X+# if tcp is chosen you can use compression X+mysql: X+ hostname: 127.0.0.1 X+ port: 3306 X+ socket: X+ username: root X+ password: X+ database: mysql2pgsql X+ compress: false X+destination: X+ # if file is given, output goes to file, else postgres X+ file: X+ postgres: X+ hostname: 127.0.0.1 X+ port: 5432 X+ username: postgres X+ password: X+ database: mysql2pgsql X+ X+# if tables is given, only the listed tables will be converted. leave empty to convert all tables. X+#only_tables: X+#- table1 X+#- table2 X+# if exclude_tables is given, exclude the listed tables from the conversion. X+#exclude_tables: X+#- table3 X+#- table4 X+ X+# if supress_data is true, only the schema definition will be exported/migrated, and not the data X+supress_data: false X+ X+# if supress_ddl is true, only the data will be exported/imported, and not the schema X+supress_ddl: false X+ X+# if force_truncate is true, forces a table truncate before table loading X+force_truncate: false X--- tests/schema.sql Thu Aug 18 13:23:12 2011 -0700 X+++ tests/schema.sql Wed Jul 04 12:03:14 2012 +0300 X@@ -718,7 +718,7 @@ X X -- SPLIT X X-INSERT INTO `type_conversion_test_2` (`tct_1_id`, `foo`) VALUES (2, 'baz'); X+INSERT INTO `type_conversion_test_2` (`tct_1_id`, `foo`) VALUES (2, 'special characters:čćžšđ'); X X -- SPLIT X X--- tests/test_reader.py Thu Aug 18 13:23:12 2011 -0700 X+++ tests/test_reader.py Wed Jul 04 12:03:14 2012 +0300 X@@ -31,7 +31,7 @@ X } X X if self.options.get('password', None): X- self.args['passwd'] = self.options.get('password', None), X+ self.args['passwd'] = self.options.get('password', None) X X if self.options.get('socket', None): X self.args['unix_socket'] = self.options['socket'] 0c3925eddecf298c646bd0c56b7311c4 echo x - py-mysql2pgsql/pkg-descr sed 's/^X//' >py-mysql2pgsql/pkg-descr << '4b6a2462b80b0f47d9fc1e47e66fc6ee' XTool for migrating/converting from mysql to postgresql. X XWWW: http://packages.python.org/py-mysql2pgsql/ 4b6a2462b80b0f47d9fc1e47e66fc6ee exit >Release-Note: >Audit-Trail: >Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201207041250.q64CoKLD085709>