-
Notifications
You must be signed in to change notification settings - Fork 1
Conversion to Python3 Notes
Python 3 has a more restrictive variable naming convention. All variables and function names are wanted to be at least 3 characters in lenght.
Also a number of names are now reserved for built-ins.
This just seems to cause noise when running pylint.
It looks like d-rats is configuredto use CAP V1.1 messages. See https://alerts.weather.gov
It looks like CAP V1.2 may need an access for application, but the link from the above page does not work. The alternative may be ATOM format.
Geocoding is conversion of Geographic postal/street address information to latitude and longitude coordinates.
D-Rats used a bundled older geopy package do use a Yahoo geocoding service. That data service is not available any more.
Using a newer free geocoding service appears to need a newer geocoding module. Re-coded to use the python-geopy package instead of the bundled geopy.
Note free geocoding services can block a source for making repeated queries for the same information and making too many queries in a period of time.
Caching is recommended to avoid this blocking.
D-rats does not appear to be making automated queries, The interface is only apparently used by the GUI. But this should be noted in case someone decides to automate something in the future.
The import is replaced with lxml.
lxml is supposed to be simpler to use and better documented.
I am having trouble finding documentation for what the some of the libxml2 actually do, but finally think I have figured things out.
Affected modules are
- d-rats.py
- d-rats/wu.py
- d-rats/map_sources.py
- d-rats/cap.py - Note unit-test URL returns a 302 error. Document could not be found.
- d-rats/mainwindow.py
# from lxml import etreeThese functions are not available, and there does not appear to be any replacements for them. So that will remove libxml2 code from d-rats.py and mainwindow.py.
- doc = libxml2.parseMemory(formxml, len(formxml))
- doc.saveFile(outfile)
- doc.freeDoc()
+ doc = etree.fromstring(formxml)
+ doc.write(outfile, pretty_print=True)- styledoc = libxml2.parseFile(self.xslpath)
- style = libxslt.parseStylesheetDoc(styledoc)
- result = style.applyStylesheet(doc, None)
- style.saveResultToFilename(outfile, result, 0)
+ styledoc = etree.parse(self.xslpath)
+ style_sheet = etree.XSLT(styledoc)
+ result = style_sheet(doc)
+ result.write(outfile, pretty_print=True)- file_handle = open(self._filename)
- data = file_handle.read()
- file_handle.close()
- self.doc = libxml2.parseMemory(data, len(data))
+ self.doc = etree.parse(self._filename)- return style.saveResultToString(result)
+ return etree.tostring(result, pretty_print=True).decode()- def __del__(self):
- self.doc.freeDoc()- self.node.addContent(value)
+ self.node.text =value- if node.children:
- text = xml_unescape(node.getContent().strip())
+ if node.getchildren():
+ text = xml_unescape(node.text.strip())This is more complex under lxml
- widget_type = node.prop("type")
+ widget_type = None
+ for attrib, value in node.items():
+ if attrib == 'type':
+ widget_type = value
+ break- self.ident = field.prop("id")
+ for attrib, value in field.items():
+ if attrib == 'id':
+ self.ident = valueProperty "name" is now "tag".
- child = node.children
+ children = node.getchildren()
- while child:
- if child.name == "caption":
+ for child in children:
+ if child.tag == "caption":
cap_node = child
- elif child.name == "entry":
+ elif child.tag == "entry":
ent_node = childNamespaces are a dict:
-ctx.xpathRegisterNS("georss", "http://www.georss.org/georss")
+namespaces = {'georss': 'http://www.georss.org/georss'}
+items = doc.xpath(nodespec, namespaces=namespaces)- ctx = doc.xpathNewContext()
- forms = ctx.xpathEval("//form")
+ forms = doc.xpath("//form")
if len(forms) != 1:
raise Exception("%i forms in document" % len(forms))
- form = forms[0]
+ for attrib, value in forms[0].items():
+ if attrib == 'id':
+ self.ident = value
- ctx.xpathFreeContext()
Stackoverflow says UserDict has been moved to "collections".
This affects sessions/files.py module.
Still seems to be able to be used, so later on the list of things to do.
Do we still need the utils ExternalHash md5 hack?
Six it apparently a tool to hide the differences if python 2 to python 3 conversion.
This can be left in place for now, but after the conversion to python3, the six routines should be replaced by python 3 modules.
So far this is mainly a conversion from gtk2 to gtk3.
Which should also work on Python 2 once it is done.
I am not trying to maintain Python 2 compatibility with these changes. I am curious if it will still run with Python 2 though.
Originally I tried stuff like:
-import gtk
-import gobject
-import pango
+import gi
+gi.require_version("Gtk", "3.0")
+from gi.repository import Gdk
+from gi.repository import Gtk
+from gi.repository import GObject
+from gi.repository import PangoBut soon discovered that I needed to do something like the following.
The require_version is needed for minimum version of the first following from include statement Which means that the order is important. The order is most important in the "main" modules.
And I ended up using the new convention globally even though it made more edits.
import gi
gi.require_version("Gtk", "3.0")
from gi.repository import Gtk
from gi.repository import Gdk
from gi.repository import GObject
from gi.repository import PangoMany of the constants have changed and so have the options.
With gtk-3, the glade api has changed completely.
Some parts are easy. wtree.get_widget() apparently becomes wtree.get_object().
In config.py, the first major change is that gtk.glade.XML is no longer available.
p = os.path.join(dplatform.get_platform().source_dir(), "ui/addport.glade")
wtree = gtk.glade.XML(p, "addport", "D-RATS")Seeing conflicting documentation as to how to handle this on the wild wild web.
So since I have no experience with either version of Glade, started with the Python-gtk3-tutorial 2.1 Glade and Gtk.Builder
For unknown reasons the addport.glade file seems to have worked with out any changes.
The mainwindow.glade though first needed some manual changes, and then it needed a pass through the Glade 3.22.1 User interface Designer. With out the edits, the Glade User Interface Designer would not process the file.
After the glade file comes out of the Glade User Interface designer it is so changed around that any difference output is unusable.
So to track your changes for review, keep a renamed copy of the original mainwindow.glade, and a copy of the manually edited mainwindow.glade files. I have had to refer to them to try to determine what additional changes are made.
It took several iterations of attempting to load the mainwindow.glade before is came into the editor with out reporting errors. There are still a lot of items reported as deprecated. Those I will try to deal with later.
The first change was to change the outer tag from "glade-interface" to "interface"
The second change is a global replace of "widget" with "object".
Two instances instance of GtkImageMenuItem needed to be changed as that object longer has property class of "image". That section needed to be commented out. I don't remember if I fixed that in the Glade UID program.
The tooltip for "Show Station List" property needed to be commented out.
An instance of the GtkNotebook object named "main_tabs" had to have the "homogeneous" property commented out. This is a setting for the spacing of the tabs.
An instance of the GtkComboBox object named "files_selectport" had to have the "items" property commented out.
An instance of the GtkComboBox object named "event_typesel" had to have the "items" property commented out.
The GtkComboBoxEntry class no longer exists. I had to comment out the "child" sections that contained them, and in the Glade UID add a GtkComboBox entry to replace them. I Still have more work to do in that area, but for the short term I have also had to comment out some of the Python code that references the portions of the GtkComboboxEntry that no longer exists. This seems to have disabled the station selection portion of the GUI.
In the Glade UIL, I also changed all the HBox and VBox instances to just Box as HBox and VBox have both been deprecated. A Box directly replaces a Hbox. A VBox is a Box with a vertical orientation explicitly set.
The import "six*" is for backwards compatibility for python2 from python3. Once this is ported to python3, then the use of python2 compatibility hacks can be removed to make the code more understandable.
It looks like D-rats was partially setup for internationalization. Many of the text constants are wrapped with "_()" functions, and pylint is complaining about them
I have not done any internationalization, but learned some stuff in trying to find out why pylint was complaining.
The cause seems to be that an import gettext or some similar imports are needed. With out the import of gettext or equivalent the "_()" is a function that just returns the string argument it is passed.
With the gettext import, the string argument is some type of keyword that is used to lookup a replacement text for the current locale.
Now in one place the code is current doing compares of both an English constant that is enclosed in a "()" and some non-English text constants. This appears wrong, as if the locale was implemented properly the original string constant would have been replaced by the "()". So that is another issue that needs fixing.
Tickets are probably needed for these issues. For now, I am not going to be fixing them.
See also: https://stackoverflow.com/questions/36805505/gtk-stock-is-deprecated-whats-the-alternative
The gist of the above is that the stock icons are going to be gone after the conversion to Gtk-3. That means that internationalization support becomes more important.
Here we have a new issue with python3. In Python3, all strings are Unicode. Data sent over sockets, or to subprocesses, etc is bytes. Bytes must be converted to strings.
The problem is in the data being sent over the sockets is not legal UTF-8 sequences, so we can not just encode and decode utf-8 to send it. We have to use an 8 bit codec like 'ISO-8859-1' instead. Not sure if that choice will have other implications at this point.
In transport.py, comms.py, ddt2.py, and yenc.py (so far) in most of the cases what needs to be done is to prefix all the string constants with a 'b' qualifier to make them byte strings. This will still work on python2. Only a few places where we need to convert the an 8 bit integer to a 'byte' type do we need to use the
banned += b"="
out = b""
for char in buf:
if char in banned:
out += b"=" + chr((char + OFFSET) % 256).encode('ISO-8859-1')
else:
out += chr(char).encode('ISO-8859-1')Until we get some documentation of the protocol over the wire so that we can determine what is sent binary and what is intended to be Unicode text, I think this is the best that we can do.
There seems to be some attempt in the past for pylint filtering. I have my Visual Studio Code set up to highlight pylint issues.
The GTK 3 modules need exclusions from pylint as they are totally incompatible.
I have tried to setup a pylintrc for the issues caused by GTK-3.
For python3 almost all variables must be 3 characters.
D-Rats is also using variables that override Python 3 built in names.
I am trying to correct the pylint issues as I go along, mainly for the GTK 3 port and Python 3 port.
I am trying to make all lines A maximum of 80 characters if I have to reformat a block.
No new code should require pylint disables of broad-except, and bare-except. Code that handles exceptions should only handle expected exceptions.
The multithreaded nature of D-rats makes it hard to trace down a fix an exception that is caused by a coding error, or a change in an API when broad-except and bare-except are caught.
Found another port in progress to GTK3 Johnathan Kelly.
No updates to it for two years, and could not get d-rats client to launch.
The Gtk code was moved to Gtk 3, the gtk, gobject, and pango, which needs to match was not.
I can get the client up and it claims to connect to the test ratflector.
The GUI is not being rendered correctly still have a lot to chase down there.
The Glade editor at least is rendering the preview display the same way as the d-rats program.
The send function is not working and the GUI is too on that.
No data is being displayed for the map.
Many of the d_rats modules can be debugged standalone.
python -m d_rats.mainwindow
d_rats
...
11/10/2020 20:10:36 Version : HTTP_CLIENT_HEADERS={'User-Agent': 'd-rats/0.3.10 beta 3 '}
11/10/2020 20:10:37 Utils :Error opening icon map /mnt/aviary/home/malmberg/work/d-rats/D-Rats/images/aprs_pri.png: 'gi.repository.Gdk' object has no attribute 'pixbuf_new_from_file'
11/10/2020 20:10:37 Utils :Error opening icon map /mnt/aviary/home/malmberg/work/d-rats/D-Rats/images/aprs_sec.png: 'gi.repository.Gdk' object has no attribute 'pixbuf_new_from_file'
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/mnt/aviary/home/malmberg/work/d-rats/D-Rats/d_rats/mainwindow.py", line 35, in <module>
lang = gettext.translation("D-RATS", localedir="./locale", languages=["en"])
File "/usr/lib/python2.7/gettext.py", line 545, in translation
raise IOError(ENOENT, 'No translation file found for domain', domain)
Getting similar results with existing and GTK3.
Now passing in the upstream for python2 and python3 and documented in another page in this wiki.
Passing in my personal fork also.
On GTK3 the UI is messed up for test selection, until I fixed the issues that running it under Python3 complained about.
This leaves Python3 complaining that the use of Gtk.Table is deprecated and new code should be using Gtk.Grid.
Gtk.Grid.attach uses a completely different set parameters than Gtk.Table.attach. I made a quick attempt to fix that and the display was messed up.
Need to fix the test so that the Station is not hard coded in the test.
On both new and old I can not seem to get a response the packages from either the ratflector or a test client.
Needed to fix a bug to try to run the test.
Seems to give the same output as dplatform. Locally patched for python3 and python2. Not sure this module is even used. Did not find a .pyc file for it until I tested it.
Takes a STATION name as a parameter. Has some tests depending on a hard coded station name. The hardcoded station name should be configured as a test dummy name.
In any event the existing program crashed
$ python -m d_rats.sessionmgr
12/13/2020 10:37:58 Comm : Host does not require authentication
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/mnt/aviary/home/malmberg/work/d-rats/master/d_rats/sessionmgr.py", line 279, in <module>
sm = SessionManager(p, sys.argv[1])
IndexError: list index out of range
This test just seems to try to parse http://www.weather.gov/alerts/fl.cap. For the existing d-rats it fails.
Now passing in the upstream for python2 and python3 and documented in another page in this wiki.
Passing in my personal fork also.
Existing outputs some data and fails on test. GTK3 gives the same results.
AGW needs some type of server setup listening on port 8000. Need to figure out how to setup a dummy server for testing that.
Seems to work the same for existing and GTK3.
Sound not working on Anti-X linux.
dPlatform : Unable to determine sound header of : [Errno 2] No such file or directory: ''
Add radio has message:
/mnt/aviary/home/malmberg/work/d-rats/master/d_rats/config.py:281: GtkWarning: GtkSpinButton: setting an adjustment with non-zero page size is deprecated
wtree = gtk.glade.XML(p, "addport", "D-RATS")
Looks like Anti-X linux does not have a set of default sounds to use.
I found some ".oga" files in /usr/share/sounds/freedesktop/stereo that the python code did not recognize.
- The Config() init function takes a mandatory parameter that is not used.
- The play button should not be enabled if there is not a path to an existing file.
- If I can find a distro vendor provided package for stock sounds, D-Rats should be able to use those as defaults.
For paths, the "..." trips "TypeError dialog can not be used to creat instances of a subclass FileChooserDialog"
For Appearance, Changing color gives a Deprecation Warning: Gtk.ColorButton.get_color is deprecated, and Gdk.Color.to_string is deprecated.
In Chat, selecting a font, DeprecationWarning: Gtk.FontButton.get_font_name is deprecated
Add Radio failed:
in line 299 prompt for port: gi.repository.GLib.Error: gtk-builder-error-quark: /mnt/aviary/home/malmberg/work/d-rats/D-Rats/ui/addport.glade:26:63 Invalid property: GtkComboBox.items (11)
Adding a TCP/IP gateway failed:
/mnt/aviary/home/malmberg/work/d-rats/D-Rats/d_rats/config.py:1470: PyGTKDeprecationWarning: The "buttons" argument must be a Gtk.ButtonsType enum value. Please use the "add_buttons" method for adding buttons. See: https://wiki.gnome.org/PyGObject/InitializerDeprecations
field_dialog = inputdialog.FieldDialog()
E-mail accounts add has log noise:
/mnt/aviary/home/malmberg/work/d-rats/D-Rats/d_rats/config.py:1470: PyGTKDeprecationWarning: The "buttons" argument must be a Gtk.ButtonsType enum value. Please use the "add_buttons" method for adding buttons. See: https://wiki.gnome.org/PyGObject/InitializerDeprecations
field_dialog = inputdialog.FieldDialog()
E-mail access add has log noise:
/mnt/aviary/home/malmberg/work/d-rats/D-Rats/d_rats/config.py:1470: PyGTKDeprecationWarning: The "buttons" argument must be a Gtk.ButtonsType enum value. Please use the "add_buttons" method for adding buttons. See: https://wiki.gnome.org/PyGObject/InitializerDeprecations
field_dialog = inputdialog.FieldDialog()
Now working for Python 2 and Python 3 for the port and the upstream.
PR submitted with fixes to upstream.
The older PIL python library seems to have been replaced with Pillow.
Added support for PNG files and possibly GIF files.
Fixed for python2 on upstream.
Much work will be needed to get this to work with Python3. Because of requirements of Python 3 to use GTK 3, once converted it will not work for python 2.
Now working for upstream and my local fork.
Patched locally to not crash for upstream for python2, but not actually testing anything.
It fails on python3. The first issue is that it imports 'rfc822'. The rfc822 import has been deprecated since version 2.3. The email package is listed as a replacement.
The module is already doing a import of e-mail.
Locally fixed, just starts what it claims to be POP3 server. Currently no idea on how to test.
Fixes submitted to upstream.
Re-coded to use the python-geopy package instead of the bundled geopy.
Yahoo data source not available anymore.
Changed to use the Nominatim data source.
Nomination requires a attribution in the copyright notice. Same attribution that is required to use openstreetmap data.
Todo: Needs HTTP agent information from version.py. Looks like this could be done better.
Existing code brought up a dialog box and issued some log messages when I clicked on the buttons. Only import button seems to work.
11/24/2020 18:07:59 Version : HTTP_CLIENT_HEADERS={'User-Agent': 'd-rats/0.3.10 beta 3 '}
11/24/2020 18:08:00 Qst : QSTWeatherWU class retired
Traceback (most recent call last):
File "/mnt/aviary/home/malmberg/work/d-rats/master/d_rats/formbuilder.py", line 570, in but_new
d = FormBuilderGUI()
File "/mnt/aviary/home/malmberg/work/d-rats/master/d_rats/formbuilder.py", line 530, in __init__
self.vbox.pack_start(self.build_formprops(), 0,0,0)
File "/mnt/aviary/home/malmberg/work/d-rats/master/d_rats/formbuilder.py", line 443, in build_formprops
path = mainapp.get_mainapp().config.get("settings", "form_logo_dir")
AttributeError: 'NoneType' object has no attribute 'config'
Buttons do not work on existing version.
Works in both the existing and the GTK3
Existing code tries same connection as the AGW connection and fails.
Submitted fix to upstream for python2.
The set_marker method referenced by the test no longer exists.
Upstream now working for python3 and python2.
Works for existing.
The GTK3 version is not as nicely aligned.
Getting same results as for existing and GTK3. Both seem to work. Both are passing with python3 and python2.
Existing code fails
File "/mnt/aviary/home/malmberg/work/d-rats/master/d_rats/mainwindow.py", line 33, in <module>
lang = gettext.translation("D-RATS", localedir="./locale", languages=["en"])
File "/usr/lib/python2.7/gettext.py", line 545, in translation
raise IOError(ENOENT, 'No translation file found for domain', domain)
IOError: [Errno 2] No translation file found for domain: 'D-RATS'
Gtk3 code has been patched for that issue, but is failing with
11/24/2020 18:25:46 Qst : QSTWeatherWU class retired
11/24/2020 18:25:46 x Mainwin : can not open locale file
11/24/2020 18:25:46 x Mainwin : sys.path= ['..', '.', '', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/gtk-2.0']
(mainwindow.py:16047): Gtk-WARNING **: 18:25:46.908: GtkGrid does not have a child property called right_attach
11/24/2020 18:25:46 Config : FILE: /home/malmberg/.d-rats-ev/d-rats.config
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/mnt/aviary/home/malmberg/work/d-rats/D-Rats/d_rats/mainwindow.py", line 402, in <module>
chat.connect("user-sent-message", test)
TypeError: <main_chat.ChatTab object at 0x7f09ecd0ee10 (d_rats+ui+main_chat+ChatTab at 0x55ebe9486160)>: unknown signal name: user-sent-message
The log was being spammed with the default configuration
11/20/2020 17:04:27 Gps : Failed to parse DPRS comment: BN *20
11/20/2020 17:04:27 Gps : CHECKSUM(WB8TYW-1): 75 != 32
11/20/2020 17:04:27 Gps : Checksum : *20
11/20/2020 17:04:27 Gps : _checksum: *4B
11/20/2020 17:04:27 Gps : astidx : 4
11/20/2020 17:04:30 Gps : Failed to parse DPRS comment: BN *20
The Default configuration is creating a malformed DPRS comment including a checksum character. For existing configurations this needs to be manually fixed in the config settings. I think I have modified d-rats to not make this error on new instances. But as I update this, I am not finding what I changed.
Now fixed in the upstream.
The sendping for one of the tests is missing 'port' as a second parameter.
In my test setup, nothing is responding to pings.
The method is mainly incremental edits and tests to see where the program crashes.
Also using pylint and working on cleaning up what that reports.
Starting with d-rats_repeater.py since it should be simpler.