Better function call matching (also decorators) #17

FichteFoll · 2014-09-03T03:41:08Z

As discussed in #16, I finally worked on the function call matching that I already thought of in #8.

Following is a snippet that I used for testing:

list(map(str, range(100)))
[str(i) for i in range(100)]

super(arg1=1)(arg2=2)
list abs, map
some.None NotImplemented

print (file=None) print . __class__
print keyword print __init__

__init__ . something()

callback(print)

.__init__
__init__ [2]


@ deco . __init__ (call=some)  # rator
def function():
    pass

Closes #16.

PS: This is unrelated, but there are a couple of include: $self occurences that are technically incorrect because in most of these cases only an expression is allowed. However, the snytax def itself is not grouped into statement vs expression, so the grouping would be a required step for this (and would also add more structure).

Essentially reverts #8, but this is in scope of a bigger change. Also clearly separate built-in functions and built-in types because some used to be duplicated.

The previous attempt just didn't work because of two reasons: 1. The "dotted_name" was already consumed in function-call's "begin" match. 2. All "special" identifiers were essentially matched before the function call which meant that the actual call would never be matched since the function to be called (the identifier) was already consumed. Simplify #dotted_name a bit in the process.

The print statement can now only be matched if it's not used in a function previously since otherwise it would've been consumed.

FichteFoll · 2014-09-03T04:18:38Z

Forgot about decorators but for some reason I can't get them to work properly right now. Maybe I'll see it after some sleep.

Edit: got it in the end

Although it looked okay-ish with background color it was not intended.

Otherwise, color schemes defining a background color for `entitiy.name.function` would make decorators span the remainder of the line.

FichteFoll · 2014-09-04T14:10:20Z

Fixed a few regressions, mostly regarding decorators.

Test snippet:

@normal.decorator
class Class():

    @indented(method)#comment
    def wrapper(self):
        pass

MattDMo · 2014-09-04T16:33:38Z

Sorry about my slow feedback, but I'm really taking my time with this, trying to understand all the changes you're making and if they are in the direction I want to go. Feel free to keep pushing more commits, I'm trying to go through them in order, but alas real work is also interfering 😩 so it may be some time before I accept anything. Thanks for your help, though!

FichteFoll · 2014-09-04T18:14:03Z

Sure thing. It's not like I depend on an upstream merge here since I can use my local version just fine, so take your time.

I only noticed the regressions as I happend to view a file with lots of decorators that happend to be open while I was testing sublimehq/sublime_text#464. Other than that I think everything's fine.

Furthermore, since I now have a feeling of how the syntax is structured I can make other changes more quickly, such as splitting the syntax def into statements and expressions, should I ever feel bored enough to do that.

Previously, `self.CONSTANT` was not matched.

Or well, it's only valid syntax in Python 3.

Even though we can't really match that correctly, at least match the leading dot on a new line or after a function call. That's more correct. Example: ident = (value.dotted .__init__)

It could be of use for regression testing later, or whatever. In any case it should illustrate most of the changes of my pull request and can easily be extended.

FichteFoll · 2014-10-01T16:04:10Z

Fixed highlighting of ALL_CAPS constants in dotted names which really annoyed me and addressed a few other issues that just crossed my mind.

I also added my personal test file for future reference. It uses the .pyw extension so that it doesn't get loaded by ST as a plugin (since it's invalid syntax anyway).

FichteFoll · 2014-12-23T01:17:07Z

poke

FichteFoll · 2015-02-04T14:30:30Z

Fixed an annoying regression with from . import statements.

It's likely harder to merge this now because you made changes on your own and they are now conflicting.

MattDMo · 2015-09-09T22:12:17Z

@FichteFoll sorry for the huge delay, but I'm finally getting around to thinking about releasing PythonImproved 2.0, and I've incorporated a lot of your changes from this PR into PythonImproved-Unicode.YAML-tmLanguage in the unicode branch. However, right off the bat, I've noticed an issue with meta.dotted-name.python — it's being applied to all meta.identifier.python scopes. If I comment out line 583 it doesn't show up in the identifier scope anymore. Thoughts?

FichteFoll · 2015-09-10T02:01:58Z

Well, pretty much everything in Python is a "dotted name" (because I made the dotted part before the last 'undotted' part just optional). #dotted_name includes #generic_names, which is your meta.identifier.python. Note that this is used for "identifier that does not have another special meaning" and does not include the dots. meta.dotted-name.python is just the "catch-all" for pretty much all dotted names you'll encounter, including the illegal ones with keywords for example.

This part would look a lot simpler with contexts from sublime-syntax btw.

FichteFoll · 2015-09-10T02:14:39Z

I have the feeling that there are quite a few parts missing which I will look into tomorrow. I'll probably prepare a syntax-test file from my fork and compare that to your unicode branch.

MattDMo · 2015-09-10T02:17:02Z

OK, thanks. I'm hoping to release 2.0 by Sunday at the latest (probably quickly followed by 2.0.1, etc.) to coincide with the release of Python 3.5. After that we can talk about making a .sublime-syntax version.

FichteFoll · 2015-09-11T16:26:40Z

I prepared a syntax test file from my previously used test file and pushed it to the branch.

Notable issues with your unicode branch (visible in testfile):

The actual reason why I made the PR in the first place (Built-in types are highlighted as support.function and not support.type if called #16): types are matched as functions instead of types when used in function calls
Built-in functions are only matched when used in a function call and not when used as a callback parameter for example
Ellipsis and NotImplemented should not be matched as attributes
parameters for magic function calls do not get highlighted
print matched as invalid in several places where it is valid, just as a function
allcaps constants not matched in attribtue access from results from function calls
decorators with spaces fail deliberately
decorators only have the initial name highlighted as entity.name.function.decorator (which is imo the wrong scope anyway) and do not continue over the entire decorator call, if there is one. This is actually wrong in my version too, but at least consistent

So yeah, it's probably 80% of my work that's still missing.

I honestly do not feel like revisiting this again. You just took too long to merge the changes I made and made too many changes on the main and unicode branch in the meantime so that merging will be a hassle.

Instead, I will most likely work on a new Python.sublime-syntax version from scratch, while taking some inspiration from my fork. There are a couple of things where contexts would be way easier than this.

MattDMo · 2015-12-04T19:15:34Z

I'm going to close this, mainly for the reasons you mentioned above - it's no longer mergeable, and things have just moved on. I think working on a .sublime-syntax version is a great idea, although the audience will be somewhat limited due to the fact that the new format isn't supported yet in the public beta. (Speaking of which, any word from Jon on when development is going to start up again?).

I've started #54 to track .sublime-syntax work, and I'll be making a new branch soon.

FichteFoll · 2015-12-04T20:23:24Z

(Speaking of which, any word from Jon on when development is going to start up again?)

None that I know of, sadly.

I made a pretty nice YAML syntax definition and submitted it to the packages repo 4 months ago (sublimehq/Packages#90), but no word since. My end goal would be to have a neutral, comprehensive and accurate Python.sublime-syntax that was bundled with Sublime Text by default, but so far I'm hesitant due to the lack of updates.
As you mentioned, .sublime-syntax is still a dev-only feature and I'd rather spend my time on plugins for all people (on ST3) rather than just the ones on dev. There are numerous ways for me do to that.

Yet, I think of how I would structure it at least twice a week.

MattDMo · 2015-12-04T20:26:58Z

Definitely understandable. I'll work on it now and again just to learn how the format works, but I'm certainly not expecting a rush effort on PI to push it out the door.

facelessuser · 2015-12-04T21:26:08Z

sublimehq/Packages#90

Nice, I didn't know about this. I will have to try it out.

Discussed in MattDMo/PythonImproved#17

FichteFoll added 4 commits September 3, 2014 04:54

Built-in functions are now only matched by ident

3ae9d47

Essentially reverts #8, but this is in scope of a bigger change. Also clearly separate built-in functions and built-in types because some used to be duplicated.

Re-add print and del statement matching

828ab43

The print statement can now only be matched if it's not used in a function previously since otherwise it would've been consumed.

Spaces are allowed in dotted idents

b847ef4

Forgot about decorators and item access

cf5b349

FichteFoll changed the title ~~Better function call matching~~ Better function call matching (also decorators) Sep 3, 2014

FichteFoll added 2 commits September 4, 2014 16:02

Fix indent of decorators being highlighted

3ea8cb8

Although it looked okay-ish with background color it was not intended.

dotted_name should not include linebreaks

42bb653

Otherwise, color schemes defining a background color for `entitiy.name.function` would make decorators span the remainder of the line.

FichteFoll added 4 commits October 1, 2014 17:51

Match ALL_CAPS constants in dotted names

61479aa

Previously, `self.CONSTANT` was not matched.

print must be an ident if followed by ,

17ea9f5

Or well, it's only valid syntax in Python 3.

Dotted names may span multiple lines

ee007cb

Even though we can't really match that correctly, at least match the leading dot on a new line or after a function call. That's more correct. Example: ident = (value.dotted .__init__)

Add my personal testing file

c9ad654

It could be of use for regression testing later, or whatever. In any case it should illustrate most of the changes of my pull request and can easily be extended.

FichteFoll added 2 commits December 23, 2014 02:16

Match print before a line break as Py3 ident

b57a009

Few more test cases

05b34f3

Fix "from . import" and add more test cases

f17fb74

This was referenced Mar 6, 2015

Swap Quotes command fails for Python triple-quoted strings facelessuser/BracketHighlighter#233

Closed

Greedy error scope #31

Closed

Unicode identifiers are not considered #19

Closed

FichteFoll mentioned this pull request Jul 4, 2015

Incorporate changes from Python Improved? sublimehq/Packages#51

Closed

MattDMo pushed a commit that referenced this pull request Sep 9, 2015

Should address the major stuff in #17

095560e

FichteFoll added 2 commits September 11, 2015 17:32

unicode

bb3185f

Restructure tests to use syntax testing features

f226221

MattDMo closed this Dec 4, 2015

infininight pushed a commit to infininight/python.tmbundle that referenced this pull request Apr 20, 2016

Improve function call matching

d8e1b63

Discussed in MattDMo/PythonImproved#17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Better function call matching (also decorators) #17

Better function call matching (also decorators) #17

FichteFoll commented Sep 3, 2014

FichteFoll commented Sep 3, 2014

FichteFoll commented Sep 4, 2014

MattDMo commented Sep 4, 2014

FichteFoll commented Sep 4, 2014

FichteFoll commented Oct 1, 2014

FichteFoll commented Dec 23, 2014

FichteFoll commented Feb 4, 2015

MattDMo commented Sep 9, 2015

FichteFoll commented Sep 10, 2015

FichteFoll commented Sep 10, 2015

MattDMo commented Sep 10, 2015

FichteFoll commented Sep 11, 2015

MattDMo commented Dec 4, 2015

FichteFoll commented Dec 4, 2015

MattDMo commented Dec 4, 2015

facelessuser commented Dec 4, 2015

Better function call matching (also decorators) #17

Better function call matching (also decorators) #17

Conversation

FichteFoll commented Sep 3, 2014

FichteFoll commented Sep 3, 2014

FichteFoll commented Sep 4, 2014

MattDMo commented Sep 4, 2014

FichteFoll commented Sep 4, 2014

FichteFoll commented Oct 1, 2014

FichteFoll commented Dec 23, 2014

FichteFoll commented Feb 4, 2015

MattDMo commented Sep 9, 2015

FichteFoll commented Sep 10, 2015

FichteFoll commented Sep 10, 2015

MattDMo commented Sep 10, 2015

FichteFoll commented Sep 11, 2015

MattDMo commented Dec 4, 2015

FichteFoll commented Dec 4, 2015

MattDMo commented Dec 4, 2015

facelessuser commented Dec 4, 2015