c13b3d1350
Problem: No support for stable Python 3 ABI Solution: Support Python 3 stable ABI Commits: 1) Support Python 3 stable ABI to allow mixed version interoperatbility Vim currently supports embedding Python for use with plugins, and the "dynamic" linking option allows the user to specify a locally installed version of Python by setting `pythonthreedll`. However, one caveat is that the Python 3 libs are not binary compatible across minor versions, and mixing versions can potentially be dangerous (e.g. let's say Vim was linked against the Python 3.10 SDK, but the user sets `pythonthreedll` to a 3.11 lib). Usually, nothing bad happens, but in theory this could lead to crashes, memory corruption, and other unpredictable behaviors. It's also difficult for the user to tell something is wrong because Vim has no way of reporting what Python 3 version Vim was linked with. For Vim installed via a package manager, this usually isn't an issue because all the dependencies would already be figured out. For prebuilt Vim binaries like MacVim (my motivation for working on this), AppImage, and Win32 installer this could potentially be an issue as usually a single binary is distributed. This is more tricky when a new Python version is released, as there's a chicken-and-egg issue with deciding what Python version to build against and hard to keep in sync when a new Python version just drops and we have a mix of users of different Python versions, and a user just blindly upgrading to a new Python could lead to bad interactions with Vim. Python 3 does have a solution for this problem: stable ABI / limited API (see https://docs.python.org/3/c-api/stable.html). The C SDK limits the API to a set of functions that are promised to be stable across versions. This pull request adds an ifdef config that allows us to turn it on when building Vim. Vim binaries built with this option should be safe to freely link with any Python 3 libraies without having the constraint of having to use the same minor version. Note: Python 2 has no such concept and this doesn't change how Python 2 integration works (not that there is going to be a new version of Python 2 that would cause compatibility issues in the future anyway). --- Technical details: ====== The stable ABI can be accessed when we compile with the Python 3 limited API (by defining `Py_LIMITED_API`). The Python 3 code (in `if_python3.c` and `if_py_both.h`) would now handle this and switch to limited API mode. Without it set, Vim will still use the full API as before so this is an opt-in change. The main difference is that `PyType_Object` is now an opaque struct that we can't directly create "static types" out of, and we have to create type objects as "heap types" instead. This is because the struct is not stable and changes from version to version (e.g. 3.8 added a `tp_vectorcall` field to it). I had to change all the types to be allocated on the heap instead with just a pointer to them. Other functions are also simply missing in limited API, or they are introduced too late (e.g. `PyUnicode_AsUTF8AndSize` in 3.10) to it that we need some other ways to do the same thing, so I had to abstract a few things into macros, and sometimes re-implement functions like `PyObject_NEW`. One caveat is that in limited API, `OutputType` (used for replacing `sys.stdout`) no longer inherits from `PyStdPrinter_Type` which I don't think has any real issue other than minor differences in how they convert to a string and missing a couple functions like `mode()` and `fileno()`. Also fixed an existing bug where `tp_basicsize` was set incorrectly for `BufferObject`, `TabListObject, `WinListObject`. Technically, there could be a small performance drop, there is a little more indirection with accessing type objects, and some APIs like `PyUnicode_AsUTF8AndSize` are missing, but in practice I didn't see any difference, and any well-written Python plugin should try to avoid excessing callbacks to the `vim` module in Python anyway. I only tested limited API mode down to Python 3.7, which seemes to compile and work fine. I haven't tried earlier Python versions. 2) Fix PyIter_Check on older Python vers / type##Ptr unused warning For PyIter_Check, older versions exposed them as either macros (used in full API), or a function (for use in limited API). A previous change exposed PyIter_Check to the dynamic build because Python just moved it to function-only in 3.10 anyway. Because of that, just make sure we always grab the function in dynamic builds in earlier versions since that's what Python eventually did anyway. 3) Move Py_LIMITED_API define to configure script Can now use --with-python-stable-abi flag to customize what stable ABI version to target. Can also use an env var to do so as well. 4) Show +python/dyn-stable in :version, and allow has() feature query Not sure if the "/dyn-stable" suffix would break things, or whether we should do it another way. Or just don't show it in version and rely on has() feature checking. 5) Documentation first draft. Still need to implement v:python3_version 6) Fix PyIter_Check build breaks when compiling against Python 3.8 7) Add CI coverage stable ABI on Linux/Windows / make configurable on Windows This adds configurable options for Windows make files (both MinGW and MSVC). CI will also now exercise both traditional full API and stable ABI for Linux and Windows in the matrix for coverage. Also added a "dynamic" option to Linux matrix as a drive-by change to make other scripting languages like Ruby / Perl testable under both static and dynamic builds. 8) Fix inaccuracy in Windows docs Python's own docs are confusing but you don't actually want to use `python3.dll` for the dynamic linkage. 9) Add generated autoconf file 10) Add v:python3_version support This variable indicates the version of Python3 that Vim was built against (PY_VERSION_HEX), and will be useful to check whether the Python library you are loading in dynamically actually fits it. When built with stable ABI, it will be the limited ABI version instead (`Py_LIMITED_API`), which indicates the minimum version of Python 3 the user should have, rather than the exact match. When stable ABI is used, we won't be exposing PY_VERSION_HEX in this var because it just doesn't seem necessary to do so (the whole point of stable ABI is the promise that it will work across versions), and I don't want to confuse the user with too many variables. Also, cleaned up some documentation, and added help tags. 11) Fix Python 3.7 compat issues Fix a couple issues when using limited API < 3.8 - Crash on exit: In Python 3.7, if a heap-allocated type is destroyed before all instances are, it would cause a crash later. This happens when we destroyed `OptionsType` before calling `Py_Finalize` when using the limited API. To make it worse, later versions changed the semantics and now each instance has a strong reference to its own type and the recommendation has changed to have each instance de-ref its own type and have its type in GC traversal. To avoid dealing with these cross-version variations, we just don't free the heap type. They are static types in non-limited-API anyway and are designed to last through the entirety of the app, and we also don't restart the Python runtime and therefore do not need it to have absolutely 0 leaks. See: - https://docs.python.org/3/whatsnew/3.8.html#changes-in-the-c-api - https://docs.python.org/3/whatsnew/3.9.html#changes-in-the-c-api - PyIter_Check: This function is not provided in limited APIs older than 3.8. Previously I was trying to mock it out using manual PyType_GetSlot() but it was brittle and also does not actually work properly for static types (it will generate a Python error). Just return false. It does mean using limited API < 3.8 is not recommended as you lose the functionality to handle iterators, but from playing with plugins I couldn't find it to be an issue. - Fix loading of PyIter_Check so it will be done when limited API < 3.8. Otherwise loading a 3.7 Python lib will fail even if limited API was specified to use it. 12) Make sure to only load `PyUnicode_AsUTF8AndSize` in needed in limited API We don't use this function unless limited API >= 3.10, but we were loading it regardless. Usually it's ok in Unix-like systems where Python just has a single lib that we load from, but in Windows where there is a separate python3.dll this would not work as the symbol would not have been exposed in this more limited DLL file. This makes it much clearer under what condition is this function needed. closes: #12032 Signed-off-by: Christian Brabandt <cb@256bit.org> Co-authored-by: Yee Cheng Chin <ychin.git@gmail.com>
Vim source code
Here are a few hints for finding your way around the source code. This doesn't make it less complex than it is, but it gets you started.
You might also want to read
:help development.
Jumping around
First of all, use :make tags to generate a tags file, so that you can jump
around in the source code.
To jump to a function or variable definition, move the cursor on the name and
use the CTRL-] command. Use CTRL-T or CTRL-O to jump back.
To jump to a file, move the cursor on its name and use the gf command.
Most code can be found in a file with an obvious name (incomplete list):
| File name | Description |
|---|---|
| alloc.c | memory management |
| arglist.c | handling argument list |
| autocmd.c | autocommands |
| blob.c | blob data type |
| buffer.c | manipulating buffers (loaded files) |
| bufwrite.c | writing a buffer to file |
| change.c | handling changes to text |
| cindent.c | C and Lisp indentation |
| clientserver.c | client server functionality |
| clipboard.c | handling the clipboard |
| cmdexpand.c | command-line completion |
| cmdhist.c | command-line history |
| debugger.c | vim script debugger |
| diff.c | diff mode (vimdiff) |
| drawline.c | drawing a window line |
| drawscreen.c | drawing the windows |
| eval.c | expression evaluation |
| evalbuffer.c | buffer related built-in functions |
| evalfunc.c | built-in functions |
| evalvars.c | vim variables |
| evalwindow.c | window related built-in functions |
| fileio.c | reading and writing files |
| filepath.c | dealing with file names and paths |
| findfile.c | search for files in 'path' |
| fold.c | folding |
| getchar.c | getting characters and key mapping |
| help.c | vim help related functions |
| highlight.c | syntax highlighting |
| indent.c | text indentation |
| insexpand.c | Insert mode completion |
| locale.c | locale/language handling |
| map.c | mapping and abbreviations |
| mark.c | marks |
| match.c | highlight matching |
| float.c | floating point functions |
| mbyte.c | multi-byte character handling |
| memfile.c | storing lines for buffers in a swapfile |
| memline.c | storing lines for buffers in memory |
| menu.c | menus |
| message.c | (error) messages |
| mouse.c | handling the mouse |
| ops.c | handling operators ("d", "y", "p") |
| option.c | options |
| optionstr.c | handling string options |
| popupmenu.c | popup menu |
| popupwin.c | popup window |
| profiler.c | vim script profiler |
| quickfix.c | quickfix commands (":make", ":cn") |
| regexp.c | pattern matching |
| register.c | handling registers |
| scriptfile.c | runtime directory handling and sourcing scripts |
| screen.c | lower level screen functions |
| search.c | pattern searching |
| session.c | sessions and views |
| sign.c | signs |
| spell.c | spell checking core |
| spellfile.c | spell file handling |
| spellsuggest.c | spell correction suggestions |
| strings.c | string manipulation functions |
| syntax.c | syntax and other highlighting |
| tag.c | tags |
| term.c | terminal handling, termcap codes |
| testing.c | testing: assert and test functions |
| textformat.c | text formatting |
| textobject.c | text objects |
| textprop.c | text properties |
| time.c | time and timer functions |
| typval.c | vim script type/value functions |
| undo.c | undo and redo |
| usercmd.c | user defined commands |
| userfunc.c | user defined functions |
| viminfo.c | viminfo handling |
| window.c | handling split windows |
Debugging
If you have a reasonable recent version of gdb, you can use the :Termdebug
command to debug Vim. See :help :Termdebug.
When something is time critical or stepping through code is a hassle, use the channel logging to create a time-stamped log file. Add lines to the code like this:
ch_log(NULL, "Value is now %02x", value);
After compiling and starting Vim, do:
:call ch_logfile('debuglog', 'w')
And edit debuglog to see what happens. The channel functions already have
ch_log() calls, thus you always see that in the log.
Important Variables
The current mode is stored in State. The values it can have are NORMAL,
INSERT, CMDLINE, and a few others.
The current window is curwin. The current buffer is curbuf. These point
to structures with the cursor position in the window, option values, the file
name, etc. These are defined in
structs.h.
All the global variables are declared in
globals.h.
The main loop
This is conveniently called main_loop(). It updates a few things and then
calls normal_cmd() to process a command. This returns when the command is
finished.
The basic idea is that Vim waits for the user to type a character and
processes it until another character is needed. Thus there are several places
where Vim waits for a character to be typed. The vgetc() function is used
for this. It also handles mapping.
Updating the screen is mostly postponed until a command or a sequence of
commands has finished. The work is done by update_screen(), which calls
win_update() for every window, which calls win_line() for every line.
See the start of
screen.c
for more explanations.
Command-line mode
When typing a :, normal_cmd() will call getcmdline() to obtain a line
with an Ex command. getcmdline() contains a loop that will handle each typed
character. It returns when hitting CR or Esc or some other character that
ends the command line mode.
Ex commands
Ex commands are handled by the function do_cmdline(). It does the generic
parsing of the : command line and calls do_one_cmd() for each separate
command. It also takes care of while loops.
do_one_cmd() parses the range and generic arguments and puts them in the
exarg_t and passes it to the function that handles the command.
The : commands are listed in ex_cmds.h. The third entry of each item is
the name of the function that handles the command. The last entry are the
flags that are used for the command.
Normal mode commands
The Normal mode commands are handled by the normal_cmd() function. It also
handles the optional count and an extra character for some commands. These
are passed in a cmdarg_t to the function that handles the command.
There is a table nv_cmds in
normal.c
which lists the first character of every command. The second entry of each
item is the name of the function that handles the command.
Insert mode commands
When doing an i or a command, normal_cmd() will call the edit()
function. It contains a loop that waits for the next character and handles it.
It returns when leaving Insert mode.
Options
There is a list with all option names in
option.c,
called options[].
The GUI
Most of the GUI code is implemented like it was a clever terminal. Typing a
character, moving a scrollbar, clicking the mouse, etc. are all translated
into events which are written in the input buffer. These are read by the
main code, just like reading from a terminal. The code for this is scattered
through gui.c.
For example, gui_send_mouse_event() for a mouse click and gui_menu_cb() for
a menu action. Key hits are handled by the system-specific GUI code, which
calls add_to_input_buf() to send the key code.
Updating the GUI window is done by writing codes in the output buffer, just
like writing to a terminal. When the buffer gets full or is flushed,
gui_write() will parse the codes and draw the appropriate items. Finally the
system-specific GUI code will be called to do the work.
Debugging the GUI
Remember to prevent that gvim forks and the debugger thinks Vim has exited,
add the -f argument. In gdb: run -f -g.
When stepping through display updating code, the focus event is triggered
when going from the debugger to Vim and back. To avoid this, recompile with
some code in gui_focus_change() disabled.
Contributing
If you would like to help making Vim better, see the
CONTRIBUTING.md
file.
This is README.md for version 9.0 of the Vim source code.