Lessons learned with Visual Studio, C89 and the preprocessor
Another two weeks of GSoC are done. Many lessons learned, many problems hit, most of them solved.
After that I concentrated on aligning this with JyNI's usual build process.
It quickly turned out that already JyNI's DemoExtension was not C89 compliant. After fixing this, I got the demo extension working on original CPython 2.7, which asserts that the ABI is correct.
With this in line, I focused on JyNI itself and checked for an automatic C99 to C89 approach. It turned out that this approach wouldn't be workable without additional efforts, so I started to manually convert the JyNI codebase to C89.
During this work I suffered from frequent mouse freezes on Windows 10. While these always occurred during working with Eclipse I cannot tell whether it was Eclipse related. Also for other reasons I preferred to return to my Linux environment and checked for ways to achieve this.
First idea was to configure gcc to behave more like MSVC, e.g. accept C89 only. This approach proved insufficient for producing MSVC compatible code, so I tried to run cl.exe using Wine. Finally I came up with a handy wine-based makefile that allows me to develop MSVC compatible code on Linux (linking not yet tested).
The next bigger challenge was that the C preprocessor works slightly differently on MSVC, beyond C89 issues. With that solved, C89 conversion continues smoothly now.
The CPython 2.7 PC-build README mentions that “Another configuration proven to work is Visual Studio 2008, Windows SDK 7.1, and Visual Studio 2013.” So I was curious if it could alternatively work with MSVS 2015 + MSVS 2008 + SDK 7.1 (Nice thing with MSVS 2015 is that it is the official tool for building Python 3.). But after installing this huge setup (for each of them the download alone took 1.5 hours for me, so more or less the whole day passed by with setup stuff) it turned out that MSVS 2015 could not detect the compiler from MSVS 2008, unless also MSVS 2010 would be installed. Indeed, that ultimately solved the problem. So, after many hours of setup and more than 20GB harddrive space filled up, I was finally able to compile CPython 2.7. Next step was to build a custom C extension using this setup and to assert that the result is workable with original CPython 2.7. First issue here was C89 compatibility of JyNI's DemoExtension. With that fixed I ran into various linking issues using MSVS GUI despite following various building tutorials pedantically. (The official documentation does not cover building with MSVS GUI). After some more head-banging-on-table it turned out that it is necessary to reference the pythoncore project in addition to linker- and include-configuration.
With that working I wanted to figure out how to accomplish this from CLI, i.e. by calling cl.exe. Since this involves complex command lines with include paths etc I directly approached this using makefiles. On Windows it takes a bit more than selecting make in the package manager, e.g. select an actual make implementation first. For sake of free software I decided to use GNU make. Rather than selecting a distribution I quickly built it from source, which was luckily straight forward with three entire distributions of MSVS installed.
It turned out that it is not suitable to run vcvarsall.bat from a makefile: The adjusted environment variables are not memorized for subsequent tasks.
So I figured out – partly from the build command issued by MSVS GUI – what libs needed to be linked etc. The only info the user must adjust in the resulting makefile is the location of MSVC, but that would have also been required to run vcvarsall.bat, because it is the location of that file.
I am using it as the default C compiler in JyNI's DemoExtension's makefile now.
The next step was to build JyNI itself on Windows. With lessons learned from the DemoExtension it was straight forward to setup a Windows makefile for JyNI itself. However, MSVC 9.0 requires C89 code and converting JyNI's entire codebase to C89 is a different story than converting the DemoExtension.
Unfortunately it lacks support for variable length arrays (VLA).
I found that alloca or _alloca/_malloca in MSVC respectively is the closest simulation of VLA in C89 that you can get. Since it is still not fully equivalent to C99 VLA (frees the memory on function exit rather than on block exit), I wrote the VLA macro to keep the original C99 VLA implementation in gcc/clang case.
Also lacking support for the “inline” keyword caused lots of trouble, but could be easily fixed by declaring it as a macro. With this it was more or less straight forward to convert several files to C89, e.g.
JyAlloc.c, JySync.c, JyNI.c, JyAbstract.c, JyHash.c, JyList.c and JyTState.c.
Then I hit a major C preprocessor issue with MSVC that goes beyond C89 vs C99 topic.
So I checked if it would be possible to configure gcc for C89 in order to do at least the C99 to C89 conversion from Linux. Unfortunately as soon as you call gcc with -std=c89 it starts complaining about “// ...” C++ style comments, which are still fine for MSVC 9.0. There is apparently no option to run gcc in C89 mode except for C++ style comments. This puzzles me, given that it looks like a popular use case to assert MSVC compatibility of cross platform code. So I got the idea to perform preprocessing separately, i.e. running the preprocessor in C99 mode to strip away comments, while doing the actual compile task in C89 mode. However, this worked, but still did not yield MSVC compliant C code – there must be more to it. Same result with using -ANSI.
Finally I gave up the ggc-approach (didn't try with clang) and tried to run cl.exe using Wine. This basically worked, but had issues with passing space-containing paths, even when surrounded by quotation marks. Maybe this would have been solvable by fiddling around with makefile escape sequence notation (I tried this in various ways, but without success, so there must be more to it). Instead I now let the makefile create symlinks to the folders in question. This works like a charm and resulted in a wine-based makefile for JyNI (linking not yet tested).
Overview
The first three days I spent with getting the original setup for building CPython 2.7 on Windows in line.After that I concentrated on aligning this with JyNI's usual build process.
It quickly turned out that already JyNI's DemoExtension was not C89 compliant. After fixing this, I got the demo extension working on original CPython 2.7, which asserts that the ABI is correct.
With this in line, I focused on JyNI itself and checked for an automatic C99 to C89 approach. It turned out that this approach wouldn't be workable without additional efforts, so I started to manually convert the JyNI codebase to C89.
During this work I suffered from frequent mouse freezes on Windows 10. While these always occurred during working with Eclipse I cannot tell whether it was Eclipse related. Also for other reasons I preferred to return to my Linux environment and checked for ways to achieve this.
First idea was to configure gcc to behave more like MSVC, e.g. accept C89 only. This approach proved insufficient for producing MSVC compatible code, so I tried to run cl.exe using Wine. Finally I came up with a handy wine-based makefile that allows me to develop MSVC compatible code on Linux (linking not yet tested).
The next bigger challenge was that the C preprocessor works slightly differently on MSVC, beyond C89 issues. With that solved, C89 conversion continues smoothly now.
Official setup for Python 2.7 on Windows
I somehow thought that I should be able to build CPython 2.7 using the original toolchain.The CPython 2.7 PC-build README mentions that “Another configuration proven to work is Visual Studio 2008, Windows SDK 7.1, and Visual Studio 2013.” So I was curious if it could alternatively work with MSVS 2015 + MSVS 2008 + SDK 7.1 (Nice thing with MSVS 2015 is that it is the official tool for building Python 3.). But after installing this huge setup (for each of them the download alone took 1.5 hours for me, so more or less the whole day passed by with setup stuff) it turned out that MSVS 2015 could not detect the compiler from MSVS 2008, unless also MSVS 2010 would be installed. Indeed, that ultimately solved the problem. So, after many hours of setup and more than 20GB harddrive space filled up, I was finally able to compile CPython 2.7. Next step was to build a custom C extension using this setup and to assert that the result is workable with original CPython 2.7. First issue here was C89 compatibility of JyNI's DemoExtension. With that fixed I ran into various linking issues using MSVS GUI despite following various building tutorials pedantically. (The official documentation does not cover building with MSVS GUI). After some more head-banging-on-table it turned out that it is necessary to reference the pythoncore project in addition to linker- and include-configuration.
With that working I wanted to figure out how to accomplish this from CLI, i.e. by calling cl.exe. Since this involves complex command lines with include paths etc I directly approached this using makefiles. On Windows it takes a bit more than selecting make in the package manager, e.g. select an actual make implementation first. For sake of free software I decided to use GNU make. Rather than selecting a distribution I quickly built it from source, which was luckily straight forward with three entire distributions of MSVS installed.
It turned out that it is not suitable to run vcvarsall.bat from a makefile: The adjusted environment variables are not memorized for subsequent tasks.
So I figured out – partly from the build command issued by MSVS GUI – what libs needed to be linked etc. The only info the user must adjust in the resulting makefile is the location of MSVC, but that would have also been required to run vcvarsall.bat, because it is the location of that file.
Windows toolchain with GNU make and Visual C++ for Python
To sort things out regarding building (from command line), the Cython Wiki was enormously helpful. It pointed me to the Visual C++ for Python distribution, a link that I really miss in the official Python documentation. This ~84MB file contains the entire toolchain required to build extensions for CPython 2.7 (and maybe even CPython 2.7 itself? Let's see...) and is thus extremely lightweight compared to the setup described above.I am using it as the default C compiler in JyNI's DemoExtension's makefile now.
The next step was to build JyNI itself on Windows. With lessons learned from the DemoExtension it was straight forward to setup a Windows makefile for JyNI itself. However, MSVC 9.0 requires C89 code and converting JyNI's entire codebase to C89 is a different story than converting the DemoExtension.
Converting C99 to C89
The first thing I tried was to check for automatic conversion tools, which brought me to ffmpeg's c99-to-c89.Unfortunately it lacks support for variable length arrays (VLA).
I found that alloca or _alloca/_malloca in MSVC respectively is the closest simulation of VLA in C89 that you can get. Since it is still not fully equivalent to C99 VLA (frees the memory on function exit rather than on block exit), I wrote the VLA macro to keep the original C99 VLA implementation in gcc/clang case.
Also lacking support for the “inline” keyword caused lots of trouble, but could be easily fixed by declaring it as a macro. With this it was more or less straight forward to convert several files to C89, e.g.
JyAlloc.c, JySync.c, JyNI.c, JyAbstract.c, JyHash.c, JyList.c and JyTState.c.
Then I hit a major C preprocessor issue with MSVC that goes beyond C89 vs C99 topic.
Return to Linux: Using gcc in C89 mode vs cl.exe with Wine
During work on C89 conversion I suffered from mouse cursor freezes on Windows 10 (luckily the touch screen kept working so I could save the work and cleanly restart). Also the system appears to be constantly slower than Linux on the same machine and finally the continuous annoyance by automatic updates contributed its part to my wish to return to my Linux environment for developing.So I checked if it would be possible to configure gcc for C89 in order to do at least the C99 to C89 conversion from Linux. Unfortunately as soon as you call gcc with -std=c89 it starts complaining about “// ...” C++ style comments, which are still fine for MSVC 9.0. There is apparently no option to run gcc in C89 mode except for C++ style comments. This puzzles me, given that it looks like a popular use case to assert MSVC compatibility of cross platform code. So I got the idea to perform preprocessing separately, i.e. running the preprocessor in C99 mode to strip away comments, while doing the actual compile task in C89 mode. However, this worked, but still did not yield MSVC compliant C code – there must be more to it. Same result with using -ANSI.
Finally I gave up the ggc-approach (didn't try with clang) and tried to run cl.exe using Wine. This basically worked, but had issues with passing space-containing paths, even when surrounded by quotation marks. Maybe this would have been solvable by fiddling around with makefile escape sequence notation (I tried this in various ways, but without success, so there must be more to it). Instead I now let the makefile create symlinks to the folders in question. This works like a charm and resulted in a wine-based makefile for JyNI (linking not yet tested).
Comments
Post a Comment