Random image of Textel, my bat fursona

COBERTOS

Bobskater - A Python obfuscation Library

Recently I had the need to obfuscate/minify some Python files before I distributed them. Coming from the Javascript world of ES6 where transpilers and processors are plentiful, I figured there would be a mature, well maintained library similar to UglifyJS in the Python ecosystem. It turns out that this isn't the case.

I decided to code and package my own, called bobskater. It turned out to be surprisingly easy with Python's ast library, the invaluable documentation at GreenTreeSnakes, and a little library called astunparse. Being completely AST based, it won't die on weird syntax or require special formatting. All obfuscation decisions are based on AST node type. There are a few limitations as of now but they're all documented in the repository.

There do exist a lot of libraries/techniques out there (free and pay) that perport to do this, with mixed results:

  • Compile to .pyc using compile(). Even at the highest optimization you can still extract the original code, variable names, and docstrings with uncompyle6 in literally just one command.
  • Encrypt your modules and hook the import functionality to decrypt them when loading them at runtime. Too simple to reverse in my opinion.
  • Compile your modules using Cython, which removes Python bytecode though introduces it's own set of problems, like not being able to call using named kwargs (see this citrisbyte article for more details). I could see this not playing nicely with Blender's import system.
  • opy - For a Regex-based obfuscator it worked surprisingly well except for one or two corner cases. Doesn't scale well though as all external identifiers that shouldn't get obfuscated need to be manually entered. Requires specific source code formatting but isn't an issue if your code is based on Python's recommendations
  • pyminifier - An unmaintained, hybrid regex/token-based obfuscator. 80 open issues and seemingly errors in very basic cases.
  • pyobfuscate - Last updated in 2013, not particularly well documented and non hopeful open issues.
  • mnfy - Python minifier but no obfuscation.
  • and a few others.
Read More »

Importing and Reloading Python Modules in Blender

Python in Blender can be tiring. A simple problem becomes an arduous trek through docs, examples, and sometimes the C API to find the Blender way to write given Python code. This is due to the many quirks of Blender's own internal Python environment.

Importing is one of those arduous tasks. Python provides a lot of functionality to import all different kinds of source and data files but Blender's implementation makes design decisions that create issues. This is my deep dive into Blender's Python import integration where at the end I provide a small module to make your future Blender importing life easier.

Importing

In a nutshell, Python's module importing/loading comes bundled in the from ... import ... as ... syntax. This syntax ultimately compiles to various forms of the __import__() function. Instead of interfacing with this directly, Python provides importlib in Python 3.4+ (previously imp in Python < 3.4) to make interfacing with __import__() much easier. Blender decides to forgo this setup for its own implementation with the intention of making their own integration work better with the way Blender handles its data. This allows it to support nifty features like code files being text datablocks and inline python snippets in other parts of the application.

Unfortunately this override seems to break some functionality including relative imports and simply makes other tasks like how to structure a multi-file project confusing. This is a real bummer for large Blender applications. The above StackOverflow answer recommends appending to sys.path which works just fine but leaves a more comprehensive system to be desired. Things like module reloading for easy development and the ability to register and unregister large applications at will to support the little checkbox in the User Preferences window.

This little checkbox is the bane of my existence

Importing%20and%20Reloading%20Python%20Modules%20in%20Blender%2082c71cae2f854e4d98a6a77b5f188726/blenderUserPreferences.png

blenderUserPreferences

NOTE: If you go very deep into the implementation, you'll find the above monkey patch of __import__() internally redirects to PyImport_ImportModuleLevel() which according to the CPython source uses __package__ and __spec__ or __name__ and __path__ to find the parent package to import from. From here, I assume Blender doesn't set these globals to standard values which causes issues traversing the package heirarchy and this is the cause of one such issue in the import system.

Reloading

Reloading Python modules is usually not a common task but is crucial in Blender. Contrast from normal Python development, the Python environment in Blender stays for as long as the application is open. It takes an application restart to clean the state of the interpretter and update any add-on changes. Module reloading alone will only take you so far because of complications from nested child modules, Blender register() functions, and other necessities.

In the amount of I was developing on a single add-on for Blender, I found it useful to make my own importer that handles specific tasks I would otherwise type in manually. Below is a small class that will handle module loading and registering for you. See bl_register(self, moduleNames) and bl_unregister(self).

"""
Utility that allow a system of registering and unregistering of different modules
within a framework/add-on to be loaded into Blender

A module imported with this can have a bl_register and a bl_unregister at the
top level to add what it needs to Blender upon add-on load and unload

Note that this will handle recursively reloading of child modules, so module loops
will definitely cause headaches. Be wary of how you use this
"""

import importlib
import sys
from types import ModuleType

def rreload(module):
    '''Recursively reload modules.'''
    importlib.reload(module) 
    for childModule in [v for v in module.__dict__.values() if v is ModuleType]:
        rreload(childModule)

class BLModuleLoader:
    '''
    A module loader that plays nice with Blenders reloading system
    '''
    def __init__(self):
        self._registeredModules = {}

    def _register(self, name):
        '''
        Imports and registers a single module where name is the "."
        separated list for the package, absolute or relative
        '''
        importEquivalentStr = "import " + name
        try:
            if name in sys.modules:
                print("Reloading import \"" + importEquivalentStr + "\"")
                rreload(sys.modules[name])
            else:
                print("Importing for first time \"" + importEquivalentStr + "\"")
                sys.modules[name] = importlib.import_module(name)
            imported = sys.modules[name]
        except ImportError as e:
            print("Importing error " + name + ": " + str(e))
            return None
        
        if "bl_register" in dir(imported) and name not in self._registeredModules:
            try:
                imported.bl_register()
            except ValueError as e:
                print("Registration error " + name + ": " + str(e))
                return None
        self._registeredModules[name] = imported
        return imported

    def _unregister(self, name):
        '''
        Unregisters a previously registered module
        '''
        module = self._registeredModules[name]
        if "bl_unregister" in dir(module):
            try:
                module.bl_unregister()
            except ValueError as e:
                print("Unregistration error " + name + ": " + str(e))
                return None
        del self._registeredModules[name]

    def bl_register(self, moduleNames):
        '''
        Registers all the passed modules, returning a dict of all
        the loaded modules, key'd by their passed names
        '''
        return { k : self._register(k) for k in moduleNames }

    def bl_unregister(self):
        '''
        Unregisters all the modules that were previously imported
        '''
        for name in list(self._registeredModules.keys()):
            self._unregister(name)

This code is backed by 5 unit tests that handle loading and unloading modules, both nested and top level, inside of a Python environment run alongside headless Blender (though creating these unit tests to dynamically create and unload modules in Python was even itself a challenge).

Usage looks like as follows:

#__init__.py on top level of module
from BLModuleUtils import BLModuleLoader

#Create the loader
ml = BLModuleLoader
def register():
    #Register your modules
    loaded = ml.bl_register(["myModuleOnPythonPath",
        "myOtherModuleOnPythonPath"])
        
    #loaded holds the loaded modules if they worked
    #at the corresponding key
    #e.g. loaded["myModuleOnPythonPath"]
    #otherwise it will be None if an error occured
    #during the load and registration process

def unregister():
    #Unregister your modules
    ml.bl_unregister()
    
#myModuleOnPythonPath.py in the same folder, or wherever if you use a . path
def bl_register():
    bpy.utils.register_class(Your_RNA_Meta_Class)
def bl_unregister():
    bpy.utils.unregister_class(Your_RNA_Meta_Class)

Toggling and untoggling the checkbox will call unregister() and register() which will reload your modules and reregister all your new class bytecode. Just make sure you watch out for things that stick around like draw call handlers from bpy.types.SpaceView3D.draw_handler_add.

A lot of this was possible thanks to some great research from those around the Internet that have made similar plunges into Python's module loading system:

Read More »

Compiling Blender as a Python Module for Windows 10 x64 using Visual Studio

If you want to do unit tests of Blender Python code, it might be to your benefit to not startup Blender every time you want to run them but to just import Blender as a Python module and run them from the command line. This is especially important/nice if you want to automate your tests. Note, before you jump in, if you just need mathutils you can get that separately here.

Luckily, Blender's build has a nifty feature by which you can compile it as a Python module and then import blender from Python...

import bpy #Starts up Blender as Python module
from mathutils import Vector #Import a Blender specific library

Unfortunately for me, all of the tutorials I found were not geared toward the target I was looking for: Blender 2.78, Windows 10 x64, Microsoft Visual Studio Express 2013+, and Python 3.5. After a couple days I was able to get it all building and working with this tool chain and it should work just as well for x86, a different Python version, or a different MSVS version.

The Build Process

  1. Install needed programs.

    • SVN: to check out the precompiled Windows dependencies
    • CMake: to make Blender
    • Microsoft Visual Studio >2013: For Blender compilation
    • Blender Source Code or Git: Blender source code as downloaded from the website or git to checkout the exact commit you want to build
    • Python installation of your desired version of the same bitness you want to build Blender in (Determining Python Bitness).
  2. Create a directory structure like the following

.
..
blender/                      #Blender source goes in here
build/                        #This is where you have CMake target
lib/win[dows,64]_vc[12,14]/   #Prebuilt binaries. The name matters! `windows` for 32 bit or `win64` for 64 bit, `vc12` for MSVS12 (2013) or `vc14` for MSVS14 (2015) and MSVS15 (2017)
  1. Checkout the precompiled Windows dependencies based on your bitness desired

  2. Download the source (or checkout the git repo) into the blender/ folder.

    • Blender's Git repos though you'll want to clone git://git.blender.org/blender.git specifically.
    • If you're using git make sure to checkout the specific commit you want and also git submodule update --init --recursive
  3. In build/, run cmake -DWITH_PLAYER=OFF -DWITH_PYTHON_INSTALL=OFF -DWITH_PYTHON_MODULE=ON -DPYTHON_VERSION=3.5 ../blender

    • For 64 bit, you might need to force CMake to use a specific generator. -G"Visual Studio 15 Win64" where 15 is your Visual Studio version
    • Replace the Python version number with your chosen Python version
    • There are other configurable options found in blender/CMakeLists.txt if you want to enable/disable other features (like the game engine)
  4. In the Visual Studio Developer Command Prompt, also in build/, run devenv Blender.sln /Build [TARGET] /Project INSTALL where [TARGET] is a Visual Studio release target. Most likely you'll want Release

    • You can see the other targets if you open the the .sln file.
  5. You now have the built files. Copy the files into the Python's global site-packages directory

copy bin\bpy.pyd C:\Python35\Lib\site-packages\
copy bin\*.dll C:\Python35\Lib\site-packages\
del C:\Python35\Lib\site-packages\python35.dll
xcopy /E bin\2.78 C:\Python35\2.78\

You can now open up python and test the module. Just open python on the command line and type import bpy and you now have access to Blender's Python modules. bpy.app will also give you useful information about the current build.

If you prefer to install it in a virtual environment, the commands above work just the same, though with the last one, make sure that 2.78 is copied into your virtual env Scripts folder and not the root

Troubleshooting:

  • Something with _Insert_n mentioning Eigen: Go into the mentioned .h (.hpp?) and change vector_base::_Insert_n to vector_base::insert in the corresponding .cpp file.
  • %1 is not a valid Win32 application: You have built the 32 bit Blender and tried running it from 64 bit Python. Rebuild with the correct bitness.
  • bpy: couldnt find 'scripts/modules', blender probably wont start.: You need to install the .pyd and related files in Python's site-packages, otherwise it cannot find the supporting files.

Thanks to all the other wonderful people who wrote tutorials that got me part of the way through this build!

Read More »