Random image of Textel, my bat fursona

COBERTOS

Bobskater - A Python obfuscation Library

Recently I had the need to obfuscate/minify some Python files before I distributed them. Coming from the Javascript world of ES6 where transpilers and processors are plentiful, I figured there would be a mature, well maintained library similar to UglifyJS in the Python ecosystem. It turns out that this isn't the case.

I decided to code and package my own, called bobskater. It turned out to be surprisingly easy with Python's ast library, the invaluable documentation at GreenTreeSnakes, and a little library called astunparse. Being completely AST based, it won't die on weird syntax or require special formatting. All obfuscation decisions are based on AST node type. There are a few limitations as of now but they're all documented in the repository.

There do exist a lot of libraries/techniques out there (free and pay) that perport to do this, with mixed results:

  • Compile to .pyc using compile(). Even at the highest optimization you can still extract the original code, variable names, and docstrings with uncompyle6 in literally just one command.
  • Encrypt your modules and hook the import functionality to decrypt them when loading them at runtime. Too simple to reverse in my opinion.
  • Compile your modules using Cython, which removes Python bytecode though introduces it's own set of problems, like not being able to call using named kwargs (see this citrisbyte article for more details). I could see this not playing nicely with Blender's import system.
  • opy - For a Regex-based obfuscator it worked surprisingly well except for one or two corner cases. Doesn't scale well though as all external identifiers that shouldn't get obfuscated need to be manually entered. Requires specific source code formatting but isn't an issue if your code is based on Python's recommendations
  • pyminifier - An unmaintained, hybrid regex/token-based obfuscator. 80 open issues and seemingly errors in very basic cases.
  • pyobfuscate - Last updated in 2013, not particularly well documented and non hopeful open issues.
  • mnfy - Python minifier but no obfuscation.
  • and a few others.
Read More »

Cubecus for Blender is here!

Cubecus is here! After a long time coming (4 years on and off), I've gotten my Blender level design add-on into a releasable state (even though in alpha and with some glitches). Don't worry though! It's going to get better with more releases where I plan to separate out the obfuscator it uses into a separate project and add more features and fixes.

Cubecus usage gif

If you're unfamiliar, give it's page a look. It explains about the different tools it adds and provides screenshots of use (like the one below).

I hope that everyone who's asked about it and even those who've not, find it useful :)

Read More »

Importing and Reloading Python Modules in Blender

Python in Blender can be tiring. A simple problem becomes an arduous trek through docs, examples, and sometimes the C API to find the Blender way to write given Python code. This is due to the many quirks of Blender's own internal Python environment.

Importing is one of those arduous tasks. Python provides a lot of functionality to import all different kinds of source and data files but Blender's implementation makes design decisions that create issues. This is my deep dive into Blender's Python import integration where at the end I provide a small module to make your future Blender importing life easier.

Importing

In a nutshell, Python's module importing/loading comes bundled in the from ... import ... as ... syntax. This syntax ultimately compiles to various forms of the __import__() function. Instead of interfacing with this directly, Python provides importlib in Python 3.4+ (previously imp in Python < 3.4) to make interfacing with __import__() much easier. Blender decides to forgo this setup for its own implementation with the intention of making their own integration work better with the way Blender handles its data. This allows it to support nifty features like code files being text datablocks and inline python snippets in other parts of the application.

Unfortunately this override seems to break some functionality including relative imports and simply makes other tasks like how to structure a multi-file project confusing. This is a real bummer for large Blender applications. The above StackOverflow answer recommends appending to sys.path which works just fine but leaves a more comprehensive system to be desired. Things like module reloading for easy development and the ability to register and unregister large applications at will to support the little checkbox in the User Preferences window.

This little checkbox is the bane of my existence

Importing%20and%20Reloading%20Python%20Modules%20in%20Blender%2082c71cae2f854e4d98a6a77b5f188726/blenderUserPreferences.png

blenderUserPreferences

NOTE: If you go very deep into the implementation, you'll find the above monkey patch of __import__() internally redirects to PyImport_ImportModuleLevel() which according to the CPython source uses __package__ and __spec__ or __name__ and __path__ to find the parent package to import from. From here, I assume Blender doesn't set these globals to standard values which causes issues traversing the package heirarchy and this is the cause of one such issue in the import system.

Reloading

Reloading Python modules is usually not a common task but is crucial in Blender. Contrast from normal Python development, the Python environment in Blender stays for as long as the application is open. It takes an application restart to clean the state of the interpretter and update any add-on changes. Module reloading alone will only take you so far because of complications from nested child modules, Blender register() functions, and other necessities.

In the amount of I was developing on a single add-on for Blender, I found it useful to make my own importer that handles specific tasks I would otherwise type in manually. Below is a small class that will handle module loading and registering for you. See bl_register(self, moduleNames) and bl_unregister(self).

"""
Utility that allow a system of registering and unregistering of different modules
within a framework/add-on to be loaded into Blender

A module imported with this can have a bl_register and a bl_unregister at the
top level to add what it needs to Blender upon add-on load and unload

Note that this will handle recursively reloading of child modules, so module loops
will definitely cause headaches. Be wary of how you use this
"""

import importlib
import sys
from types import ModuleType

def rreload(module):
    '''Recursively reload modules.'''
    importlib.reload(module) 
    for childModule in [v for v in module.__dict__.values() if v is ModuleType]:
        rreload(childModule)

class BLModuleLoader:
    '''
    A module loader that plays nice with Blenders reloading system
    '''
    def __init__(self):
        self._registeredModules = {}

    def _register(self, name):
        '''
        Imports and registers a single module where name is the "."
        separated list for the package, absolute or relative
        '''
        importEquivalentStr = "import " + name
        try:
            if name in sys.modules:
                print("Reloading import \"" + importEquivalentStr + "\"")
                rreload(sys.modules[name])
            else:
                print("Importing for first time \"" + importEquivalentStr + "\"")
                sys.modules[name] = importlib.import_module(name)
            imported = sys.modules[name]
        except ImportError as e:
            print("Importing error " + name + ": " + str(e))
            return None
        
        if "bl_register" in dir(imported) and name not in self._registeredModules:
            try:
                imported.bl_register()
            except ValueError as e:
                print("Registration error " + name + ": " + str(e))
                return None
        self._registeredModules[name] = imported
        return imported

    def _unregister(self, name):
        '''
        Unregisters a previously registered module
        '''
        module = self._registeredModules[name]
        if "bl_unregister" in dir(module):
            try:
                module.bl_unregister()
            except ValueError as e:
                print("Unregistration error " + name + ": " + str(e))
                return None
        del self._registeredModules[name]

    def bl_register(self, moduleNames):
        '''
        Registers all the passed modules, returning a dict of all
        the loaded modules, key'd by their passed names
        '''
        return { k : self._register(k) for k in moduleNames }

    def bl_unregister(self):
        '''
        Unregisters all the modules that were previously imported
        '''
        for name in list(self._registeredModules.keys()):
            self._unregister(name)

This code is backed by 5 unit tests that handle loading and unloading modules, both nested and top level, inside of a Python environment run alongside headless Blender (though creating these unit tests to dynamically create and unload modules in Python was even itself a challenge).

Usage looks like as follows:

#__init__.py on top level of module
from BLModuleUtils import BLModuleLoader

#Create the loader
ml = BLModuleLoader
def register():
    #Register your modules
    loaded = ml.bl_register(["myModuleOnPythonPath",
        "myOtherModuleOnPythonPath"])
        
    #loaded holds the loaded modules if they worked
    #at the corresponding key
    #e.g. loaded["myModuleOnPythonPath"]
    #otherwise it will be None if an error occured
    #during the load and registration process

def unregister():
    #Unregister your modules
    ml.bl_unregister()
    
#myModuleOnPythonPath.py in the same folder, or wherever if you use a . path
def bl_register():
    bpy.utils.register_class(Your_RNA_Meta_Class)
def bl_unregister():
    bpy.utils.unregister_class(Your_RNA_Meta_Class)

Toggling and untoggling the checkbox will call unregister() and register() which will reload your modules and reregister all your new class bytecode. Just make sure you watch out for things that stick around like draw call handlers from bpy.types.SpaceView3D.draw_handler_add.

A lot of this was possible thanks to some great research from those around the Internet that have made similar plunges into Python's module loading system:

Read More »

Compiling Blender as a Python Module for Windows 10 x64 using Visual Studio

If you want to do unit tests of Blender Python code, it might be to your benefit to not startup Blender every time you want to run them but to just import Blender as a Python module and run them from the command line. This is especially important/nice if you want to automate your tests. Note, before you jump in, if you just need mathutils you can get that separately here.

Luckily, Blender's build has a nifty feature by which you can compile it as a Python module and then import blender from Python...

import bpy #Starts up Blender as Python module
from mathutils import Vector #Import a Blender specific library

Unfortunately for me, all of the tutorials I found were not geared toward the target I was looking for: Blender 2.78, Windows 10 x64, Microsoft Visual Studio Express 2013+, and Python 3.5. After a couple days I was able to get it all building and working with this tool chain and it should work just as well for x86, a different Python version, or a different MSVS version.

The Build Process

  1. Install needed programs.

    • SVN: to check out the precompiled Windows dependencies
    • CMake: to make Blender
    • Microsoft Visual Studio >2013: For Blender compilation
    • Blender Source Code or Git: Blender source code as downloaded from the website or git to checkout the exact commit you want to build
    • Python installation of your desired version of the same bitness you want to build Blender in (Determining Python Bitness).
  2. Create a directory structure like the following

.
..
blender/                      #Blender source goes in here
build/                        #This is where you have CMake target
lib/win[dows,64]_vc[12,14]/   #Prebuilt binaries. The name matters! `windows` for 32 bit or `win64` for 64 bit, `vc12` for MSVS12 (2013) or `vc14` for MSVS14 (2015) and MSVS15 (2017)
  1. Checkout the precompiled Windows dependencies based on your bitness desired

  2. Download the source (or checkout the git repo) into the blender/ folder.

    • Blender's Git repos though you'll want to clone git://git.blender.org/blender.git specifically.
    • If you're using git make sure to checkout the specific commit you want and also git submodule update --init --recursive
  3. In build/, run cmake -DWITH_PLAYER=OFF -DWITH_PYTHON_INSTALL=OFF -DWITH_PYTHON_MODULE=ON -DPYTHON_VERSION=3.5 ../blender

    • For 64 bit, you might need to force CMake to use a specific generator. -G"Visual Studio 15 Win64" where 15 is your Visual Studio version
    • Replace the Python version number with your chosen Python version
    • There are other configurable options found in blender/CMakeLists.txt if you want to enable/disable other features (like the game engine)
  4. In the Visual Studio Developer Command Prompt, also in build/, run devenv Blender.sln /Build [TARGET] /Project INSTALL where [TARGET] is a Visual Studio release target. Most likely you'll want Release

    • You can see the other targets if you open the the .sln file.
  5. You now have the built files. Copy the files into the Python's global site-packages directory

copy bin\bpy.pyd C:\Python35\Lib\site-packages\
copy bin\*.dll C:\Python35\Lib\site-packages\
del C:\Python35\Lib\site-packages\python35.dll
xcopy /E bin\2.78 C:\Python35\2.78\

You can now open up python and test the module. Just open python on the command line and type import bpy and you now have access to Blender's Python modules. bpy.app will also give you useful information about the current build.

If you prefer to install it in a virtual environment, the commands above work just the same, though with the last one, make sure that 2.78 is copied into your virtual env Scripts folder and not the root

Troubleshooting:

  • Something with _Insert_n mentioning Eigen: Go into the mentioned .h (.hpp?) and change vector_base::_Insert_n to vector_base::insert in the corresponding .cpp file.
  • %1 is not a valid Win32 application: You have built the 32 bit Blender and tried running it from 64 bit Python. Rebuild with the correct bitness.
  • bpy: couldnt find 'scripts/modules', blender probably wont start.: You need to install the .pyd and related files in Python's site-packages, otherwise it cannot find the supporting files.

Thanks to all the other wonderful people who wrote tutorials that got me part of the way through this build!

Read More »

Projected Balance Calculator Pt Deux

I updated the balance projection calculator thingy a week or so ago but I thought I'd write about it. Here's a demo image:

Graph of > 1 year worth of financial transaction projections

Graph of > 1 year worth of financial transaction projections. The blue graph represents the balance, the orange represents balance with a savings percentage (in this case %30), and the green represents the total amount possible to spend without hitting 0.

New features include:

  • Filled graph with colors for extra spice
  • Multiple graphs including maximum possible expenditures and savings percentages
  • Labels to show exactly what charges come out when (using a given label when setting up the charges)

One of the issues I noticed after I generated this is that you need to generate a sufficient amount into the future to make sure that really far off charges aren't missed in the factors applied to the maximum possible expenditures graph.

As of now this solves my problem so I don't know if I'll need to add any more features. Perhaps if I ever have larger saving/investing goals.

Read More »

Projected Balance Calculator

I always felt that if I was able to see how much money I would have in the future I could more accurately save, spend, and make bill payments instead of just worrying all the time. Such a tool exists as a part of Quicken but Quicken is a slow, confusing piece of software that I bought and ended up returning because it was that bad(!). I looked but Mint.com doesn't have it, GnuCash doesn't have it, and doing it in Excel would require I learn VB, which, let's be honest, would probably result in me developing some serious medical condition.

I coded up the class model while I was at the barber shop in a Google Note and finished it in 3-ish hours at home. It honestly took more time to find the exact days that all my different services charge me on and put that in the program. It's pretty small and it needs a few more useful features before I can really use it but even in its current state I already feel comfortable where I am financially. Code can be found on GitHub.

Graph of projected balance over 90 days from today using a mixture of bogus and real data.

Graph of projected balance over 90 days from today using a mixture of bogus and real data.

The nicest part about this was I found libraries for both the plotting (Plotly) and for the reoccuring date problem (python-dateutil). My resultant program, including tests, was only about 100 lines long.

Anyway, it's been a pretty tough week and I don't think this will be my last endeavor with Python in the coming months. I want to revisit my music library and update that program so I guess we'll see how that goes.

Read More »

Why Python is so exciting for me

Mp3 TaggingMp3Python

I’m in love with Python. Like madly in love. If I wasn’t pretty crazy about this one girl right now I’m sure that I’d probably start hanging out with Python more, take it out to dinner, get to know it better, you know how it is.

I remember when I first looked at Python and went “What in the hell, who would want to program in this?! No braces? Colons?? How do you read the code??” Eventually I grew to really love it and has become possibly my favorite to program in (on the same level as C#). I know some that will try to force their beliefs on others when it comes to languages and IDEs. I only want to share with you my excitement for this language. In terms of enjoyment though, I really do believe a favorite language is based on personal taste and people should respect that.

But why do I love it? I could probably go on and on about the exact features I like but that’d get a little repetitive. If I had to choose one thing I’d have to say the simplicity of the collections matched with their flexibility and power is my favorite feature. Also, the fact that there’s a library that pretty much does anything you want just one console command away doesn’t hurt either (with PyPI).

Python is also the first language I could just rattle out simple tools to serve my life with.

  • Need to backup my PC? Built a backup script before packing up to leave for college.
  • Need to run 20 files through some programs for school over the next 7 weeks? Wrote up a simple script to saves me hours of work in looking over the logs of the programs.
  • My latest project with Python (pictured below) has been to tag all my music files. I just wanted the album and album art for my songs. The program corrects all needed searching fields, searches Discogs and Google for album info, and then uses an external program to get an image to embed in the mp3 all in just two days of work. Two Days!

Sample program output (a search on Discogs)

#[3] Try discogs search, list first few
req = requests.get("http://www.discogs.com/search/?q=" + searchTextEnc)
if req.status_code == 200:
    results = []
    album = None
    try:
        soup = BeautifulSoup(req.text, "html.parser")
        resEl = soup.find(id="search_results")
        for divEl in resEl.select("div.card"):
            #Try to get the artist and album
            try:
                foundAlbum = divEl.find("h4").find("a").string.strip()
                foundArtist = divEl.find("h5").find("a").string.strip()
            except: #Possibly no artist or album for given "card"
                continue
                
            results.append((foundAlbum, foundArtist))
            if len(results) > 20:
                break
    except:
        results = None
    
    if results:
        #Display results for user to choose
        for i, (alb, art) in enumerate(results):
            printStr = "[" + str(i) + "] " + alb + " by " + art
            print(printStr.encode("ascii", "replace").decode("ascii"))

I tried a similar thing years ago with Perl for managing a large texture database. I was just not able to get the same speed or robustness from the program that I am akin to with my Python abilities.

While Python is my most recently learned language the speed at which I can code and the way the code fits with my mindset just works so well. While I do dread the ever nearing obsolescence date of any coding language that I use, I wonder if at some point in the future another language will be created that will top my enjoyment for Python.

Read More »