Random image of Textel, my bat fursona

COBERTOS

Bobskater - A Python obfuscation Library

Recently I had the need to obfuscate/minify some Python files before I distributed them. Coming from the Javascript world of ES6 where transpilers and processors are plentiful, I figured there would be a mature, well maintained library similar to UglifyJS in the Python ecosystem. It turns out that this isn't the case.

I decided to code and package my own, called bobskater. It turned out to be surprisingly easy with Python's ast library, the invaluable documentation at GreenTreeSnakes, and a little library called astunparse. Being completely AST based, it won't die on weird syntax or require special formatting. All obfuscation decisions are based on AST node type. There are a few limitations as of now but they're all documented in the repository.

There do exist a lot of libraries/techniques out there (free and pay) that perport to do this, with mixed results:

  • Compile to .pyc using compile(). Even at the highest optimization you can still extract the original code, variable names, and docstrings with uncompyle6 in literally just one command.
  • Encrypt your modules and hook the import functionality to decrypt them when loading them at runtime. Too simple to reverse in my opinion.
  • Compile your modules using Cython, which removes Python bytecode though introduces it's own set of problems, like not being able to call using named kwargs (see this citrisbyte article for more details). I could see this not playing nicely with Blender's import system.
  • opy - For a Regex-based obfuscator it worked surprisingly well except for one or two corner cases. Doesn't scale well though as all external identifiers that shouldn't get obfuscated need to be manually entered. Requires specific source code formatting but isn't an issue if your code is based on Python's recommendations
  • pyminifier - An unmaintained, hybrid regex/token-based obfuscator. 80 open issues and seemingly errors in very basic cases.
  • pyobfuscate - Last updated in 2013, not particularly well documented and non hopeful open issues.
  • mnfy - Python minifier but no obfuscation.
  • and a few others.
Read More »

Friends make weird noises

codepenjavascriptexperimentvisualsound

Made this in like 3 hours. Ripped the audio from a video my friend sent me and it felt like a good fit.

Read More »

Cubecus for Blender is here!

Cubecus is here! After a long time coming (4 years on and off), I've gotten my Blender level design add-on into a releasable state (even though in alpha and with some glitches). Don't worry though! It's going to get better with more releases where I plan to separate out the obfuscator it uses into a separate project and add more features and fixes.

Cubecus usage gif

If you're unfamiliar, give it's page a look. It explains about the different tools it adds and provides screenshots of use (like the one below).

I hope that everyone who's asked about it and even those who've not, find it useful :)

Read More »

Importing and Reloading Python Modules in Blender

Python in Blender can be tiring. A simple problem becomes an arduous trek through docs, examples, and sometimes the C API to find the Blender way to write given Python code. This is due to the many quirks of Blender's own internal Python environment.

Importing is one of those arduous tasks. Python provides a lot of functionality to import all different kinds of source and data files but Blender's implementation makes design decisions that create issues. This is my deep dive into Blender's Python import integration where at the end I provide a small module to make your future Blender importing life easier.

Importing

In a nutshell, Python's module importing/loading comes bundled in the from ... import ... as ... syntax. This syntax ultimately compiles to various forms of the __import__() function. Instead of interfacing with this directly, Python provides importlib in Python 3.4+ (previously imp in Python < 3.4) to make interfacing with __import__() much easier. Blender decides to forgo this setup for its own implementation with the intention of making their own integration work better with the way Blender handles its data. This allows it to support nifty features like code files being text datablocks and inline python snippets in other parts of the application.

Unfortunately this override seems to break some functionality including relative imports and simply makes other tasks like how to structure a multi-file project confusing. This is a real bummer for large Blender applications. The above StackOverflow answer recommends appending to sys.path which works just fine but leaves a more comprehensive system to be desired. Things like module reloading for easy development and the ability to register and unregister large applications at will to support the little checkbox in the User Preferences window.

This little checkbox is the bane of my existence

Importing%20and%20Reloading%20Python%20Modules%20in%20Blender%2082c71cae2f854e4d98a6a77b5f188726/blenderUserPreferences.png

blenderUserPreferences

NOTE: If you go very deep into the implementation, you'll find the above monkey patch of __import__() internally redirects to PyImport_ImportModuleLevel() which according to the CPython source uses __package__ and __spec__ or __name__ and __path__ to find the parent package to import from. From here, I assume Blender doesn't set these globals to standard values which causes issues traversing the package heirarchy and this is the cause of one such issue in the import system.

Reloading

Reloading Python modules is usually not a common task but is crucial in Blender. Contrast from normal Python development, the Python environment in Blender stays for as long as the application is open. It takes an application restart to clean the state of the interpretter and update any add-on changes. Module reloading alone will only take you so far because of complications from nested child modules, Blender register() functions, and other necessities.

In the amount of I was developing on a single add-on for Blender, I found it useful to make my own importer that handles specific tasks I would otherwise type in manually. Below is a small class that will handle module loading and registering for you. See bl_register(self, moduleNames) and bl_unregister(self).

"""
Utility that allow a system of registering and unregistering of different modules
within a framework/add-on to be loaded into Blender

A module imported with this can have a bl_register and a bl_unregister at the
top level to add what it needs to Blender upon add-on load and unload

Note that this will handle recursively reloading of child modules, so module loops
will definitely cause headaches. Be wary of how you use this
"""

import importlib
import sys
from types import ModuleType

def rreload(module):
    '''Recursively reload modules.'''
    importlib.reload(module) 
    for childModule in [v for v in module.__dict__.values() if v is ModuleType]:
        rreload(childModule)

class BLModuleLoader:
    '''
    A module loader that plays nice with Blenders reloading system
    '''
    def __init__(self):
        self._registeredModules = {}

    def _register(self, name):
        '''
        Imports and registers a single module where name is the "."
        separated list for the package, absolute or relative
        '''
        importEquivalentStr = "import " + name
        try:
            if name in sys.modules:
                print("Reloading import \"" + importEquivalentStr + "\"")
                rreload(sys.modules[name])
            else:
                print("Importing for first time \"" + importEquivalentStr + "\"")
                sys.modules[name] = importlib.import_module(name)
            imported = sys.modules[name]
        except ImportError as e:
            print("Importing error " + name + ": " + str(e))
            return None
        
        if "bl_register" in dir(imported) and name not in self._registeredModules:
            try:
                imported.bl_register()
            except ValueError as e:
                print("Registration error " + name + ": " + str(e))
                return None
        self._registeredModules[name] = imported
        return imported

    def _unregister(self, name):
        '''
        Unregisters a previously registered module
        '''
        module = self._registeredModules[name]
        if "bl_unregister" in dir(module):
            try:
                module.bl_unregister()
            except ValueError as e:
                print("Unregistration error " + name + ": " + str(e))
                return None
        del self._registeredModules[name]

    def bl_register(self, moduleNames):
        '''
        Registers all the passed modules, returning a dict of all
        the loaded modules, key'd by their passed names
        '''
        return { k : self._register(k) for k in moduleNames }

    def bl_unregister(self):
        '''
        Unregisters all the modules that were previously imported
        '''
        for name in list(self._registeredModules.keys()):
            self._unregister(name)

This code is backed by 5 unit tests that handle loading and unloading modules, both nested and top level, inside of a Python environment run alongside headless Blender (though creating these unit tests to dynamically create and unload modules in Python was even itself a challenge).

Usage looks like as follows:

#__init__.py on top level of module
from BLModuleUtils import BLModuleLoader

#Create the loader
ml = BLModuleLoader
def register():
    #Register your modules
    loaded = ml.bl_register(["myModuleOnPythonPath",
        "myOtherModuleOnPythonPath"])
        
    #loaded holds the loaded modules if they worked
    #at the corresponding key
    #e.g. loaded["myModuleOnPythonPath"]
    #otherwise it will be None if an error occured
    #during the load and registration process

def unregister():
    #Unregister your modules
    ml.bl_unregister()
    
#myModuleOnPythonPath.py in the same folder, or wherever if you use a . path
def bl_register():
    bpy.utils.register_class(Your_RNA_Meta_Class)
def bl_unregister():
    bpy.utils.unregister_class(Your_RNA_Meta_Class)

Toggling and untoggling the checkbox will call unregister() and register() which will reload your modules and reregister all your new class bytecode. Just make sure you watch out for things that stick around like draw call handlers from bpy.types.SpaceView3D.draw_handler_add.

A lot of this was possible thanks to some great research from those around the Internet that have made similar plunges into Python's module loading system:

Read More »

Loading Screen Polish

Codingbrowserwebgl3dloadingUI

A loading screen inspired by Darkspider's Garry's Mod gamemode "Base Raiders"

Polish and small details are so important. With this loading screen, getting the text to not clip with the cube was a challenge but having it hover in screen space at a distance solves this and adds a nice subtle effect.

Read More »

Automated Memory Leak Testing in the Browser

Large single page web applications have a disadvantage over multi-page applications in that no page reload occurs. This means memory leaks are able to cause more performance problems in single page web applications as the lack of page reloads will not clear the javascript heap. While catching these leaks can be done by manually memory profiling, I will show you an automated test that will give a glimpse at whether memory leaks may be occuring.

The actual test is pretty simple though it requires Chrome to run. This is because Chrome comes with some nifty flags that let in browser tests do things other browsers can't.

  1. Chrome is the only browser that has the ability to read precise memory usage via javascript. This is enabled by the --enable-precise-memory-info flag.
  2. Chrome also is the only browser that allows you to trigger the garbage collector from javascript. This is enabled by the custom javascript flag --js-flags="--expose-gc"

Where I currently work, we use the Karma test runner for testing so setting up different browsers is as easy as installing an npm module and adding the browser to the build with the given flags.

After adding Chrome to your test runner, you will want to add the actual memory test. The memory test consists of a few parts.

  • A header that guards against browsers that don't support the features we need.
const isDef = (o)=>o !== undefined && o !== null;
if (!isDef(window.performance) || !isDef(window.performance.memory)) {
    console.log("Unsupported environment, window.performance.memory is unavailable");
    this.skip(); //Skips test (in Chai)
    return;
}
if (typeof window.gc !== "function") {
    console.log("Unsupported environment, window.gc is unavailable");
    this.skip();
    return;
}
  • A function for getting a memory profile
const getMemoryProfile = ()=>{ 
    window.gc(); //Trigger GC to get precise used memory reading
    return window.performance.memory.usedJSHeapSize; //Return used memory
};
  • An object that tracks memory profiles over time (optional though useful)
const profile = {
    samples: [],
    diffs: [],
    averageUsage: 0,
    averageChange: 0,
    //Collects a sample of memory and updates all the values in the
    //profile object
    sample() {
        const runningAverage = (arr, newVal, oldAvg)=>{
            return ((oldAvg * (arr.length-1) + newVal) / arr.length);
        };

        let newSample = getMemoryProfile();
        this.samples.push(newSample);
        this.averageUsage = runningAverage(this.samples, newSample, this.averageUsage);
        if(sampleLen >= 2) {
            let newDiff = this.samples[sampleLen - 1] - this.samples[sampleLen - 2];
            this.diffs.push(newDiff);
            this.averageChange = runningAverage(this.diffs, newDiff, this.averageChange);
        }
    }
};
  • The portion of your application you want to profile between calls to profile.sample(). In my case, this was the loading of a particularly heavy WebGL scene whereby a 50MB+ model was loaded in addition to many other operations where it would then be "disposed" of and all references to created objects broken (or we hope!).
profile.sample();
//Your test code here!
profile.sample();
  • The test assertions that are specific to the memory requirements of your application. Some sample assertions are given below (in this case using the chai expect framework)
const inMB = (n)=>n/1000000;

//Check average change in memory samples to not be over 10MB
expect(inMB(profile.averageChange).to.be.at.most(10);

//Check the final memory usage against the first usage, there should be little change if everything was properly deallocated
expect(inMB(getMemoryProfile())).to.be.at.most(profile.samples[0] + 0.25);

This test has saved me a couple of times though isn't bullet proof. Chrome and Firefox are very different beasts when it comes to memory management and, with specific "experimental" technologies like WebGL (come on, we already have WebGL2!), Firefox really likes to leak. This is the point where you start using the memory profiling tools native to your browser (Firefox profiler and Chrome Profiler). I prefer Chrome's toolset just because it gives you so much more control but Firefox's will probably get there with time.

Read More »

Compiling Blender as a Python Module for Windows 10 x64 using Visual Studio

If you want to do unit tests of Blender Python code, it might be to your benefit to not startup Blender every time you want to run them but to just import Blender as a Python module and run them from the command line. This is especially important/nice if you want to automate your tests. Note, before you jump in, if you just need mathutils you can get that separately here.

Luckily, Blender's build has a nifty feature by which you can compile it as a Python module and then import blender from Python...

import bpy #Starts up Blender as Python module
from mathutils import Vector #Import a Blender specific library

Unfortunately for me, all of the tutorials I found were not geared toward the target I was looking for: Blender 2.78, Windows 10 x64, Microsoft Visual Studio Express 2013+, and Python 3.5. After a couple days I was able to get it all building and working with this tool chain and it should work just as well for x86, a different Python version, or a different MSVS version.

The Build Process

  1. Install needed programs.

    • SVN: to check out the precompiled Windows dependencies
    • CMake: to make Blender
    • Microsoft Visual Studio >2013: For Blender compilation
    • Blender Source Code or Git: Blender source code as downloaded from the website or git to checkout the exact commit you want to build
    • Python installation of your desired version of the same bitness you want to build Blender in (Determining Python Bitness).
  2. Create a directory structure like the following

.
..
blender/                      #Blender source goes in here
build/                        #This is where you have CMake target
lib/win[dows,64]_vc[12,14]/   #Prebuilt binaries. The name matters! `windows` for 32 bit or `win64` for 64 bit, `vc12` for MSVS12 (2013) or `vc14` for MSVS14 (2015) and MSVS15 (2017)
  1. Checkout the precompiled Windows dependencies based on your bitness desired

  2. Download the source (or checkout the git repo) into the blender/ folder.

    • Blender's Git repos though you'll want to clone git://git.blender.org/blender.git specifically.
    • If you're using git make sure to checkout the specific commit you want and also git submodule update --init --recursive
  3. In build/, run cmake -DWITH_PLAYER=OFF -DWITH_PYTHON_INSTALL=OFF -DWITH_PYTHON_MODULE=ON -DPYTHON_VERSION=3.5 ../blender

    • For 64 bit, you might need to force CMake to use a specific generator. -G"Visual Studio 15 Win64" where 15 is your Visual Studio version
    • Replace the Python version number with your chosen Python version
    • There are other configurable options found in blender/CMakeLists.txt if you want to enable/disable other features (like the game engine)
  4. In the Visual Studio Developer Command Prompt, also in build/, run devenv Blender.sln /Build [TARGET] /Project INSTALL where [TARGET] is a Visual Studio release target. Most likely you'll want Release

    • You can see the other targets if you open the the .sln file.
  5. You now have the built files. Copy the files into the Python's global site-packages directory

copy bin\bpy.pyd C:\Python35\Lib\site-packages\
copy bin\*.dll C:\Python35\Lib\site-packages\
del C:\Python35\Lib\site-packages\python35.dll
xcopy /E bin\2.78 C:\Python35\2.78\

You can now open up python and test the module. Just open python on the command line and type import bpy and you now have access to Blender's Python modules. bpy.app will also give you useful information about the current build.

If you prefer to install it in a virtual environment, the commands above work just the same, though with the last one, make sure that 2.78 is copied into your virtual env Scripts folder and not the root

Troubleshooting:

  • Something with _Insert_n mentioning Eigen: Go into the mentioned .h (.hpp?) and change vector_base::_Insert_n to vector_base::insert in the corresponding .cpp file.
  • %1 is not a valid Win32 application: You have built the 32 bit Blender and tried running it from 64 bit Python. Rebuild with the correct bitness.
  • bpy: couldnt find 'scripts/modules', blender probably wont start.: You need to install the .pyd and related files in Python's site-packages, otherwise it cannot find the supporting files.

Thanks to all the other wonderful people who wrote tutorials that got me part of the way through this build!

Read More »

Projected Balance Calculator Pt Deux

I updated the balance projection calculator thingy a week or so ago but I thought I'd write about it. Here's a demo image:

Graph of > 1 year worth of financial transaction projections

Graph of > 1 year worth of financial transaction projections. The blue graph represents the balance, the orange represents balance with a savings percentage (in this case %30), and the green represents the total amount possible to spend without hitting 0.

New features include:

  • Filled graph with colors for extra spice
  • Multiple graphs including maximum possible expenditures and savings percentages
  • Labels to show exactly what charges come out when (using a given label when setting up the charges)

One of the issues I noticed after I generated this is that you need to generate a sufficient amount into the future to make sure that really far off charges aren't missed in the factors applied to the maximum possible expenditures graph.

As of now this solves my problem so I don't know if I'll need to add any more features. Perhaps if I ever have larger saving/investing goals.

Read More »

Projected Balance Calculator

I always felt that if I was able to see how much money I would have in the future I could more accurately save, spend, and make bill payments instead of just worrying all the time. Such a tool exists as a part of Quicken but Quicken is a slow, confusing piece of software that I bought and ended up returning because it was that bad(!). I looked but Mint.com doesn't have it, GnuCash doesn't have it, and doing it in Excel would require I learn VB, which, let's be honest, would probably result in me developing some serious medical condition.

I coded up the class model while I was at the barber shop in a Google Note and finished it in 3-ish hours at home. It honestly took more time to find the exact days that all my different services charge me on and put that in the program. It's pretty small and it needs a few more useful features before I can really use it but even in its current state I already feel comfortable where I am financially. Code can be found on GitHub.

Graph of projected balance over 90 days from today using a mixture of bogus and real data.

Graph of projected balance over 90 days from today using a mixture of bogus and real data.

The nicest part about this was I found libraries for both the plotting (Plotly) and for the reoccuring date problem (python-dateutil). My resultant program, including tests, was only about 100 lines long.

Anyway, it's been a pretty tough week and I don't think this will be my last endeavor with Python in the coming months. I want to revisit my music library and update that program so I guess we'll see how that goes.

Read More »

Movin' Day

TumblrMetaGhost

So I moved from Tumblr to Ghost. Tumblr (the platform, not the community) is absolute garbage. I actually like the community: tons of fandoms, great art, weird memes, and so many old memories. The problem with Tumblr is that the theme editing is absolute garbage. There's no real offline editing, they rewrite all your links, they inject tons of heavy scripts into the page load (upwards of 3+ seconds initial plus more async garbage), etc, etc.

I moved over just 6 posts as it's not easy picking apart their markup to actually put into a new blog. I also ported my theme and I have to say, being able to use Grunt, watch, livereload, and a local copy of Ghost has to be the best workflow I could ever want. If the desire so strikes me I might end up going back and reposting stuff but I guess we'll just have to wait and see.

It's good to be back, more or less. Hope to post some cool stuff.

Read More »
Older postsNewer posts