Unity on Linux and !CreateDirectoryRecursive(fullpath)

Having installed the latest Unity 5.1.0f3 (game development software/framework) on Kubuntu, every time I would try to create or open a project I could get an error dialog that said simply:

!CreateDirectoryRecursive(fullpath)

And there were two buttons: one said "Quit", and did nothing.  The other said nothing, and did nothing. To exit Unity I'd have to kill or ctrl-c if I started it from command line.

This went on for quite some time. I even tried re-installing, and deleting every configuration file I could find. I searched the internet, but the only one similar report I found on this error was not helpful.

So finally I straced the thing and found the problem... ~/.cache/unity3d was for some reason owned by root.

Simple, but amazingly frustrating to finally track down.

(I also installed unity on Arch linux using AUR, and it worked fine with no such complications.)

Django issues

I continue to use Django, because it is the blessed python framework, and because I really want to take advantage of the ecosystem and not have to re-implement things such as social authentication. But, wow, it's frustrating a lot of the time. It feels like a toy. And half of the things that save one time, one ends up paying for (and then some) somewhere down the line if one tries to do something slightly off the beaten path.

Take for example, a slightly complex SQL statement. Really, just joining an aggregate count from a couple of tables to another few tables: in short, it requires a simple subselect. There's no way the fairly pathetic Django ORM will do this. So there are some choices to be made.

• Use a database view 
• Django however provides no support for creating views with syncdb
• South, the essential third party tool for DB migration because apparently Django devs don't think it's essential enough to include, also has no support for views
• One has to create a "dummy" (non-managed) model just to run the ORM against the view
• Use a "raw" SQL query
• One loses most of the ORM's conveniences 
• If one needs to make the SQL dynamic (such as for sorting or filtering), one finds oneself basically rebuilding a mini-ORM or at least SQL generator for oneself, and eventually question why bother with django's ORM at all... since it's mostly only good for CRUD, which is not a difficult problem.
• Then there's the issue of where to do the raw query. Embedded raw SQL in the view? Create Manager object and patch it into the model just to do one query? Apparently this latter is the suggested method, for the sake of code separation. It's pretty ugly no matter how you do it.
• Have to explicitely name tables in query -- potential refactoring headaches.
• Abuse the .extra() method to hack in a subselect
• This used to be possible, but no longer is possible due to all the "select" elements being quoted.
• One can monkeypatch the quote routine not to do quotes, but... monkeypatch!
• Do the join in software
• Now you require extra database hits for extra queries
• Possible complications if the dataset is large and needs filtering
• The data join can not be done at the template level (pass in data and dictionary to join) because django templates do not support looking up dynamic values (ie. a dynamic dictionary key held in another variable)
• This problem can be solved by creating a custom filter which takes the variable as an argument and does the lookup for you; but now you have to manually include your registered filter in every template you use it in. Really?? This basic functionality of a basic python lookup isn't included?
• Unholy hack of ORM generated SQL (in order to take advantage of programatic filter/sort), with the subselect crudely hacked in, and then run through as a "raw" query to get the results. 
• Actually augment the ORM with some hacks (or sane extensions) that allow a little more flexibility in the SQL generation. The django-orm-extensions should be applauded for attempting a few specific cases of this, but in my quick look at the docs I find the integration provided (SQLExpression, etc), again, extremely awkward.
• Use Aldjemy to integrate Alchemy for robust SQL query building purposes. Aldjemy looks like a really convenient hack to get SQLAlchemy support into Django... but do I really want more dependencies and two different ORM syntaxes in one application? (There is also an older django-sabridge project along these lines.)

It's all kind of disheartening and crazy to have to even think about this stuff. It makes one believe that most people don't do anything serious with django beyond glorified key-value lookups. Or djangos are masochistic contortionists. But I may be in a bit of a negative frame of mind just now.

Fixing another /var/lib/dpkg/status corruption

This has happened a few times on my debian/ubuntu/apt based system, sending me searching for answers. I always end up hacking the file to make it work, which makes me a bit uneasy, but it is in fact apparently the only choice in many of these situations. This "database" is just a plain text file.

Sometimes the problem is obvious, but often it is not. For example, here is my latest error message:

dpkg-query: error: parsing file '/var/lib/dpkg/status' near line 7848 package 'fontconfig': field name `

There is no ` character anywhere near line 7848, field name or otherwise.

The easiest solution to this, and similar, issues is to remove the whole section belonging to the package named. So in this case I remove everything starting at the line containing

Package: fontconfig

down to, and including, the next blank line.

But wow apt/dpkg no longer no longer knows the status of your fontconfig package. The easiest way to solve that is just re-install it. (sudo apt-get install fontconfig)

Make a backup of the status file first, if you're nervous.

Inside a Sony PRS-T1 Reader

Having had passed to me a broken PRS-T1, I decided to rip it apart. Below is what it looks like, for the curious. Note the battery (which has been disconnected in the pictures) is soldered to the board, so not very easily replacable/reusable.

Here is the PRS-T1 service manual with instructions on how to open the thing up. Basically it has (pretty tight) clips along the inside sides.

(Right click and save, or open in a new tab/window, if you want full resolution to zoom in on.)

Join / Merge / Combine / Concatenate PDF pages into a single PDF

There are many free (and open source) methods available.

pdfjoin / pdfjam - command line utility. Very easy to use, but has some limitations:

• Output pages are all the same size
• Hyperlinks are stripped

Use ghostscript to "print" all of the files into a new PDF file. With this method, you can use other ghostscript capabilities to further modify the output (such s resampling graphics, etc):

•  gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=output.pdf input1.pdf input2.pdf

pdftk (PDF Toolkit) - yet more command line tools for PDF manipulation. There are a bunch of different styles of command line (examples can be seen on web site). The simplest is:

• pdftk input1.pdf input2.pdf input3.pdf cat output output.pdf
• powerful but complicated GUI to PDFtk is available 

pyPDF - a tool for the programming / python oriented. It can split, merge, watermark, rotate, and extract information with simple python scripting. There is a GTK-based GUI available called PDF-Shuffler.

PDF Split and Merge is a cross-platform java based GUI (and command line) tool.

There are more! With so many choices, how does one choose? Whim, personal preferences, scale requirements, tools and platform on hand (ease of installation/integration), or just random.

Resampling jpegs inside a PDF (to shrink/reduce file size)

My wife is applying for various teaching jobs. Most of the positions require uploading documents to various school board applicant tracking systems. I carefully prepared the documents for her, also wanting to keep them for historical archival purposes. The uploads produce a bizarre error. Contacting the site administrator, we find out that there is a 1 megabyte file size limit on uploads and this is what produces the cryptic error. Sigh. (To add insult to injury, after resampling the scanned PDFs, it then turns out there's also crazy filename limitations... so all the nice descriptively  named files had to be renamed to eliminate various characters, and maintain a certain length). These are the gatekeepers of our children's education.

But I digress.

This info is a lot of places on the net already, but I want to add it here for my own reference as I've forgotten the simple command line a few times already and had to re-search it. 

In short, use ghostscript:

gs -sDEVICE=pdfwrite -dPDFSETTINGS=/ebook -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf

The PDFSETTINGS presets equate to the following resolutions:

• /screen ... screen-view-only quality, 72 dpi images 
• /ebook ... low quality, 150 dpi images
• /printer ... high quality, 300 dpi images 
• /prepress ... high quality, color preserving, 300 dpi images
• /default ... similar to /screen

Other suggested command line arguments made various places:

• -dCompatibilityLevel=1.4 
• -dColorImageResolution=38 -dColorImageDownsampleType=/Average -dGrayImageDownsampleType=/Average -dGrayImageResolution=38 -dMonoImageResolution=38 -dMonoImageDownsampleType=/Average -dOptimize=true -dDownsampleColorImages=true -dDownsampleGrayImages=true -dDownsampleMonoImages=true -dUseCIEColor -dColorConversionStrategy=/sRGB
• -dMaxSubsetPct=100

This page is quite good for further tips (such as recompressing with lossless compression).

Force java to use IPV4

Having problems with a java server application listening on IPV6. Adding the following to the java command line causes Java to use IPV4.

-Djava.net.preferIPv4Stack=true

 It's come in handy a few times. Sadly.