true false maybe

So you want to find out why your Pylons app is running slowly? Well most likely it has to do with your SQL queries, and the best way to see what's going on and how long each request is taking is to install Dozer (by benbangert of Pylons), and load it up with a TimerProxy (by zzzeek of SQLAlchemy).

Sound like fun? Well, here's how to do it.

Install Dozer:

sudo easy_install -U http://www.bitbucket.org/bbangert/dozer/get/b748d3e1cc87.gz

Add this to your middleware:

# Add this to your middleware.py, right before return app
    if asbool(config['debug']):
        from dozer import Logview
        app = Logview(app, config)

Add this to your development.ini

# Add to development.ini
logview.sqlalchemy = #faa
logview.pylons.templating = #bfb

(you can customize the colors here)

Next, modify your configuration ini as well as you like to configure what shows up in the log. Note that I have root set to INFO which will squelch a lot of messages. Change this to DEBUG to see more of what's going on in each request.

# Logging configuration
[loggers]
keys = root, YOURPROJ

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = INFO
handlers = console

[logger_YOURPROJ]
level = DEBUG
handlers =
qualname = YOURPROJ.lib

[logger_sqlalchemy]
level = INFO
handlers =
qualname = sqlalchemy.engine

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(asctime)s,%(msecs)03d %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S

Add this file to /lib/

querytimer.py

from sqlalchemy.interfaces import ConnectionProxy
import time
 
import logging
log = logging.getLogger(__name__)
 
class TimerProxy(ConnectionProxy):
    def cursor_execute(self, execute, cursor, statement, parameters, context, executemany):
        now = time.time()
        try:
            return execute(cursor, statement, parameters, context)
        finally:
            total = time.time() - now
            log.debug("Query: %s" % statement)
            log.debug("Total Time: %f" % total)
 

Okay, one last thing, modify your SQLAlchemy engine in environment.py to this:

engine = engine_from_config(config, 'sqlalchemy.', proxy=TimerProxy())

and add an import at the top:

from YOURPROJ.lib.querytimer import TimerProxy

So that's it! Restart paster, and load up a request in your web browser. There will now be a bar at the top that you can click on and see all the requests.

If you want to run TimerProxy on it's own (that is without Pylons and Dozer, see zzzeek's post on "Timing All Queries".

This is a stitched image from Burning Man Earth of Burning Man 2008. Unlike my last post, this one isn't straight down, as I wanted to create a dramatic view of the city, and is taken on a different day. At some point we'll have a timelapse from above. How cool is that?

If anyone is interested in wallpapers, let me know.

This is one preview from the many aerial shots of Burning Man from this year, 2008. More to come.

This is the work of Jeffrey of Pict'Earth. The Burning Man Org also contributed to help this project happen. It is part of the Burning Man Earth project (not associated with Google).

From http://abstrusegoose.com/73.

At the most basic level, a graph can communicate key information, like a company's P&L, the median price of a house, or daily average rainfall for a county over a year, but when used effectively, a graph can tell a story.

Much like the rings on a tree can tell you the story of their lives, this graph from the NYTimes, tells the all too true story that we're living right now, every one of us -- the economy. How it varies from the LA Times dead tree edition though is dramatic because it interactively allows you to explore how this bear market relates to the bear markets of our parents, grandparents, and great grandparents.

The NYTimes effectively use a timeline that slides to specific bear markets, making it incredibly easy to see the differences between today and the history that led up to this crisis. Having closely watched the markets, this paints an entirely new picture, however grim it may be.

This is a quickstart guide for installing the OSX Flex SDK on your Mac. I adapted this from a video tutorial at learnhub. If you prefer to watch videos, go check it out. This is just a more straight forward guide to getting you up and running.

1. Download the SDK from http://www.adobe.com/products/flex/flexdownloads/

(click on the "I have read the Adobe Flex 3 SDK License" checkbox, and the download link will appear below)

2. Use stuffit expander / unzip to uncompress the archive.

3. Open the terminal (if not open already), and go to the folder where you uncompressed the archive to.

4. Type the following:

sudo cp -r flex_sdk_3/ /Developer/SDKs/

5. Edit your ~/.bash_login to include the following line:

export PATH="/Developer/SDKs/flex_sdk_3/bin:$PATH"/

This will add the binaries to the path. If this file doesn't exist yet, go ahead and create one. Next, create a new terminal window to have the right binaries in your path.

6. Try it out. Create a file named "helloworld.mxml" and paste the following into it.

<?xml version="1.0" encoding="utf-8"?>
 
<mx:Application
    xmlns:mx="http://www.adobe.com/2006/mxml"
    viewSourceURL="src/HelloWorld/index.html"
    horizontalAlign="center" verticalAlign="middle"
    width="300" height="160"
>
    <mx:Panel
        paddingTop="10" paddingBottom="10"
        paddingLeft="10" paddingRight="10"
        title="My Application"
    >
 
        <mx:Label text="Hello World!"
           fontWeight="bold" fontSize="24"/>
    </mx:Panel>
</mx:Application>

Make sure there is no whitespace before the XML declaration (the

7. Try it out. Go back to your terminal and type (from the directory where helloworld.mxml is) type the following:

mxmlc helloworld.mxml

This will compile the file using the Flex 3 SDK. It should create a file in the same directory called "helloworld.swf".

8. Open up helloworld.swf in Firefox. If you have Flash 9 installed, it will show you your compiled Flex application!

---
Update: This is only true for Python before version 2.5. Otherwise, ternary operators are perfectly valid. Thank you Janzert / Danmc for jumping on top of my post and correcting me. See the comments for their examples and why not to do type(foo) == type(42) (spoiler: ducks don't like typing)
---

No ternary? This surprises me. I wanted to make my code a tiny bit cleaner, and use a ternary operator, as I (somewhat) recently did with Flex. Python apparently doesn't allow this kind of thing. Instead, one website suggested I do the following:

val = float(raw_input("Age: "))
status = ("working","retired")[val>65]
print "You should be",status

--wellho.net

Unfortunately this only works with simple examples that don't have a problem being evaluated prior to the val>65 test. Sure it looks pretty, but it doesn't work if you substitute it with this example:

def int_to_float(v):
    return (0.0, float(v))[type(v) == type(42)]

Unfortunately in this situation, if you supplied int_to_float with "a string "instead of a number, it would error out with a ValueError because it would try to evaluate int("a string") before looking to see if "a string" is of type int or not.

My solution? Not as pretty as I'd hoped:

def int_to_float(v):
    if type(v) != type(42):
        v = 0.0
    return float(v)

Better code always appreciated. Internal progress is perfection.

Migrations are useful for emerging projects because they allow developers to evolve their models, and incrementally define schema change scripts so that old database records can be upgraded and downgraded easily.

When I started using Pylons, I set up a tinderbox to allow a 2nd copy to live for our team to play with the system and create data structures. In order to allow this data to live between major model changes, this meant I had to find a way to manage migrations. Using "sqlalchemy-migrate", I was able to achieve just that. Here's my quickstart on how to get up and running with sqlalchemy migrations under pylons. Now whenever I update my tinderbox (it's not automatic quite yet), I just run "svn up; python dbmanage.py upgrade".

Quickstart Guide:

1. Install sqlalchemy-migrate

sudo easy_install sqlalchemy-migrate

2. Go to your project directory

cd myapp

3. Create your sqlalchemy-migrate repository in your project

migrate create myapp/migrations/ "MyApp"

This will create the directory "migrations" in your myapp/myapp/ directory.

4. Add the migrate tables to your database:

migrate version_control mysql://user:pass@myhost:3306/database myapp

5. See what version the repository is at:

migrate version banyan/migrations

4. Create the dbmanage script. This is mostly a configuration script to make sa-migrate easier to use.

migrate manage dbmanage.py --repository=myapp/migrations/ --url=mysql://user:pass@myhost:3306/database

5. Create the first migration script

migrate script "Add initial tables"

Other documentation says to do 'migrate script script.py', but this didn't work for me properly. What this will do is create

6. Edit your first migration script.

Open up the file in your favorite editor, and change it to meet your needs. In my case, it looks like this:

import time, datetime
from sqlalchemy import *
from migrate import *
 
meta = MetaData(migrate_engine)
 
users_table = Table('users', meta,
                    Column('id', Integer, primary_key=True),
                    Column('email_address', Unicode(255), unique=True),
                    Column('display_name', Unicode(255)),
                    Column('password', Unicode(40)),
                    Column('created', DateTime, default=datetime.datetime.now)
                    )
 
def upgrade():
    # Upgrade operations
    users_table.create()
 
def downgrade():
    # Operations to reverse the above upgrade
    users_table.drop()

7. Test your new script.

python dbmanage.py test

8. "Commit" your changes.

python dbmanage.py upgrade

Note, the documentation I read said to use the command "commit" instead of "upgrade". I have no idea why this is, the dbmanage script doesn't know what to do when you ask it to commit. When you upgrade though, it will run the changes to your database, and change the migrations version from 0 to 1.

9. Rinse and repeat.
Do steps 5 through 8 to whenever you have new database changes you want to add to your database.

Here's another example to show an alter. The module migrate.changeset allows you to alters that SQLAlchemy at this time does not.

1. Create another revision script.
Lets say we wanted to change the column display_name to first_name, and add a last_name column.

python dbmanage script "Change display_name column to real names"

2. Edit that file.

My one looks like this:

import time, datetime
from sqlalchemy import *
from migrate import *
import migrate.changeset
 
meta = MetaData(migrate_engine)
 
users_table_new = Table('users', meta,
                    Column('id', Integer, primary_key=True),
                    Column('email_address', Unicode(255), unique=True),
                    Column('first_name', Unicode(255)),
                    Column('last_name', Unicode(255)),
                    Column('password', Unicode(40)),
                    Column('created', DateTime, default=datetime.datetime.now)
                    )
users_table_old = Table('users', meta,
                    Column('id', Integer, primary_key=True),
                    Column('email_address', Unicode(255), unique=True),
                    Column('display_name', Unicode(255)),
                    Column('password', Unicode(40)),
                    Column('created', DateTime, default=datetime.datetime.now),
                    useexisting=True
                    )
 
def upgrade():
    # This assumes we're moving all existing accounts to have their first_name
    # be their previous display_name
    users_table_old.c.display_name.alter(name="first_name")
    # Note the above users_table_new has a new last_name column already defined in
    # it. This column has not been created yet in the database, so we will do that here:
    users_table_new.c.last_name.create()
 
def downgrade():
    # Again, for every upgrade, we have to have a downgrade
    users_table_new.c.first_name.alter(name="display_name")
    # Any downgrade will result in loss of last_name data with this script.
    users_table_new.c.last_name.drop()

3. Test the changes (again).

python dbmanage test

Everything good?

4. "Commit" the changes (yes, again):

python dbmanage upgrade

Awesome! If you have anything to add or catch any mistakes, please drop me a comment.

I'm putting together a costume for Burning Man. It's all around the theme of the American Dream, gone horribly horribly wrong. The truth of the world today is just as sinister, just not so visually repulsive.

Here are the main pieces of the costume. I have yet to get them, so I haven't been able to assemble it yet. They will be modified somewhat to look more "official", possibly incorporating a government looking seal.

EyeClops Nightvision Mask ($80)

Respirator, Coveralls, Chemical Apron, and 28" Chemical Gloves, all totaling approximately $65.

These parts combined with a pair of black boots will create a look that will cover up the entire body, leaving no sense of humanity beyond the humanoid shape, much like HalfLife 2's Combine Soldiers. Combined with some audio on loop possibly similar to 1984 - War is Peace, Freedom is Slavery, Ignorance is Strength, or possibly more aptly, incorrectly telling people they do not have the right to take photographs and to report suspicious behavior immediately.

Maybe this isn't the Burning Man many know and love. It's dark. It's dystopian. It is performance art designed to provoke thought, not laughter or smiles. Is it my dream? No, but I am tired of the security theater and crumbling of civil liberties that is the reality of today's world.

The OpenView Project is a open source, creative commons project for building and taking interactive panoramas much like Google Street View, but of more interesting places than streets, such as farmers markets, concerts, art festivals, and hiking trails, just to name a few. I'm blogging my work on it at OpenView Project (openviewproject.org). I'm very thrilled to be working with photography again, and excited about the prospect of giving people a new pespective of the world around them.