Use more values


Posted on 2021-11-12


While working with Django ORM very often .only() is used to speed up DB query. Unfortunately is also opens code to some nasty db queries related bugs. Let's for example take simple model:

1
2
3
4
5
class Foo(models.Model):

    field_a = models.CharField(max_length=16)
    field_b = models.CharField(max_length=50)
    field_c = models.BooleanField()

Everything looks good here. Now we have a developer that should implement listing of all the names of model Foo. As they are thinking about performance they use "only" to limit size of the query (Django ORM will query only for the mentioned field).

1
2
3
4
5
6
def some_view():
    ret = []
    foos = Foo.object.all().only("field_a")
    for foo in foos:
        ret.append({"field_name": foo.field_a})
    return ret

And everybody is happy. This view is making one query to db that returns only the required data.

But after few weeks new feature is added that need also field_b value. This task is being taken by some other developer. So they jump into code and modify the view to look like this:

1
2
3
4
5
6
def some_view():
    ret = []
    foos = Foo.object.all().only("field_a")
    for foo in foos:
        ret.append({"field_name": foo.field_a, "field_bool": foo.field_b})
    return ret

Code is working. Great work. But because new dev didn't notice only("field_a") or was not experienced enough to know what it does. But the code is working. Test were adjusted to get new return dicts. Things are well. Or are they? Because that simple oversight view that was generating one query to db is generating now n queries, where n is number of Foo instances. And that can get big pretty fast.

"But that should be caught by code review" you can say. Yes. You would be right, but unfortunately we are people. We make mistakes. It's very easy to miss that especially if the queryset is build way earlier in the code.

I think it's better to use values_list or values in such situations. Not only they are faster and take less memory (as instances of Foo model doesn't have to be created), but they are also protect you in a way from making such mistake.

For example this code:

1
2
3
4
5
6
def some_view():
    ret = []
    foos = Foo.object.all().values_list("field_a", "field_b", named=True)
    for foo in foos:
        ret.append({"field_name": foo.field_a, "field_bool": foo.field_b})
    return ret

will be faster and will also cause exception to be thrown in case somebody tries to access field_c on foo.

Leave a comment

PySeaweed update


Posted on 2016-06-12


I've updated my package PySeaweed (and changed it name from PyWeed).

Changes:

  • Overall code cleanup
  • Added option yo use reqests.Session() for better performance while dealing with huge ammount of files
  • Fixed some tests
  • Merge in one fork

PySeaweed on PyPi and Sources

Leave a comment

Getting package version from code without importing


Posted on 2015-04-07


I use this code for getting version info from package code without importing it, as it has to be installed to be imported and it can't be installed without version number. Classic 'Chicken or the egg' dilema.

I'm using ast module from Python stdlib to parse file from package code. In example I'm using __init__.py but one can use any other file.

It's works with module level variables (only string or number) and it's alternative to using eval() or exec().

setup.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
# coding=utf-8
import os
import ast
from setuptools import setup, find_packages


def get_info(filename):
    info = {}
    with open(filename) as _file:
        data = ast.parse(_file.read())
        for node in data.body:
            if type(node) != ast.Assign:
                continue
            if type(node.value) not in [ast.Str, ast.Num]:
                continue
            name = None
            for target in node.targets:
                name = target.id
            if type(node.value) == ast.Str:
                info[name] = node.value.s
            elif type(node.value) == ast.Num:
                info[name] = node.value.n
    return info

file_with_packageinfo = "packagename/__init__.py"
info = get_info(file_with_packageinfo)

here = os.path.abspath(os.path.dirname(__file__))
# README = open(os.path.join(here, 'README.txt')).read()
# CHANGES = open(os.path.join(here, 'CHANGES.txt')).read()


requires = [
]

setup(name='packagename',
      version=info.get('__version__', '0.0.0'),
      description='Package description',
      # long_description=README + '\n\n' + CHANGES,
      classifiers=[
          "Programming Language :: Python ",
      ],
      author=info.get('__author__', 'Łukasz Bołdys'),
      author_email='[email protected]',
      url='http://dev.utek.pl',
      keywords='',
      packages=find_packages(),
      include_package_data=True,
      zip_safe=False,
      test_suite='packagename',
      install_requires=requires,
      entry_points="""\
      [console_scripts]
          ddd = packagename.scripts.analyze:main
      """,
      )

packagename/__init__.py

1
2
3
4
# coding=utf-8

__version__ = "0.5.1"
__author__ = "Łukasz Bołdys"

Leave a comment

Extending setup.py install commands


Posted on 2015-03-11


Lately one of my associates asked if there is a way to use different install_requires lists in setup.py depending on way of installing.

If you are doing python setup.py install you would get one set of requires and doing python setup.pl develop would give you another set of requires (either extended or totally different).

Here's my solution:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
#!/usr/bin/env python
# coding=utf-8

from setuptools import setup, find_packages
from setuptools.command.develop import develop

requires = [
    'fabric',
    'pelican'
]


class ExtendedDevel(develop):
    ''' Adding ipython to requirements '''
    def run(self):
        requires.append('ipython')
        develop.run(self)


setup(name='lalala',
      version='0.0.1',
      description='That\'s stupid',
      long_description='DESCRIPTION',
      classifiers=[
      ],
      author='Łukasz Bołdys',
      author_email='[email protected]',
      url='',
      keywords='',
      license='LICENSE',
      packages=find_packages(),
      include_package_data=True,
      zip_safe=False,
      test_suite='test',
      install_requires=requires,
      cmdclass={'develop': ExtendedDevel}
      )

Using this code in setup.py will install fabric, pelican and ipython when doing python setup.py develop but will only install fabric and pelican if installing via python setup.py install.

If you don't want to change default behavior of develop you can add new command with changing cmdclass={'develop': ExtendedDevel} to cmdclass={'newcommand': ExtendedDevel}.

Edit 2015-04-22 10:47:20:

As I first needed method explained above to add some lib to development enviroment that was not needed in production enviroment. But there is better method to do it.

You can pass 'extras_require' argument to setup(). 'extras_require' is and dictionary of 'targets' as key and list of additional requirements as value. See documentation for more info.

Leave a comment

Remember when exporting/importing database on MS SQL


Posted on 2014-03-26


Just a little remainder (mostly to myself).

When moving MSSql database between two servers and using Import/Export tool from MSSql remember that there are no constraints or indices created.

Leave a comment

Fun with maps II


Posted on 2013-08-19


Last time I've created map with overlay that shows lightnings. Unfortunately standard OpenStreetMaps style is too bright for lightnings to be visible enough.

My solution was to add overlay layer that darken map to enhance visibility of Bliztordung.org layer. For this I've used OpenLayers.Layer.Image. This layer display provided image on the map (using provided bounds and size).

First. I created png image filled black, size 100x100px. Then created ImageLayer:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
var shade = new OpenLayers.Layer.Image(
    "Shade",
    "http://utek.pl/examples/fun-with-maps2/darken.png",
    new OpenLayers.Bounds(-20037508, -20037508, 20037508, 20037508.34),
    new OpenLayers.Size(100, 100), {
        isBaseLayer: false,
        visibility: true,
        opacity: 0.5
    }
);

Leave a comment

Fun with maps


Posted on 2013-08-13


While working on my geo enabled appliction I wondered if one could use data from Blitzortung.org to display map of lightings.

I'm using OpenLayers to display map using data from OpenStreetMap. So to display data from Blitzortung.org I needed to create new layer. But instead of using OpenLayers.Layer.OSM I'm using OpenLayers.Layer.XYZ.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
var lightings_15 = new OpenLayers.Layer.XYZ(
    "0-15 minutes", [
        "http://images.lightningmaps.org/blitzortung/europe/index.php?tile&zoom=${z}&x=${x}&y=${y}&type=0"
    ], {
        attribution: "Lightning data from <a href='http://blitzortung.org'>Blitzortung.org</a>",
        tileSize: new OpenLayers.Size(1024, 1024),
        transitionEffect: "resize",
        isBaseLayer: false,
        visibility: true,
        sphericalMercator: true
    }
);

To "refresh" lightings map I created small function that replace url in lightings_15 layer and force redraw on that layer.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
var url = "http://images.lightningmaps.org/blitzortung/europe/index.php?tile&zoom=${z}&x=${x}&y=${y}&type={type}&bo_t={date_str}";

function updateLayer() {
    var date = new Date();
    date_str = date.getUTCDate() + "_" + date.getUTCHours() + "_" + date.getUTCMinutes();
    new_url = url.replace("{type}", i);
    new_url = new_url.replace("{date_str}", date_str);
    lightings_15.url[0] = new_url; // As we have only one url in layer XYZ
    if (lightings_15.visibility){
        lightings_15.redraw({
            force: true
         });
    }
};

Finally after page load I'm setting interval using setInverval and defined updateLayer function

1
setInterval(updateLayer, 5*60000);

Check example

Todo:

Leave a comment

Ignoring tables in Alembic


Posted on 2013-08-12


While working on spatial enabled application I came up to a problem with spatial table in my postgres database (spatial_ref_sys). Alembic insisted on deleting this table as it wasn't declared in my models.py.

I didn't wanted to define it just to keep Alembic from removing it so I've added following changes to .ini and env.py files:

development.ini

1
2
[alembic:exclude]
tables = spatial_ref_sys

env.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
def exclude_tables_from_config(config_):
    tables_ = config_.get("tables", None)
    if tables_ is not None:
        tables = tables_.split(",")
    return tables

exclude_tables = exclude_tables_from_config(config.get_section('alembic:exclude'))

def include_object(object, name, type_, reflected, compare_to):
    if type_ == "table" and name in exclude_tables:
        return False
    else:
        return True

def run_migrations_online():
    if isinstance(engine, Engine):
        connection = engine.connect()
    else:
        raise Exception('Expected engine instance got %s instead' % type(engine))

    context.configure(
        connection=connection,
        target_metadata=target_metadata,
        include_object=include_object
    )

    try:
        with context.begin_transaction():
            context.run_migrations()
    finally:
        connection.close()

Leave a comment

WebPy middleware authorization


Posted on 2012-11-12


middleware.py

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
class AuthApp(SimpleHTTPRequestHandler):
    def __init__(self, environ, start_response):
        self.headers = []
        self.environ = environ
        self.start_response = start_response

    def send_response(self, status, msg=""):
        self.status = str(status) + " " + msg

    def send_header(self, name, value):
        self.headers.append((name, value))

    def end_headers(self):
        pass

    def log_message(*a): pass

    def __iter__(self):
        environ = self.environ
        self.send_header('WWW-Authenticate','Basic realm="App authorization"')
        self.send_response(401, "Unauthorized")
        self.start_response(self.status, self.headers)
        yield "Unauthorized"

    AUTH = settings.AUTH

    class AuthMiddleware:
        """WSGI Middleware for authentication."""
        def __init__(self, app):
            self.app = app
        def __call__(self, environ, start_response):
            #allowed = [("test", "test")]
            allowed = AUTH
            auth = environ.get('HTTP_AUTHORIZATION')
            authreq = False
            if auth is None:
                authreq = True
            else:
                auth = re.sub('^Basic ','',auth)
                username,password = base64.decodestring(auth).split(':')
                if (username,password) in allowed:
                    return self.app(environ, start_response)
                else:
                    authreq = True
            if authreq:
                return AuthApp(environ, start_response)

In your main webpy project code

1
2
3
4
5
6
7
8
app = web.application(urls, locals())
app.add_processor(load_sqla)

application = app.wsgifunc(AuthMiddleware) # for WSGI

if __name__ == "__main__":
    pass
    app.run(AuthMiddleware) # for direct execution

Leave a comment

Categories

Tags

Links