Inside the Head of Pydanny

Hi, I'm Daniel Roy Greenfeld, and welcome to my blog. I write about Python, Django, and much more.
Two Scoops of Django 1.11 is Out!

Two Scoops of Django

When we started the Two Scoops of Django project back in 2012, I never thought it would become a book series. Well, it's turned into just that, and so I'm pleased to announce the "Early Release" or "BETA" release of the Two Scoops of Django: Best Practices for Django 1.11 PDF ebook.

Co-authored with Audrey Roy Greenfeld, the 1.11 edition of Two Scoops of Django is filled to the brim with knowledge to help make Django projects better. We introduce various tips, tricks, patterns, code snippets, and techniques that we've picked up over the years. What we didn't know or weren't certain about, once again we found the best experts in the world and asked them for the answers. Then we packed the result into a 530+ page book.

What's Next?

  • We'll be adding more material to the 1.11 edition in the near future, hence the term, "Early Release". Everyone who buys the 1.11 ebook from us gets all 1.11 ebook updates.
  • Once we're happy with the ebook, we'll release a print paperback edition, scheduled for May or June.
  • We're selling the book in PDF format, as well as taking pre-orders for the print edition.


You can purchase the "Early Release" Two Scoops of Django: Best Practices for Django 1.11 PDF ebook at the Two Scoops Press store.

Python F-Strings Are Fun!

Python F-Strings Are Fun!

In python 3.6 we saw the adoption of Literal String Interpolation, or as they are known more commonly, f-strings. At first I was hesitant because... well... we've got multiple string tools already available:

one, two = 1, 2
_format = '{},{}'.format(one, two)
_percent = '%s,%s' % (one, two)
_concatenation = str(one) + ',' + str(two)
_join = ','.join((str(one),str(two)))
assert _format == _percent == _concatenation == _join

Adding f-strings to this mix didn't seem all that useful:

_fstring = f'{one},{two}'
assert _fstring == _format == _percent == _concatenation == _join

I was doubtful, but then I tried out f-strings on a non-trivial example. Now I'm hooked. Be it on local utility scripts or production code, I now instinctively gravitate toward their usage. In fact, f-strings are so useful that going back to earlier versions of Python now feels cumbersome.

The reason why I feel this way is that f-strings are concise but easy to understand. Thanks to intuitive expression evaluation I can compress more verbose commands into smaller lines of code that are more legible. Take a look:

_fstring = f'Total: {one + two}'  # Go f-string!
_format = 'Total: {}'.format(one + two)
_percent = 'Total: %s' % (one + two)
_concatenation = 'Total: ' + str(one + two)
assert _fstring == _format == _percent == _concatenation

The f-string example is four characters shorter than the closest alternative and is extremely easy to read. Indeed, put the f-string example in front of a non-programmer and they'll understand it fast. The same won't apply to the alternatives, odds are they'll ask what .format(), str(), and the % mean.

F-Strings Are Addictive

The conciseness and power of the intuitive expression evaluation can't be understated. On the surface f-strings seem like a small step forward for Python, but once I started using them I realized they were a huge step in codability for the language.

Now I'm hooked. I'm addicted to f-strings. When I step back to Python 3.5 or lower I feel like less of a Python coder. Yes, I have a problem with how much I lean on f-strings now, but I acknowledge my problem. I would go to therapy for it, but I believe I can manage the addiction for now.

Okay, enough joking, f-strings are awesome. Try them out.

A Utility Script Example

We just released Two Scoops of Django 1.11, which is written in LaTeX. Like most programming books we provide code examples in a repo for our readers. However, as we completey revised the code-highlighting, we had to rewrite our code extractor from the ground up. In a flurry of cowboy coding, I did so in thirty minutes using Python 3.6 while leaning on f-strings:

"""Two Scoops of Django 1.11 Code Extractor"""
import os
import shutil
from glob import glob

    print('Removed old code directory')
except FileNotFoundError:
print('Created new code directory')

STAR = '*'


    '\\begin{python}': '.py',
    '\\begin{badpython}': '.py',
    '\\begin{django}': '.html',
    '\\begin{baddjango}': '.html',
    '\\begin{plaintext}': '.txt',
    '\\begin{badplaintext}': '.txt',
    '\\begin{sql}': '.sql',
    '\\begin{makefile}': '',
    '\\begin{json}': '.json',
    '\\begin{bash}': '.txt',
    '\\begin{xml}': '.html',

LANGUAGE_END = {x.replace('begin', 'end'):y for x,y in LANGUAGE_START.items()}

def is_example(line, SWITCH):
    for key in SWITCH:
        if line.strip().startswith(key):
            return SWITCH[key]
    return None

def makefilename(chapter_num, in_example):
    return f'code/chapter_{chapter_num}_example_{str(example_num).zfill(2)}{in_example}'

if __name__ == '__main__':

    in_example = False
    starting = False
    for path in glob('chapters/*.tex'):
            chapter_num = int(path[9:11])
            chapter_num = path[9:11]
        except ValueError:
            if not path.lower().startswith('appendix'):
        example_num = 1
        with open(path) as f:
            lines = (x for x in f.readlines())
        for line in lines:
            if starting:
                # Crazy long string interpolation that should probably
                # be broken up but remains because it's easy for me to read
                filename =  f'code/chapter_{chapter_num}_example_{str(example_num).zfill(2)}{in_example}'
                dafile = open(filename, 'w')
                if in_example in ('.py', '.html'):
            if not in_example:
                mime = None
                in_example = is_example(line, LANGUAGE_START)
                if in_example:
                    starting = True
            mime = is_example(line, LANGUAGE_END)
            starting = False
            if mime:
                in_example = False
                example_num += 1

Using Python and Google Docs to Build Books

Python F-Strings Are Fun!

When I started my latest fiction book, The Darkest Autumn, I wrote out the chapters as individual files. I did it in a text editor (Sublime) and saved the files to a git repo. The names of the files determined their order, chapters being named in this pattern:

the-darkest-autumn $ tree

As the book developed I thought about moving it to Scrivener. If you don't know, Scrivener is an excellent tool for writing. It encourages you to break up your work into chapters and scenes. The downside is that Scrivener is complex (I want to write, not figure out software) and Scrivener isn't designed for simultaneous collaboration. The latter issue is a very serious problem, as I like to have others review and comment on my writing as I go.

What I really wanted to do is combine the chapter breaking of Scrivener with the simplicity and collaboration of Google Docs. Preferably, I would put the book chapters into Google Docs as individual files and then send invites to my editor, wife, and my beta readers. By using Google Docs I could ensure anyone could access the work without having to create a new account and learn an unfamiliar system.

Unfortunately, at this time Google Docs has no way to combine multiple Google Docs contained in one directory into one large document for publication. To use Google Docs thhe way I want involves manually copy/pasting content from dozens of files into one master document any time you want to update a work. With even 5 to 10 documents this is time consuming and error prone (for me) to the point of being unusable. This is a problem because my fiction books have anywhere from 30 to 50 chapters.

Fortunately for me, I know how to code. By using the Python programming language, I can automate the process of combining the Google Docs into one master file which can be converted to epub, mobi (kindle), or PDF.

How I Combine Google Doc Files

First, I download all the files in the book's Google Docs directory.

Selecting Files With Google Docs

This generates and downloads a zip file called something like I use unzip to open it:

unzip -d the-darkest-autumn

Inside the new the-darkest-autumn folder are a bunch of MS Word-formatted files named identically to what's stored on Google Docs:

$ tree the-darkest-autumn/
├── 01. Beginnings.docx
├── 02. Town of Ravenna.docx
├── 03. Walls of Ravenna.docx
├── 04. Gatehouse of Ravenna.docx
├── 05. Courage.docx
├── 06. To the Upper Valley.docx
├── _afterward.docx
├── _copyright.docx
├── _dedication.docx
└── _title.docx

Now it's time to bring in the code. By leveraging the python-docx library, I combine all the Word files into one large Word files using this Python (3.6 or higher) script:

""" Combines multiple Word docs in a folder.


import os
import sys
from glob import glob

    from docx import Document
    from docx.enum.text import WD_ALIGN_PARAGRAPH
    from docx.enum.text import WD_COLOR_INDEX
    from docx.shared import Inches, Pt
except ImportError:
    raise ImportError("You need to 'pip install python-docx'")

TEXT_FONT = "Trebuchet MS"

def add_matter(master_document, filename, chapter, after=False):
    """Builds """
    if not os.path.exists(filename):
        return master_document

    if after:

    # Build the heading
    heading = master_document.add_heading('', level=1)
    heading.alignment = WD_ALIGN_PARAGRAPH.CENTER
    runt = heading.add_run(chapter)
    runt.font.color.theme_color = WD_COLOR_INDEX.WHITE

    # Add the material
    document = Document(docx=filename)
    for index, paragraph in enumerate(document.paragraphs):
        new_paragraph = master_document.add_paragraph()
        new_paragraph.paragraph_format.alignment = paragraph.paragraph_format.alignment =
        # Loop through the runs of a paragraph
        # A run is a style element within a paragraph (i.e. bold)
        for j, run in enumerate(paragraph.runs):
            # Copy over the old style
            text = run.text
            # Add run to new paragraph
            new_run = new_paragraph.add_run(text=text)
            # Update styles for run
            new_run.bold = run.bold
            new_run.italic = run.italic
            new_run.font.size = run.font.size
            new_run.font.color.theme_color = WD_COLOR_INDEX.BLACK
    print(f'Adding {chapter}')
    return master_document

def add_chapter(master_document, filename, chapter):
    """Build chapters, i.e. where the story happens."""
    # Build the chapter
    document = Document(docx=filename)

    # Build the heading
    heading = master_document.add_heading('', level=1)
    heading.alignment = WD_ALIGN_PARAGRAPH.CENTER

    heading.add_run(chapter).font.color.theme_color = WD_COLOR_INDEX.BLACK
    heading.paragraph_format.space_after = Pt(12)

    for index, paragraph in enumerate(document.paragraphs):
        new_paragraph = master_document.add_paragraph()
        # Loop through the runs of a paragraph
        # A run is a style element within a paragraph (i.e. bold)
        for j, run in enumerate(paragraph.runs):

            text = run.text
            # If at start of paragraph and no tab, add one
            if j == 0 and not text.startswith('\t'):
                text = f"\t{text}"
            # Add run to new paragraph
            new_run = new_paragraph.add_run(text=text)
            # Update styles for run
            new_run.bold = run.bold
            new_run.italic = run.italic

        # Last minute format checking
        text = new_paragraph.text

    # Destroy the document object
    del document
    return master_document

def main(book):
    master_document = Document()

    master_document = add_matter(
      chapter='Title Page'
    master_document = add_matter(
        chapter='Copyright Page'
    master_document = add_matter(

    for filename in glob(f"{book}/*"):
        if filename.startswith(f"{book}/_"):
            print(f'skipping {filename}')

        # Get the chapter name
        book, short = filename.split('/')
        chapter = short.replace('.docx', '')
        if chapter.startswith('0'):
            chapter = chapter[1:]
        print(f'Adding {chapter}')
        master_document = add_chapter(master_document, filename, chapter)

    master_document = add_matter(
        chapter='About the Author',
    master_document = add_matter(

if __name__ == '__main__':
        book = sys.argv[1]
    except IndexError:
        msg = 'You need to specify a book. A book is a directory of word files.'
        raise Exception(msg)


This is what it looks like when I run the code:

$ python the-darkest-autumn/
Adding Title Page
Adding Copyright Page
Adding Dedication
Adding 1. Beginnings
Adding 2. Town of Ravenna
Adding 3. Walls of Ravenna
Adding 4. Gatehouse of Ravenna
Adding 5. Courage
Adding 6. To the Upper Valley
skipping the-darkest-autumn/_afterward.docx
skipping the-darkest-autumn/_copyright.docx
skipping the-darkest-autumn/_dedication.docx
skipping the-darkest-autumn/_title.docx
Adding Afterward

And now I've got a Word document in the same directory called the-darkest-autumn.docx.

Converting Word to EPUB

While Kindle Direct Publishing (KDP) will accept .docx files, I like to convert it to .epub using Calibre:

$ ebook-convert the-darkest-autumn.docx the-darkest-autumn.epub \
--authors "Daniel Roy Greenfeld" \
--publisher "Two Scoops Press" \
--series Ambria \
--series-index 1 \
--output-profile kindle

And now I can check out my results by using Calibre's book viewer:

$ ebook-viewer the-darkest-autumn.epub

Add the Links!

As python-docx doesn't handle HTTP links at this time, I manually add them to the book using Calibre's epub editor. I add links to:

How Well Does It Work?

For me it works wonders for my productivity. By following a "chapters as files" pattern within Google Docs I get solid collaboration power plus some (but not all) of the features of Scrivener. I can quickly regenerate the book at any time without having to struggle with Scrivener or have to add tools like Vellum to the process.

I have a secondary script that fixes quoting and tab issues, written before I realized Calibre could have done that for me.

The book I started this project for, The Darkest Autumn, is available now on Amazon. Check it out and let me know what you think of what the script generates. Or if you want to support my writing (both fiction and non-fiction), buy The Darkest Autumn on Amazon and leave an honest review.

Thinking About the Future

Right now this snippet of code generates something that looks okay, but could be improved. I plan to enhance it with better fonts and chapter headers, the goal to generate something as nice as what Draft2Digital generates.

I've considered adding the OAuth components necessary in order to allow for automated downloading. The problem is I loathe working with OAuth. Therefore I'm sticking with the manial download process.

For about a week I thought about leveraging it and my Django skills to build it as a paid subscription service and rake in the passive income. Basically turn it into a startup. After some reflection I backed off because if Google added file combination as a feature, it would destroy the business overnight.

I've also decided not to package this up on Github/PyPI. While Cookiecutter makes it trivial for me to do this kind of thing, I'm not interested in maintaining yet another open source project. However, if someone does package it up and credits me for my work, I'm happy to link to it from this blog post.

Cover for The Darkest Autumn

Two Scoops of Django 1.11 Is Printed!

Two Scoops of Django

After longer than we expected, the shiny new print copies of Two Scoops of Django 1.11 are finally ready. We've shipped all pre-orders, and many of you who ordered it in advance should have the book in your hands. We're delighted by how well it's been received, which is due only to the help and encouragement of our readers.

Right now you can order the print version of Two Scoops of Django 1.11 at either:

  • Two Scoops Press (for autographed copies directly from us), or
  • Amazon (this link sends you to your regional Amazon store)

If you purchase the book on Amazon, please leave an honest review. Simply put, Amazon reviews make or break a book, each one responsible for additional sales. It's our dream that if sales of Two Scoops of Django 1.11 are good enough, we'll be able to justify doing less consulting and instead, writing more technical books.

Two Scoops of Django Birthday Giveaway

Coldest Spring

Today is a special day. You see, the latest fantasy book I co-authored, Coldest Spring (3rd in the Ambria series), is out AND it's my birthday. In honor of of these occasions, the first book of the series, Darkest Autumn, for 0.99 cents on Amazon. Plus, I'm doing a Python-related giveaway.

The Giveaway: There's a python programming easter egg in the book. One of the characters mentions the elegance of Python. The first person who tells me in the comments below the name of the father and grandfather of that character gets an autographed Two Scoops of Django 1.11 book shipping anywhere in the world.

Finally, if you want to give me a truly special present for my birthday, please review any or all of my books on Amazon. Positive or negative, every review makes a difference. The positive stuff feeds my already inflated ego, and the negative stuff tells me what I need to do in order to improve.


  • July 24th 3pm: A commenter by the name of @lingot answered first and they'll get the autographed book. However, there is an even more precise response than the one they gave. If anyone can provide that answer, they get an autographed TSD 1.11 as well. Good luck!

2018 New Years Resolutions

Lasergun Light Painting

Happy New Year!

The last time I wrote down resolutions was way back in 2014. I had done it for many years at that point, dating back to even before my old blog. Somehow I fell out of what I consider a positive habit. Well, it's time to pick it up again!

So here are my resolutions for 2018:

  1. Weight down to 160.
  2. Work out for 60 minutes a day. I got lazy in the last year.
  3. Start martial arts again. Because of knee and ankle injuries, Capoeira is probably right out. Already up Kali/Escrima under Guro Mestre Xingú instead.
  4. Write at least 3 books (last year we did 5 books, so this is doable!). Out of those, one will be about coding in some way.
  5. Blog at least once a month, about anything. Python, Django, serverless coding, martial arts, whatever. And with this post, January is done!
  6. Release some coding projects I can't talk about yet.
  7. Travel outside the USA. That looks to be a trip to Colombia to speak at PyCon Colombia! If you are in South America, please meet me (and Audrey) there! I'll be sharing more details soon. :-)

note: The photo is from a light painting session I did with Audrey at the start of last year.

Use Bitcoin to Get Two Scoops of Django at 25% Off

BitCoin and Two Scoops of Django!

Like the title of this blog post says, for Bitcoin purchases we're offering a 25% discount for purchases of Two Scoops of Django. That puts the ebook version at $34.36 and the autographed hardcopy at $38.36. Pretty awesome, right? If you want to take advantage of this awesome deal, the Bitcoin discount is applied during checkout.

Combining the Bitcoin Discount With the Bulk Discount

Yes, the Bitcoin discount can be combined with bulk orders. So if you order 15 books or more, you get both the 20% bulk discount and the 25% bitcoin discount. That means each book is bought at $28.77 versus $47.95! Furthermore, bulk orders are shipped free to anywhere in the world. This makes it an incredible deal for companies, organizations, and user groups.

Stay tuned!

Hola, PyCon Colombia!

We (me and Audrey) are going to be giving a keynote speech at PyCon Colombia on February 11th! Hooray!

We'll be arriving in Medellin late evening on February 7th and staying a while longer after the conference so we can have the time to explore the lovely city of Medellin. We're very excited, because travel is a rarity for us now and Medellin (and the surrounding area) is supposed to be quite beautiful. Plus, all the Colombians we know online are excellent people - we can't wait to meet them!

Our hope is that on the day(s) after PyCon Colombia we can see the sights and eat the foods with Colombians who know Medellin. So let me know if you want to meet up!

Hasta pronto!

When to Use MongoDB with Django

Short Answer

You don't.

Long Answer

First off, let's get one thing out of the way. This isn't a bash on MongoDB. MongoDB works great with lots of things (Flask, Tornado, Node, etc), but it's a mismatch with Django. In other words, this article is about using the right tool for the right job.

Second, I'm not speaking from ignorance. In fact, I have quite a bit of experience combining MongoDB and Django. You can see some of my early work with combining these tools in the defunct django-mongonaut.

Okay then, let's get into the background of this post: On various Django help forums, you'll hear requests from new-ish Django developers on how to use MongoDB with Django. Most of the time they want to replace the Django ORM with calls to MongoDB. Here are the reasons I've heard so far.

The 90% Reason: JSON storage

Most of the time people want to replace SQL with MongoDB in Django, the reason is they want to store JSON data and search it.

In which case, they should use Django's built-in implementation of PostgreSQL's JSON field. It's not just a string field, it's fully searchable. Implementation example below:

from django.contrib.postgres.fields import JSONField
from django.db import models

class Product(models.Model):

    metadata = JSONField(null=True, blank=True)

Just save your data into this field using a JSON-serializable dict. It's that easy. Even better, in this previous article, I show how you can pretty print the JSON in the Django admin.

Using this field, you get all the built-in features of Django and a searchable JSON field without the mess of stapling a non-relational database (MongoDB) into a framework and vibrant ecosystem designed to work with relational databases (Django).

For those using MySQL instead of PostgreSQL, there's always Django-MySQL's jsonfield.

The 5% Reason: Performance

A fraction of people want to use MongoDB with Django because of supposed performance reasons. Sure, if you run MongoDB without any write safeguards and decide to forego database transactions, it will run faster than any relational storage. However, that's a dangerously insecure approach to things. It's simply not worth the risk of corrupted data.

Don't take my word for it, spend an hour searching for articles about write safety in MongoDB. Ignore the hype articles published by, read what real businesses and projects case studies have to say.

Also, if you want to speed database i/o up with Django, standard practice is to employ asynchronous tools such as Celery before switching out the datastore.

The 4% Reason: Scaling Up

Anyone who tells you that relational databases can't scale as well as MongoDB (or anything else) is selling you something. Or was sold on something and don't want to admit they bought the hype.

Again, don't take my word for it, spend an hour searching for articles about scaling issues with MongoDB. Again, ignore the marketing and read real case studies.

The 1% Reason: Management

Every once in a while someone tells me that using MongoDB with Django is a management decision. In which case, they should send their boss(es) to this blog post.

Management should know that Django is designed to be used with a relational database backend (PostgreSQL, MySQL) and a key/value store for ephemeral data (Redis, Memcached). Going beyond that design is going to make development slower and frustrate the team. Even if your team can work around this issue, they'll be hampered by not being able to fully exploit tools within the Django ecosystem.

If You Must Use MongoDB, Use Flask Instead

There's nothing wrong with MongoDB. However, it's suboptimal when used with Django. If you use it with correct write permissions, MongoDB doesn't provide any speed benefits with Django. You also lose many of the advantages of Django (database transactions, rock-solid security, forms, easy Django REST framework use, hundreds of third-party packages, etc). There is quite a lot of things you are going to have to rewrite.

And if you're going to have to rewrite that much of Django's functionality to use MongoDB, you might as well be using Flask. Honestly, this isn't a bad choice, as the flexibility of Flask makes it perfect for use with non-relational databases.

I know, because this is how we use Flask on the job, which is not with relational data. We have dozens of microservices that rely on DynamoDB. While DynamoDB isn't MongoDB, they are similar enough that I can tell you this approach is delightful. Keep an eye out for my upcoming article about it.

Stay Tuned!

If you found this article useful and want to see more like it, or if you want to encourage me to do more open source work, hit me up on Patreon.

Implemementing Manual Schema with Django REST Framework

BitCoin and Two Scoops of Django!

This is what will hopefully be the first in a series of reference articles for using Core API libraries with Django REST Framework (DRF).

This is an extraction from an existing production view running Django 1.11/2.0 on Python 3.6. The original code did something else, but for contract reasons I'm demonstrating this code with sending email.

Please note that this article is very terse, with almost no description, no tests, and no URL routing. Just enough code so that if you have a decent understand of DRF, you can make custom views work with Core API.

First, the serializer:

from django.core.mail import send_mail
from markdown import markdown
from rest_framework import serializers

class EmailSerializer(serializers.Serializer):
    to_addresses = serializers.ListField(
    from_email = serializers.EmailField(required=True)    
    subject = serializers.CharField(required=True)
    message = serializers.CharField(required=True) 
    htmlize = serializer.BooleanField(required=False, default=False)  

    def create(self, validated_data):
        if validated_data['htmlize']:
            validated_data['html_message'] = markdown(validated_data['message'])

Now the view:

import coreapi
import coreschema
from rest_framework import schemas
from rest_framework.views import APIView

from .serializers import EmailSerializer

class EmailCreateAPIView(APIView):
    """ Assumes you have set permissions and authentication in ``"""
    serializers = EmailSerializer
    schema = schemas.ManualSchema(fields=[
              description="List of email addresses"

    def post(self, request, format=None):
        serializer = EmailSerializer(
        if serializer.is_valid():
            return Response(, status=status.HTTP_201_CREATED)
        return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

Stay Tuned!

I've decided to start posting my coding notes online again. These aren't tutorials, and may not be beginner friendly. Rather, these are code examples extracted from production systems that I'm putting up in a location I can reference easily that's 100% under my control.

If you like what I'm doing, hit me up on Patreon.