On: Python Is Not Java

[pchivers]

XML is not the answer. It is not even the question.

I have a question for the Python hackers in the audience: what file format(s) do you use for configuration files? I've been using INI style files up until now, but I've been thinking of switching to XML for more complicated formats. For example, what about something like the following example. What format would you use to store this type of configuration data for a Python application?

  <?xml version="1.0" encoding="utf-8" ?>
  <directoriesToBackUp>
  	<directoryInfo>
		<directoryName>/media/programming</directoryName>
		<excludedDirs>
			<dir>/media/programming/log_files</dir>
			<dir>/media/programming/temp_files</dir>
		</excludedDirs>
	</directoryInfo>
	<directoryInfo>
		<directoryName>/media/documents</directoryName>
		<excludedDirs>
			<dir>/media/documents/archive</dir>
		</excludedDirs>
	</directoryInfo>
  </directoriesToBackUp>

[moe → pchivers]

what file format(s) do you use for configuration files?

YAML.

  directoriesToBackup:
      /media/programming:
          exclude:
              - /media/programming/log_files
              - /media/programming/temp_files
      /media/documents:
          exclude:
              - /media/documents/archive

[Deestan → moe]

Wow! I've heard YAML mentioned before, but this is the first actual example I've seen.

I'm very embarrassed that this beautiful standard has been around for ~5 years without me noticing it.

/me goes off to read http://yaml.org/spec/1.2/

[ovi256 → Deestan]

  /me goes off to read http://yaml.org/spec/1.2/

There's another problem. YAML is so simple, you don't need to read the damn manual for it. That example already taught you 80%. Ok, the easy 80%. But if you come from the Java (and XML) world, reading the manual is what you do just to get started.

[moe → ovi256]

Reading the manual once (at least the part with the examples) is indeed a good idea to get an overview of the possible constructs but once you have them down it quickly becomes a no-brainer. Under the hood YAML is surprisingly complex (can do full blown object serialization if you want to) but the everyday stuff is just perfect for typing it into an editor without thinking about it - and without matching up braces all the time like in JSON.

[arockwell → pchivers]

Try JSON:

  { 
  "directories_to_backup": [
    {"directory_name": "/media/programming",
     "exclude_dirs": ["/media/programming/log_files",  
                      "/media/programming/temp_files" ]
    },
    {"directory_name": "/media/documents",
     "exclude_dirs": ["/media/documents/archive"]
    }

}

Also, why not just use a python data structure? Do you really need a plain text config file?

[dnaquin → pchivers]

config.py:

  backup = [{
        'name': '/media/programming',
        'exclude': ['log_files', 'temp_files'],
    }, {
        'name': '/media/documents',
        'exclude': ['archive'],
    }]

of the top of my head.

*editted for formatting

[dnaquin → dnaquin]

  jeff:disqus devin$ wc settings.py settings.xml
       1      15     142 settings.py
      16      19     461 settings.xml
      17      34     603 total

30% of the words. and, in my opinion, just as readable if not more.

[fauigerzigerk → dnaquin]

Or with a little less clutter :-)

  <backup>
    <dir name="/media/programming" exclude="log_files temp_files" />
    <dir name="/media/documents" exclude="archive" />
  </backup>

[snprbob86 → dnaquin]

i'd probably define some helper functions to check argument values and such:

  config = backup(
    directory('/media/programming', exclude=['log_files', 'temp_files']),
    directory('/media/documents', exclude='archive')
  )

[pchivers → dnaquin]

Yeah, I've done that as well.

Are there any security considerations to think about if using that approach?

[dnaquin → pchivers]

well. it's mutable. your code can change it. don't do it (often). then you're just programming with lots of global state.

[statictype → pchivers]

Thats interesting because I recently tackled the same problem of storing configuration details for backing up files.

I took Python's ConfigParser module and modified it to suit my needs. The config file looks a bit like this:

    [directory-details]
    path = /media/programming
    exclude_dirs = *log_files, *temp_files

    [directory-details]
    path = /media/documents
    exclude_dirs = /media/documents/archive

Honestly, though, xml is a perfectly valid format for storing such configuration.

[moe → statictype]

Honestly, though, xml is a perfectly valid format for storing such configuration.

Not when a human has to touch it without tool support.

[njharman → pchivers]

Disclosure: I write very little code targeted at non-technical endusers, in fact most my code is for my own use. These facts strongly influence my config file solution.

If find the need for a config file (I tend to like command line options better) 95% of the time I use python code and import.

The benefits are many; zero work required, known syntax, powerful syntax, arbitrarily complex, no parsing required, nice traceback/errors with screwed up syntax.

    #file config.py
    
    debug = True
    
    directories_to_backup = [
    #   path                  exclude these subdirs
       ("/media/programming", ("log_files","temp_files")),
       ("/media/documents",   ("archive",)),
    ]
    
    # or, if you relish verbosity
    directory_info = dict
    directories_to_backup = [
       directory_info(path="/media/programming", exclude=("log_files","temp_files")),
       directory_info(path="/media/documents", exclude=("archive",)),
       ]
    
    # the options are as endless as your knowledge of Python

Usage

    import config
    
    for path, excludes in config.directories_to_backup:
      garbleblaster(path, excludes)

[callahad → pchivers]

There's a talk specifically about this conundrum at PyCon this year! You should come!

http://us.pycon.org/2009/conference/schedule/event/5/