I have some census data in a fixed width format I’d like to parse. Stuff like “columns 23-26 are the year in 4 digit form”. It’s easy enough to ad hoc parse this with string slicing. But by the time you handle naming, and type conversion, and stripping padding, and validation, etc etc you end up with a fair amount of code. You can parse CSV with just string split too, but anyone sane uses the CSV module. Is there a good fixed width module?
Not that I could find. I gave up and just did the ad hoc thing.
I thought FixedWidth was a candidate but after 20 minutes trying it, gave up on it. There’s packaging problems and the docs are poor. The tests are incomplete. The API is weird and seems more designed for emitting fixed width than parsing it. The final reason I gave up is it seems to require you specify a full schema; you can’t parse columns 99-103 unless you’ve said what to do with columns 1-98 forst. That was a nuisance.
The other option I found was Pandas read_fwf . I didn’t try it because Pandas is overkill for my project. But I know from CSV work that DataFrame is really nice, and the Pandas CSV module is quite comprehensive. I also know that even after parsing with read_csv you still have to do a lot of work to get it into a clean DataFrame. I’d definitely look into using this for more serious work.
Related question: are there standard metadata descriptions for fixed width formats? The census data has this thing called data dictionaries that are clearly meant to be parseable. But they’re in at least two formats right on the site. I feel like I’ve seen other government records with similar metadata descriptions.
Further reading: Extract, transform, and load census data with python .