Documentation for geocoder.py¶
parse_with_passyunk_parser(parser, address_col: str, lf: pl.LazyFrame) -> pl.LazyFrame
¶
Given a polars LazyFrame, parses addresses in that LazyFrame using passyunk parser, and adds output address.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
parser
|
A passyunk parser instance |
required | |
address_col
|
str
|
The address column to parse |
required |
lf
|
LazyFrame
|
The polars lazyframe with an address field to parse |
required |
Returns:
| Type | Description |
|---|---|
LazyFrame
|
A polars lazyframe with output address, and address validity booleans |
LazyFrame
|
added. |
build_enrichment_fields(ais_enrichment_fields: list, srid_4326: bool, srid_2272: bool) -> tuple[list, list]
¶
Given a config dictionary, returns two lists of fields to be added to the input file. One list is the address file fieldnames, the other is the AIS fieldnames.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ais_enrichment_fields
|
list
|
A list of fields to append |
required |
srid_4326
|
bool
|
Whether or not to append SRID 4326 |
required |
srid_2272
|
bool
|
Whether or not to append SRID 2272 |
required |
Returns:
| Type | Description |
|---|---|
tuple[list, list]
|
A tuple with AIS fieldnames and address file fieldnames. |
add_address_file_fields(geo_filepath: str, input_data: pl.LazyFrame, address_fields: list, srid_4326: bool, srid_2272: bool) -> tuple[pl.LazyFrame, dict]
¶
Given a list of address fields to add, adds those fields from the address file to each record in the input data. Does so via a left join on the full address.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
geo_filepath
|
str
|
The filepath to the address_file. This is the main |
required |
input_data
|
LazyFrame
|
A lazyframe containing the input data to be enriched |
required |
address_fields
|
list
|
A list of one or more address fields |
required |
srid_4326
|
bool
|
Whether or not to append SRID 4326 |
required |
srid_2272
|
bool
|
Whether or not to append SRID 2272 |
required |
Returns:
| Type | Description |
|---|---|
tuple[LazyFrame, dict]
|
The appended data and a dict of renamed fields if there were fieldname conflicts |
Geocoder
¶
Handles the full geocoding pipeline for a batch of addresses. Does so without loading full address file into memory.
Pipeline
- Parse and join addresses to a local address file (Polars lazy, sink to disk)
- Iterate through sunk output, process unmatched records via AIS/TomTom APIs
- Write enriched records incrementally to output file
__init__(config: dict)
¶
Given a config, initialize the Geocoder object.
geocode() -> None
¶
Runs the full geocoding pipeline on the configured input file.
Joins input addresses to the local address file via Polars lazy execution, then iterates through unmatched records and attempts enrichment via AIS and TomTom. Writes output incrementally to avoid memory pressure on large files.
If resume is set in the config, skips already-written rows and continues
from where the previous run left off.
Output is written to the same directory as the input file, with _enriched
appended to the filename.