← Back to Explorer

About & Methodology

How this tool works, what we discovered building it, and why it matters.

1. Why This Tool Exists

NYC's 76 Business Improvement Districts (BIDs) manage streetscape, sanitation, and public safety in commercial corridors across all five boroughs. To understand quality-of-life issues in their district, BID managers need to see 311 complaint patterns — what types of complaints are filed, where they cluster, how quickly they're resolved, and how trends change over time.

The existing NYC 311 portal at portal.311.nyc.gov/check-status/ offers filters for borough, address radius, problem type, and date range — but not by BID. There is no way to draw a BID boundary and see all complaints within it. This tool fills that gap.

Existing Portal Filter Parameters

Parameter	Type	Limitation
borough	Dropdown	Too broad — a borough contains dozens of BIDs
minlatitude / maxlatitude / minlongitude / maxlongitude	Bounding box	Rectangular only, can't follow BID boundaries
fromdate / todate	Date range	Works well, no issues
problemarea / problem	CRM option set IDs	Internal Dynamics 365 values, not documented
communitydistrict	Text	BIDs don't align with community districts

None of these parameters support polygon-based spatial queries. A BID manager would need to manually adjust a bounding box, visually filter results, and repeat for each date range — an impractical workflow for ongoing monitoring.

2. Data Sources

NYC Open Data — 311 Service Requests

The primary data source is the NYC Open Data 311 Service Requests dataset, containing over 20 million records since 2010. It is updated daily, provides rich metadata per record, and offers a fully public SODA API with spatial query support.

• Endpoint: data.cityofnewyork.us/resource/erm2-nwe9.json
• CORS: Fully open (Access-Control-Allow-Origin: *)
• Spatial queries: within_box(), within_circle(), within_polygon()
• Record limit: None (paginated, unlimited)

NYC 311 Portal — Internal API

The 311 portal at portal.311.nyc.gov runs on Microsoft Dynamics 365 Power Apps Portals. It has an internal pin-fetching endpoint used by its map interface.

• Endpoint: portal.311.nyc.gov/entity-pin-fetch-service-requests/
• CORS: Locked to www1.nyc.gov only
• Record limit: 100 records maximum, no pagination
• Response: Returns HTML page wrapping JSON data

Field Comparison

Field	Open Data	Portal
Unique Identifier	unique_key (e.g., 64365278)	srnumber (e.g., 311-22260438)
Complaint Type	complaint_type	data.problem
Descriptor/Details	descriptor	Detail page only
Status	status	data.status
Agency	agency, agency_name	Not available
Created Date	created_date (UTC)	submitteddate (Eastern)
Closed Date	closed_date	Detail page only
Resolution	resolution_description	Detail page only
Coordinates	lat/lng + state plane	lat/lng
Address	Structured fields	Combined string
BBL (Parcel ID)	bbl	Not available
Community Board	community_board	Not available
Channel	open_data_channel_type	Not available
Portal Detail Link	Not available	CRM GUID for /sr-details/

3. How We Discovered the Portal API

The 311 portal's internal API is not documented. We discovered it by reverse-engineering the JavaScript on the /check-status/ page. The portal uses an Esri ArcGIS map with clustered markers, and a FilterManager class that aggregates multiple filter types before sending requests to the backend.

The key endpoint is /entity-pin-fetch-service-requests/, which accepts GET parameters for borough, date range, bounding box coordinates, and CRM-internal option set IDs for problem area and problem type. Filter option values are loaded dynamically from:

• /get-optionset-filter-options/?entity=n311_srproblem&attribute=n311_problemareaglobal
• /get-lookup-filter-options/?lookupentity=n311_srproblem&lookupidattribute=n311_srproblemid&...

Each pin in the response contains:

{
  "id": "70f3d629-59e0-ef11-95f5-7c1e52e4043a",  // Dynamics 365 CRM GUID
  "label": "Noise - Street/Sidewalk",
  "sublabel": "616 WEST 184 STREET, MANHATTAN (NEW YORK), NY, 10033",
  "latitude": "40.8513685798",
  "longitude": "-73.9334866503",
  "data": {
    "address": "616 WEST 184 STREET, MANHATTAN (NEW YORK), NY, 10033",
    "problem": "Noise - Street/Sidewalk",
    "submitteddate": "2/1/2025 4:59:48 AM",
    "status": "Closed",
    "srnumber": "311-21789902"
  }
}

Portal Limitations

How We Bypass the 100-Record Cap

A single portal request for a BID over a 30-day range might contain 500+ complaints, but the API only returns the 100 most recent. Pagination parameters don't exist. To capture all records, our server uses an adaptive multi-phase strategy:

Phase 1: Daily Time Windows

Instead of one request for the full date range, the server splits it into one request per day. Most BIDs receive 20-50 complaints per day — well under the 100 cap. For a 30-day query, this means 30 parallel requests (running 8 at a time) instead of 1, capturing nearly 100% of records.

Phase 2: Spatial Subdivision (Fallback)

If any single day returns exactly 100 records (meaning it hit the cap), that day is automatically re-fetched using spatial tiling. The BID's bounding box is divided into a 2×2 grid of smaller sub-boxes, and each tile is queried separately for that day. This works because the portal's date filter only accepts whole dates — you can't split by time-of-day, so space is the only remaining axis to subdivide.

In practice, Phase 2 rarely triggers. Even Times Square — one of NYC's busiest BIDs — stayed under 100 complaints per day across a full month of testing. Phase 2 exists as a safety net for extreme cases (e.g., a major event or outage generating a surge of complaints in one district on one day).

Interactive Demo: How Spatial Splitting Works

Use the slider below to simulate increasing complaint volume for a single day in the Hudson Square BID (bounding box: 819m × 530m). Watch what happens when the 100-record portal cap is reached.

Phase 3: Deduplication & Caching

After spatial tiling, records near tile boundaries may appear in multiple tiles. The server deduplicates by SR number before returning results. The complete result set is then cached in memory for 10 minutes, so repeat queries are instant.

Phase	What It Does	When It Runs	Calls
1	Split date range into daily windows, fetch in parallel (8 concurrent)	Always	1 per day
2	Re-split capped days into 2×2 spatial tiles	Only if a day hits 100	4 per capped day
3	Deduplicate by SR number + cache 10 min	Always	0

Scenario	Total Calls	Estimated Time
Quiet BID, 7 days	7	<1 second
Busy BID, 30 days (no caps)	30	~3 seconds
Busy BID, 30 days (2 capped days)	30 + 8 = 38	~4 seconds

4. Record Matching: No Primary Key

The two data sources use completely different identifier systems with no crosswalk:

• Open Data: unique_key — numeric, e.g., 64365278 (64 million range for 2025 records)
• Portal: srnumber — prefixed, e.g., 311-22260438 (22 million range for the same period)

Initial hypothesis: strip "311-" from the SR number to get the unique_key. This is wrong. The numbers are in entirely different ranges and represent independent ID systems.

Composite Key Matching

Through systematic testing of 200+ records across Manhattan and Brooklyn, we developed a composite key matching strategy:

1. Timestamp alignment: The two sources store timestamps with a consistent offset equal to the Eastern Time UTC offset (4-5 hours depending on DST). Empirically verified: subtracting the ET offset from the portal's numeric timestamp produces the Open Data timestamp exactly. We apply this correction and allow a ±3 minute window to handle timestamp rounding in certain complaint types.
2. Complaint type matching: Case-insensitive comparison. Portal may use "Heat/Hot Water" while Open Data uses "HEAT/HOT WATER".
3. Address verification: Extract street name (remove house number and borough suffix) and check for substring match.

Verified Match Rates

Test	Method	Match Rate
Manhattan, exact timestamp	Timestamp + type + address	80/100 (80%)
Manhattan, 3-min window	Fuzzy timestamp + case-insensitive	91/100 (91%)
Brooklyn, 3-min window	Fuzzy timestamp + case-insensitive	91/100 (91%)
Reverse (OD → Portal)	Same composite key	96/102 (94%)

Why ~9% Don't Match

• ~7-8% genuinely absent from Open Data — likely NYPD-sensitive or privacy-withheld records
• Timestamp rounding: Generic "Noise", "Food Poisoning", "Street Light Condition" have seconds zeroed out in Open Data
• Case mismatches in complaint types (e.g., title case vs ALL CAPS)
• Intersection records with incident_address = NULL in Open Data

5. Why Open Data Is the Source of Truth

• CORS fully open — browser can call it directly, no proxy needed
• No record limit — paginated queries return all matching results (vs. portal's 100-record cap)
• Richer fields — agency, descriptor, resolution, BBL, community board, channel
• Spatial query support — within_box() lets us query by bounding box natively
• Parcel-level geocoding — 95% of records have BBL (Borough-Block-Lot), 100% have state plane coordinates
• 20+ million records going back to 2010, updated daily

The portal's value is its CRM GUID which enables deep-linking to the /sr-details/ page for full agency response details. When records match (~91%), we include this link in the map popup.

6. The BID Polygon Challenge

The BID boundary dataset from NYC Open Data (data.cityofnewyork.us/resource/7jdm-inj8) does not contain district boundaries. Instead, it contains tax lot parcel polygons — the individual building footprints within each BID. Streets, sidewalks, and public rights-of-way are excluded.

The Problem

BID	Polygon Rings	Area Covered	Street Gaps
Times Square Alliance	45	36%	64%
Alliance for Downtown NY	119	37%	63%
Village Alliance	40	12%	88%
Union Square Partnership	20	13%	87%
Diamond District	3	31%	69%

If we used the raw polygons for point-in-polygon filtering, we would miss 60-88% of 311 requests — especially street-level complaints (noise, potholes, sidewalk conditions, encampments) which are geocoded to street centerlines.

The Solution: Polygon Buffering

We buffer each BID's parcel polygons by 30 meters (approximately half the width of a typical NYC street) using Turf.js. The buffered shapes overlap across streets, creating a continuous boundary that covers the full district area.

Buffer Distance	Gap Coverage	Notes
10m	55%	Too narrow for wide avenues
15m	70%	Covers most side streets
20m	76%	Good balance
30m	84%	Used in this tool — covers avenues
50m	93%	Risk of including adjacent non-BID areas

On the map, you can see both layers: the blue dashed outlines are the raw tax lot parcels, and the yellow solid outline is the 30m-buffered boundary used for filtering.

7. Coordinate Accuracy

The two data sources geocode addresses independently, producing slightly different coordinates for the same complaint:

Metric	Value
Mean distance between sources	37.68m
Max distance	89m
Within 50m	85% of records
Within 100m	100% of records

Open Data provides parcel-level geocoding: 95% of records have a BBL (Borough-Block-Lot) identifier, and 100% have state plane coordinates. This means Open Data coordinates point to the actual tax lot associated with the address.

The portal appears to use a different geocoding service that produces rooftop or street-centerline coordinates. For map display, we use Open Data coordinates as primary (matched and OD-only records) and portal coordinates only for portal-only records.

8. How the Map Works

Data Flow

1. User selects a BID from the dropdown
2. The raw tax lot parcels are buffered by 30m and merged into a single boundary
3. A bounding box (with 50m padding) is computed around the buffered boundary
4. Two parallel API calls are made:
   • Open Data (direct from browser): SODA within_box() query with date range, paginated
   • Portal (via server proxy): Pin-fetch with bounding box coordinates, max 100 results
5. Returned points are filtered client-side using Turf.js booleanPointInPolygon() against the buffered boundary
6. Records are matched between sources using the composite key (timestamp + type + address)
7. Results are classified as Matched (both), Open Data only, or Portal only
8. Pins are plotted with color-coding by complaint type (fill) and source (border)

Pin Types

Pin Border	Source	Expected %	What You Get
• Green	Matched (both)	~85%	Full Open Data fields + portal detail link
• Blue	Open Data only	~9%	Rich fields, no portal link
• Orange	Portal only	~6%	Basic fields + portal detail link

9. Future Expansion

This tool's architecture — BID polygons + geospatial data overlay — can be extended to incorporate other NYC datasets for a comprehensive district dashboard:

10. Technical Stack

Component	Technology	Why
Server	Node.js + Express	Proxies portal API (CORS bypass), serves static files, easy deployment
Map	Leaflet + MarkerCluster	Free, lightweight, handles thousands of pins with clustering
Spatial Processing	Turf.js	Client-side polygon buffering, point-in-polygon, bounding box computation
Charts	Chart.js	Lightweight charting with no build step
Date Picker	Flatpickr	Tiny, range-mode support, dark theme
Heatmap	Leaflet.heat	Density visualization layer
Primary API	NYC Open Data SODA	CORS open, unlimited records, spatial queries
Secondary API	311 Portal (proxied)	Portal detail links, supplementary data

Built with Claude · Data sourced from NYC Open Data and the NYC 311 Portal